Image processing device, method and program

Abstract

To allow the accuracy of positioning with a three-dimensional image to be checked in a two-dimensional image such as a transparent image in image processing device, method, and program.SOLUTION: A processor specifies a target position in a three-dimensional image acquired by imaging a subject before treatment, executes positioning between two-dimensional images sequentially acquired of the subject under treatment and the three-dimensional image, derives an evaluation result indicating reliability of positioning at least at the target position of the three-dimensional image, derives a range in which there is a possibility of the target position being present in the two-dimensional images on the basis of the positioning result and the evaluation result, and displays the two-dimensional image in which the range is superimposed.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, a method, and a program. [Background technology]

[0002] An ultrasound endoscope having an endoscopic observation unit and an ultrasound observation unit at its tip is inserted into a lumen such as the digestive organs or bronchi of a subject to capture endoscopic images of the lumen and ultrasound images of a lesion or other site outside the lumen wall. Biopsy procedures are also performed in which tissue from a site outside the lumen wall is sampled using a treatment tool such as forceps attached to the tip of the endoscope.

[0003] When performing such treatment using an ultrasonic endoscope, it is important to accurately guide the ultrasonic endoscope to the target location within the subject. For this reason, during treatment, radiation is continuously irradiated onto the subject from a radiation source, and fluoroscopic images obtained by this are displayed in real time, thereby enabling the positional relationship between the ultrasonic endoscope and the human body structure to be grasped.

[0004] Here, since the fluoroscopic image contains overlapping anatomical structures such as organs, blood vessels, and bones within the subject, it is not easy to recognize the lumen and lesion. For this reason, a 3D image of the subject is acquired in advance before treatment using a CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, or the like, and the lesion position is identified in the 3D image. Then, the 3D image and the fluoroscopic image are aligned to identify the lesion position in the fluoroscopic image (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2020-137796 Summary of the Invention [Problem to be solved by the invention]

[0006] The method described in Patent Document 1 aligns a three-dimensional image with a two-dimensional fluoroscopic image. As a result, there may be areas in the fluoroscopic image where the alignment is not accurate enough. However, with the method described in Patent Document 1, it is unclear how accurately the alignment is performed, so it may take a long time to guide the ultrasound endoscope to the target position, such as a lesion, while viewing the fluoroscopic image.

[0007] The present disclosure has been made in consideration of the above circumstances, and aims to make it possible to check the accuracy of alignment between a two-dimensional image such as a perspective image and a three-dimensional image. [Means for solving the problem]

[0008] The image processing device according to the present disclosure includes at least one processor, the processor identifying a target position in a three-dimensional image acquired by imaging a subject before treatment; registering sequentially acquired two-dimensional and three-dimensional images of the subject undergoing treatment; Deriving an evaluation result representing the reliability of the registration at least at the target position of the three-dimensional image; deriving a range in the two-dimensional image in which the target position may exist based on the alignment result and the evaluation result; Displays a 2D image with the range superimposed.

[0009] In the image processing device according to the present disclosure, the processor may superimpose a corresponding target position corresponding to the target position in the two-dimensional image on the two-dimensional image.

[0010] In the image processing device according to the present disclosure, the processor may display the evaluation result superimposed on the two-dimensional image.

[0011] Furthermore, in the image processing device according to the present disclosure, the processor may derive the evaluation result by projecting a standard deformation amount between the first time phase and the second time phase of the organ including the target position, which is derived in advance based on a group of multiple three-dimensional images including a three-dimensional image of the first time phase and a three-dimensional image of the second time phase, onto the three-dimensional image of the subject.

[0012] In the image processing device according to the present disclosure, the three-dimensional image of the first time phase may be a three-dimensional image of an inhalation phase, and the three-dimensional image of the second time phase may be a three-dimensional image of an exhalation phase.

[0013] In addition, in the image processing device according to the present disclosure, the processor derives a pseudo two-dimensional image that simulates the movement of the organ including the target position from the acquired three-dimensional image, By aligning the pseudo two-dimensional image with the acquired three-dimensional image, a registration error is derived that represents the relative amount and direction of deformation between the pseudo two-dimensional image and the acquired three-dimensional image; The evaluation result may be derived based on the alignment error.

[0014] In addition, in the image processing device according to the present disclosure, the processor repeatedly aligns the two-dimensional image and the three-dimensional image, A relationship between the number of times alignment is performed and the similarity between the aligned two-dimensional image and the aligned three-dimensional image, which is derived each time alignment is performed, is derived; Derive multiple local solutions in the relation, The evaluation result may be derived based on statistics of the registration error between the two-dimensional image and the three-dimensional image when each of the multiple local solutions is derived.

[0015] In addition, in the image processing device according to the present disclosure, the processor may derive an evaluation result using a trained model that has been machine-learned to output an evaluation result when a pseudo-two-dimensional image in which a three-dimensional image is projected in the direction in which the two-dimensional image is captured and a two-dimensional image are input.

[0016] In the image processing device according to the present disclosure, the three-dimensional image may be a CT image.

[0017] In the image processing device according to the present disclosure, the target position may be included in the lungs of the subject.

[0018] In the image processing device according to the present disclosure, the target position may be a position where a lung lesion exists.

[0019] In the image processing device according to the present disclosure, the target position may be a branching position in the bronchi.

[0020] The image processing method according to the present disclosure includes identifying a target position in a three-dimensional image acquired by photographing a subject before treatment; registering sequentially acquired two-dimensional and three-dimensional images of the subject undergoing treatment; Deriving an evaluation result representing the reliability of the registration at least at the target position of the three-dimensional image; deriving a range in the two-dimensional image in which the target position may exist based on the alignment result and the evaluation result; Displays a 2D image with the range superimposed.

[0021] The image processing program according to the present disclosure includes a procedure for identifying a target position in a three-dimensional image acquired by imaging a subject before treatment; a step of registering two-dimensional and three-dimensional images sequentially acquired of a subject undergoing treatment; deriving an evaluation result representing the reliability of registration at least at the target position of the three-dimensional image; a step of deriving a range in which the target position may exist in the two-dimensional image based on the alignment result and the evaluation result; and displaying a two-dimensional image on which the range is superimposed. [Effects of the Invention]

[0022] According to the present disclosure, it is possible to check the accuracy of alignment between a two-dimensional image such as a perspective image and a three-dimensional image. [Brief explanation of the drawings]

[0023] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of a medical information system to which an image processing device according to a first embodiment of the present disclosure is applied. [Figure 2] FIG. 1 is a diagram showing a schematic configuration of an image processing apparatus according to a first embodiment; [Figure 3] Functional configuration diagram of an image processing apparatus according to a first embodiment [Figure 4] FIG. 1 is a diagram schematically illustrating processing performed by an image processing apparatus according to a first embodiment. [Figure 5] A diagram showing the range in which the corresponding target position may exist. [Figure 6] A diagram showing the range in which the corresponding target position may exist. [Figure 7] 1 is a flowchart showing the processing performed in the first embodiment. [Figure 8] FIG. 10 is a diagram showing fluoroscopic images sequentially acquired in the first embodiment. [Figure 9] FIG. 10 is a diagram for explaining derivation of a deformation amount in the second embodiment; [Figure 10] FIG. 10 is a diagram schematically illustrating processing performed by an image processing apparatus according to a second embodiment. [Figure 11] FIG. 10 is a diagram for explaining derivation of evaluation results in the third embodiment. [Figure 12] FIG. 10 is a diagram for explaining derivation of evaluation results in the fourth embodiment. [Figure 13] FIG. 13 is a diagram for explaining generation of training data used to build a trained model in the fifth embodiment. [Figure 14] FIG. 13 is a diagram for explaining learning in the fifth embodiment. [Figure 15] FIG. 13 is a diagram for explaining derivation of evaluation results in the fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0024] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. First, the configuration of a medical information system to which an image processing device according to a first embodiment is applied will be described. Fig. 1 is a diagram showing a schematic configuration of the medical information system. In the medical information system shown in Fig. 1, a computer 1 incorporating the image processing device according to the first embodiment, a three-dimensional image capturing device 2, a fluoroscopic image capturing device 3, and an image storage server 4 are connected in a communicable state via a network 5.

[0025] The computer 1 includes an image processing device according to the first embodiment and has the image processing program of the first embodiment installed. The computer 1 is installed in a treatment room where a treatment is performed on a subject, as will be described later. The computer 1 may be a workstation or personal computer operated directly by a medical professional performing the treatment, or may be a server computer connected to either of these via a network. The image processing program is stored in a storage device of a server computer connected to the network or in network storage in an externally accessible state, and is downloaded and installed into the computer 1 used by a doctor upon request. Alternatively, the image processing program may be recorded and distributed on a recording medium such as a DVD (Digital Versatile Disc) or a CD-ROM (Compact Disc Read Only Memory), and then installed into the computer 1 from the recording medium.

[0026] The three-dimensional imaging device 2 is a device that captures an image of a diagnostic target region of the subject H to generate a three-dimensional image representing the region, and specifically, is a CT device, an MRI device, a PET (Positron Emission Tomography) device, or the like. The three-dimensional image, consisting of a plurality of tomographic images, generated by the three-dimensional imaging device 2 is transmitted to and stored in the image storage server 4. In this embodiment, the treatment target region of the subject H is the lungs, and the three-dimensional imaging device 2 is a CT device. As will be described later, before treatment on the subject H, an image of the subject H's chest is captured to obtain a CT image including the subject H's chest as a three-dimensional image in advance, and the CT image is stored in the image storage server 4.

[0027] The fluoroscopic imaging device 3 has a C-arm 3A, an X-ray source 3B, and an X-ray detector 3C. The X-ray source 3B and the X-ray detector 3C are attached to both ends of the C-arm 3A, respectively. In the fluoroscopic imaging device 3, the C-arm 3A is configured to be rotatable and movable so that the subject H can be imaged from any direction. As will be described later, during a treatment of the subject H, the fluoroscopic imaging device 3 performs fluoroscopic imaging in which the subject H is continuously irradiated with X-rays at a predetermined frame rate and the X-rays that have passed through the subject H are sequentially detected by the X-ray detector 3C, thereby sequentially acquiring X-ray images of the subject H. In the following description, the sequentially acquired X-ray images will be referred to as fluoroscopic images. A fluoroscopic image is an example of a two-dimensional image according to the present disclosure.

[0028] The image storage server 4 is a computer that stores and manages various data and is equipped with a large-capacity external storage device and database management software. The image storage server 4 communicates with other devices via a wired or wireless network 5, sending and receiving image data and the like. Specifically, it acquires various data via the network, including image data of 3D images acquired by the 3D imaging device 2 and fluoroscopic images acquired by the fluoroscopic imaging device 3, and stores and manages the data on a recording medium such as a large-capacity external storage device. The storage format of the image data and communication between devices via the network 5 are based on protocols such as DICOM (Digital Imaging and Communication in Medicine).

[0029] In this embodiment, while fluoroscopic imaging of the subject H is being performed, a biopsy procedure is performed in which a portion of a lesion, such as a pulmonary nodule, present in the lungs of the subject H is excised and the presence of the disease is examined in detail. For this purpose, the fluoroscopic imaging device 3 is arranged in a treatment room for performing the biopsy. An ultrasound endoscope device 6 is also installed in the treatment room. The ultrasound endoscope device 6 is equipped with an endoscope 6A having an ultrasound probe and treatment tools such as forceps attached to its tip. In this embodiment, to perform a biopsy of the lesion, the operator inserts the endoscope 6A into the bronchi of the subject H, captures fluoroscopic images of the subject H using the fluoroscopic imaging device 3, and while displaying the captured fluoroscopic images in real time, confirms the position of the tip of the endoscope 6A within the subject H in the fluoroscopic images and moves the tip of the endoscope 6A to the location of the target lesion.

[0030] Since lung lesions such as pulmonary nodules occur outside the bronchi, not inside, the operator moves the tip of the endoscope 6A to the target position, then uses the ultrasound probe to capture an ultrasound image of the outside of the bronchi, displays the ultrasound image, and performs a procedure to extract part of the lesion using a treatment tool such as forceps while confirming the location of the lesion on the ultrasound image.

[0031] Next, an image processing device according to a first embodiment will be described. Fig. 2 is a diagram showing the hardware configuration of the image processing device according to the first embodiment. As shown in Fig. 2, the image processing device 10 includes a CPU (Central Processing Unit) 11, non-volatile storage 13, and memory 16 as a temporary storage area. The image processing device 10 also includes a display 14 such as a liquid crystal display, input devices 15 such as a keyboard and a mouse, and a network I / F (Interface) 17 connected to a network 5. The CPU 11, storage 13, display 14, input devices 15, memory 16, and network I / F 17 are connected to a bus 18. The CPU 11 is an example of a processor in the present disclosure.

[0032] The storage 13 is realized by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. The storage 13 as a storage medium stores an image processing program 12. The CPU 11 reads the image processing program 12 from the storage 13, loads it into the memory 16, and executes the loaded image processing program 12.

[0033] Next, the functional configuration of the image processing device according to the first embodiment will be described. FIG. 3 is a diagram showing the functional configuration of the image processing device according to the first embodiment. FIG. 4 is a diagram showing a schematic diagram of the processing performed by the image processing device according to the first embodiment. As shown in FIG. 3, the image processing device 10 includes an image acquisition unit 21, a target position identification unit 22, an alignment unit 23, a first derivation unit 24, a second derivation unit 25, and a display control unit 26. When the CPU 11 executes the image processing program 12, the CPU 11 functions as the target position identification unit 22, the alignment unit 23, the first derivation unit 24, the second derivation unit 25, and the display control unit 26.

[0034] The image acquisition unit 21 acquires a three-dimensional image V0 of the subject H from the image storage server 4 in response to an instruction from an operator via the input device 15. The image acquisition unit 21 also sequentially acquires fluoroscopic images T0 acquired by the fluoroscopic imaging device 3 during treatment of the subject H.

[0035] The target position identifying unit 22 detects a lesion from the three-dimensional image V0 using a known computer-aided diagnosis (CAD) algorithm. Then, it identifies the center of gravity or the like of the detected lesion as the target position S0. As a detection method using CAD, a method using a machine learning model that has been machine-learned to detect a lesion can be used. Note that the method of detecting a lesion by the target position identifying unit 22 is not limited to this, and any method such as template matching can be used. Furthermore, the target position identifying unit 22 may identify a position specified by the operator using the input device 15 in the displayed three-dimensional image V0 as the target position.

[0036] The alignment unit 23 aligns the sequentially acquired perspective images T0 and the three-dimensional image V0. Here, the perspective images T0 are two-dimensional images. Therefore, the alignment unit 23 aligns the two-dimensional images and the three-dimensional images. In this embodiment, the alignment unit 23 first projects the three-dimensional image V0 in the same direction as the imaging direction of the perspective images T0 to derive a two-dimensional pseudo perspective image VT0. Then, the alignment unit 23 aligns the two-dimensional pseudo perspective image VT0 and the perspective images T0. During alignment, the alignment unit 23 derives the amount and direction of relative deformation between each pixel of the perspective images T0 and each pixel of the two-dimensional pseudo perspective image VT0 as alignment errors. Furthermore, through alignment, the alignment unit 23 derives a corresponding target position S1 in the perspective images T0 that corresponds to the target position S0 in the three-dimensional image V0 identified by the target position identification unit 22.

[0037] Any method can be used to align 2D and 3D images, such as those described in "Markelj, Primoz, et al. "A review of 3D / 2D registration methods for image-guided interventions." Medical image analysis 16.3 (2012): 642-661" and "Toth, et al., "3D / 2D model-to-image registration by imitation learning for cardiac procedures.", IJCARS, 2018."

[0038] In the first embodiment, the evaluation map M0 is two-dimensional. The first derivation unit 24 derives an evaluation result indicating the reliability of the registration. In this embodiment, the first derivation unit 24 derives, as the evaluation result, an evaluation map M0 having, as an evaluation value, a pixel value of each pixel corresponding to the amount of deformation included in the registration error derived by the registration unit 23. Note that the evaluation map M0 may be two-dimensional or three-dimensional, but in the first embodiment, the two-dimensional evaluation map M0 is derived as the evaluation result. Each pixel of the two-dimensional evaluation map M0 corresponds to each pixel of the perspective image T0. Meanwhile, each pixel of the three-dimensional evaluation map M0 corresponds to each pixel of the three-dimensional image V0.

[0039] Here, if the alignment by the alignment unit 23 is performed with high accuracy, the amount of deformation derived by the alignment will be small, and therefore the evaluation value will be small. On the other hand, if the alignment accuracy is low, the amount of deformation will be large, and therefore the evaluation value will be large. Therefore, the smaller the evaluation value in the evaluation map M0, the higher the reliability of the alignment. Note that the first derivation unit 24 may derive only the evaluation value at the target position S0 of the three-dimensional image V0 as the evaluation result.

[0040] The second derivation unit 25 derives a range in the perspective image T0 in which the corresponding target position S1 may exist, based on the alignment result by the alignment unit 23 and the evaluation map M0 derived by the first derivation unit 24. Specifically, the second derivation unit 25 references the evaluation value of the corresponding target position S1 in the evaluation map M0 and derives a range A0 in the three-dimensional image V0 in which the target position S0 may exist, the range A0 having a size corresponding to the evaluation value. In this case, the smaller the evaluation value, the smaller the size of the range A0 in which the target position S0 may exist. For example, when the evaluation value is small, the size of the range A0 is small as shown in FIG. 5, whereas when the evaluation value is large, the size of the range A0 is large as shown in FIG. 6.

[0041] As shown in Fig. 5 or 6, the display control unit 26 displays the perspective image T0 on the display 14, on which the range A0 derived by the second derivation unit 25 is superimposed. The range A0 may be a circular area having a radius corresponding to the evaluation value, but is not limited to this. In Fig. 5 and 6, the corresponding target position S1 is also superimposed on the perspective image T0 in addition to the range A0, but this is not limited to this. Only the range A0 may be superimposed on the perspective image T0 without displaying the corresponding target position S1.

[0042] Next, the processing performed in the first embodiment will be described. Fig. 7 is a flowchart showing the processing performed in the first embodiment. First, the image acquisition unit 21 acquires a three-dimensional image V0 from the image storage server 4 (step ST1), and the target position identification unit 22 identifies a target position S0 in the three-dimensional image V0 (step ST2). Next, the image acquisition unit 21 acquires a perspective image T0 (step ST3), and the alignment unit 23 aligns the three-dimensional image V0 with the perspective image T0 (step ST4).

[0043] Next, the first derivation unit 24 derives an evaluation map M0 indicating the reliability of the registration as an evaluation result (step ST5), and the second derivation unit 25 derives a range A0 in the perspective image T0, in which the target position S0 in the three-dimensional image V0 may exist, having a size corresponding to the evaluation value of the corresponding target position S1 in the evaluation map M0 (step ST6). Then, the display control unit 26 displays the perspective image T0 with the range A0 superimposed on the display 14 (step ST7), and returns to step ST3. Thereby, the processes of steps ST4 to ST7 are performed on the sequentially acquired perspective images T0.

[0044] 8 is a diagram showing fluoroscopic images T0 sequentially acquired in the first embodiment. In this embodiment, diseased tissue is sampled for biopsy using the ultrasound endoscopic device 6. To do this, the operator moves the tip of the endoscope 6A toward the range A0 while watching the fluoroscopic images T0 displayed on the display 14. As a result, the tip of the endoscope 6A gradually moves toward the range A0 in the sequentially acquired fluoroscopic images T0.

[0045] In this embodiment, the perspective image T0 is displayed with an area A0 having a size according to the evaluation result superimposed thereon, so that the accuracy of alignment between the perspective image T0 and the three-dimensional image V0 can be confirmed based on the size of the area A0.

[0046] Here, if the size of range A0 is small, it is possible to sample the diseased tissue by bringing the tip of endoscope 6A into range A0 without taking an ultrasound image with the ultrasound probe. On the other hand, if range A0 is large, it is unclear where in range A0 the lesion is located, so an ultrasound image of the vicinity of range A0 is acquired with the ultrasound probe and displayed on display 14, and the location of the lesion can be confirmed using the displayed ultrasound image before sampled.

[0047] Therefore, according to this embodiment, it is possible to determine whether or not to use an ultrasonic probe based on the size of the area A0 and sample the lesion.

[0048] In the first embodiment, the evaluation map M0 is two-dimensional, but is not limited to this. The three-dimensional evaluation map M0 may be derived by deriving, as a registration error, the amount and direction of deformation for each pixel of the three-dimensional image V0 relative to each corresponding pixel of the perspective image T0. In this case, the second derivation unit 25 may refer to the evaluation value of the target position S0 in the three-dimensional evaluation map M0 and derive a range A0 in which the target position may exist, having a size corresponding to the evaluation value.

[0049] Next, a second embodiment of the present disclosure will be described. Note that the functional configuration of the image processing device according to the second embodiment is the same as the functional configuration of the image processing device according to the first embodiment shown in Fig. 3, so a detailed description of the configuration will be omitted here. The image processing device according to the second embodiment differs from the first embodiment in the method of deriving the evaluation result in the first derivation unit 24.

[0050] In the second embodiment, the first derivation unit 24 derives a standard deformation amount between the first time phase and the second time phase of an organ including a target position before a treatment on the subject H, based on a plurality of 3D image groups including a 3D image at a first time phase and a 3D image at a second time phase. In this embodiment, a treatment is performed to collect lung tissue from a lung lesion as a biopsy. For this purpose, the organ including the target position is the lung, the 3D image at the first time phase is a 3D image at an inhalation phase, and the 3D image at the second time phase is a 3D image at an exhalation phase. The plurality of 3D image groups includes a 3D image group of a subject H other than the subject H to be biopsyed. The plurality of 3D image groups are stored in the image storage server 4, and are acquired by the image acquisition unit 21 and stored in the storage 13 before processing by the first derivation unit 24.

[0051] FIG. 9 is a diagram for explaining the derivation of the deformation amount in the second embodiment. The first derivation unit 24 performs non-rigid registration between a three-dimensional image V1 of a first time phase, i.e., the inhalation phase, and a three-dimensional image V2 of a second time phase, i.e., the exhalation phase, and derives the deformation amount and deformation direction, i.e., the deformation vector, between corresponding pixels in the three-dimensional images V1 and V2. In FIG. 9, the deformation vectors at each pixel are indicated by multiple arrows. As a result, a deformation vector field H0 is acquired for each pixel position in the three-dimensional images V1 and V2.

[0052] The first derivation unit 24 derives such a deformation vector field H0 for multiple 3D image groups to derive a standard lung deformation amount during inspiration and expiration. To this end, the first derivation unit 24 extracts lung regions from each of the multiple 3D image groups and derives the average shape of the extracted lungs as a standard lung shape. Furthermore, the first derivation unit 24 derives statistical values ​​of the deformation vector field for each pixel within the derived standard lung as a standard deformation vector field SH0. Note that the statistical values ​​may include, but are not limited to, the mean, median, and variance.

[0053] Here, in the standard deformation vector field SH0, the accuracy of registration between the perspective image T0 and the three-dimensional image V0 is low at positions in the lung where the amount of deformation is large, whereas the accuracy of registration between the perspective image T0 and the three-dimensional image V0 is high at positions in the lung where the amount of deformation is small.

[0054] 10 , in the second embodiment, the first derivation unit 24 derives the evaluation map M0 by projecting the standard deformation vector field SH0 onto a three-dimensional image V0 of the subject H. Note that the evaluation map M0 is three-dimensional in the second embodiment. Therefore, in the second embodiment, the second derivation unit 25 derives a two-dimensional evaluation map by projecting the evaluation map M0 in the imaging direction of the perspective image T0, and derives a range A0 in which the target position may exist, having a size corresponding to the evaluation value, by referring to the evaluation value of the corresponding target position S1 in the two-dimensional evaluation map.

[0055] By deriving the evaluation map M0 in this manner, as in the first embodiment, the accuracy of the alignment between the perspective image T0 and the three-dimensional image V0 can be confirmed depending on the size of the range A0 superimposed on the perspective image T0.

[0056] Next, a third embodiment of the present disclosure will be described. Note that the functional configuration of the image processing device according to the third embodiment is the same as the functional configuration of the image processing device according to the first embodiment shown in Fig. 3, so a detailed description of the configuration will be omitted here. The image processing device according to the third embodiment differs from the first embodiment in the method of deriving the evaluation result in the first derivation unit 24.

[0057] FIG. 11 is a diagram illustrating the derivation of evaluation results in the third embodiment. As shown in FIG. 11 , in the third embodiment, the first derivation unit 24 derives a pseudo-perspective image VT1 that simulates lung movement from a three-dimensional image V0. Chest CT images are often captured while holding the breath in an inhalation state. In this embodiment, the three-dimensional image V0 is also assumed to be an inhalation-phase three-dimensional image V0 acquired by capturing the image while holding the breath in an inhalation state. Therefore, the first derivation unit 24 first transforms the inhalation-phase three-dimensional image V0 to derive an exhalation-phase three-dimensional image V3. The first derivation unit 24 then projects the derived exhalation-phase three-dimensional image V3 in a predetermined direction to derive a pseudo-perspective image VT1. Therefore, the pseudo-perspective image VT1 represents a pseudo two-dimensional exhalation-phase perspective image.

[0058] Alternatively, the three-dimensional image V0 may be projected in a predetermined direction to derive a two-dimensional image first, and then the derived two-dimensional image may be transformed to derive the pseudo perspective image VT1 of the expiratory phase.

[0059] Next, the first derivation unit 24 aligns the pseudo perspective image VT1 of the expiratory phase with the three-dimensional image V0 of the inhalation phase. The alignment may be performed in the same manner as the alignment unit 23 in the first embodiment. As a result, the first derivation unit 24 derives, as alignment errors, the amount and direction of deformation for each pixel of the two-dimensional pseudo perspective image VT1 relative to the corresponding pixel of the three-dimensional image V0. The first derivation unit 24 then derives an evaluation map M0 based on the derived alignment errors. Note that the evaluation map M0 may be derived only for the corresponding target position S1 in the pseudo perspective image VT1.

[0060] In the third embodiment, the evaluation map M0 is two-dimensional, but is not limited to this. The three-dimensional evaluation map M0 may be derived by deriving, as a registration error, the amount and direction of deformation for each pixel of the three-dimensional image V0 relative to each corresponding pixel of the two-dimensional pseudo perspective image VT1. In this case, the second derivation unit 25 may refer to the evaluation value of the target position S0 in the three-dimensional evaluation map M0 and derive a range A0 in which the target position may exist, having a size corresponding to the evaluation value.

[0061] By deriving the evaluation map M0 in this manner, as in the first embodiment, the accuracy of the alignment between the perspective image T0 and the three-dimensional image V0 can be confirmed depending on the size of the range A0 superimposed on the perspective image T0.

[0062] Next, a fourth embodiment of the present disclosure will be described. Note that the functional configuration of the image processing device according to the fourth embodiment is the same as the functional configuration of the image processing device according to the first embodiment shown in Fig. 3, and therefore a detailed description of the configuration will be omitted here. The image processing device according to the fourth embodiment differs from the first embodiment in the method of deriving the evaluation result in the first derivation unit 24.

[0063] Fig. 12 is a diagram for explaining derivation of an evaluation result in the fourth embodiment. As shown in Fig. 12, in the fourth embodiment, the first derivation unit 24 optimizes the registration. Specifically, the registration is optimized by repeatedly performing non-rigid registration between the three-dimensional image V0 and the perspective image T0.

[0064] The first derivation unit 24 then derives the similarity between the three-dimensional image V0 and the perspective image T0 each time alignment is performed, and derives the relationship R0 between the number of alignments and the similarity. In this case, the similarity can be calculated by, for example, summing up the correlation values ​​between all pixels of the perspective image T0 and corresponding pixels of the three-dimensional image V0. Therefore, the smaller the similarity value, the more accurately the alignment is performed.

[0065] FIG. 12 shows the relationship R0 between the number of alignments and the similarity. In the relationship R0, the similarity repeatedly increases and decreases as the number of alignments increases. The first derivation unit 24 identifies a minimum point of the similarity in the relationship R0 and derives the similarity at the minimum point as a local solution. The first derivation unit 24 then derives the registration error between the perspective image T0 and the three-dimensional image V0 when each of the multiple local solutions is derived. Here, the registration error is the amount and direction of deformation for each pixel of the perspective image T0 relative to the corresponding pixel of the three-dimensional image V0 after alignment. The first derivation unit 24 derives the registration error for each pixel of the three-dimensional image V0 for the number of derived local solutions. In the relationship R0 shown in FIG. 12, three local solutions R1 to R3 are derived, and therefore three registration errors E1 to E3 are derived for each pixel of the perspective image T0.

[0066] Next, the first derivation unit 24 derives, as an evaluation result, statistics of the registration errors E1 to E3 for each pixel of the perspective image T0. Examples of the statistics that can be used include, but are not limited to, the mean, median, and variance of the registration errors. Furthermore, the first derivation unit 24 derives an evaluation map M0 based on the registration error statistics for each pixel of the three-dimensional image V0. The evaluation map M0 represents the distribution of the registration error statistics in the perspective image T0.

[0067] In the fourth embodiment, the evaluation map M0 is two-dimensional, but is not limited to this. The three-dimensional evaluation map M0 may be derived by deriving, as a registration error, the amount and direction of deformation for each pixel of the three-dimensional image V0 relative to each corresponding pixel of the perspective image T0. In this case, the second derivation unit 25 may refer to the evaluation value of the target position S0 in the three-dimensional evaluation map M0 and derive a range A0 in which the target position may exist, having a size corresponding to the evaluation value.

[0068] By deriving the evaluation map M0 in this manner, as in the first embodiment, the accuracy of the alignment between the perspective image T0 and the three-dimensional image V0 can be confirmed depending on the size of the range A0 superimposed on the perspective image T0.

[0069] In the fourth embodiment, the first derivation unit 24 may derive the relationship R0 between the number of alignments and the similarity for only the target position in the three-dimensional image V0, and derive only the evaluation result for the target position in the three-dimensional image V0.

[0070] Next, a fifth embodiment of the present disclosure will be described. Note that the functional configuration of an image processing device according to the fifth embodiment is the same as the functional configuration of the image processing device according to the first embodiment shown in FIG. 3, and therefore a detailed description of the configuration will be omitted here. The image processing device according to the fifth embodiment differs from the first embodiment in the method used to derive the evaluation result in the first derivation unit 24. Specifically, in the fifth embodiment, the evaluation result, i.e., the evaluation map M0, is derived using the method described in "Eppenhof, et al., "Error estimation of deformable image registration of pulmonary CT scans using convolutional neural networks," Journal of Medical Imaging, 2018."

[0071] The method described in the paper by Eppenhof et al. is a method of deriving two deformed CT images by applying different deformation fields to the CT images, deriving the pixel differences between the two deformed CT images as an error map, and using the two deformed CT images as training data and the error map as correct answer data as training data, training a neural network so that when two CT images are input, it outputs an error map of the two CT images, thereby constructing a trained model.

[0072] In the fifth embodiment, the method of Eppenhof et al. is applied to two-dimensional projection images to derive an error map between a three-dimensional image V0 and a perspective image T0 as an evaluation map M0. In the fifth embodiment, training data is prepared for constructing a trained model. FIG. 13 is a diagram for explaining the generation of training data used to construct a trained model in the fifth embodiment. As shown in FIG. 13, a reference three-dimensional image Vm0 is prepared, and two different deformation fields are applied to the three-dimensional image Vm0 to derive a first deformed three-dimensional image Vm1 and a second deformed three-dimensional image Vm2. Then, the first deformed three-dimensional image Vm1 and the second three-dimensional image Vm2 are projected two-dimensionally, respectively, to derive a first deformed pseudo-projection image Tm1 and a second deformed pseudo-projection image Tm2.

[0073] Then, by aligning the first deformed pseudo projection image Tm1 with the second deformed pseudo projection image Tm2, ​​the relative amount and direction of deformation for each pixel between the first deformed pseudo projection image Tm1 and the second deformed pseudo projection image Tm2 are derived as an error map Em0. This generates training data using the first deformed pseudo projection image Tm1 and the second deformed pseudo projection image Tm2 as training data and the error map Em0 as correct answer data. The error map Em0 is a map that two-dimensionally represents the distribution of errors.

[0074] FIG. 14 is a diagram illustrating learning in the fifth embodiment. As shown in FIG. 14, a first deformed pseudo-projection image Tm1 and a second deformed pseudo-projection image Tm2 are input to the network 30 to be trained, and an error map Es is derived. The difference between the error map Es and the error map Em0, which is the correct data, is derived as a loss L0, and the network 30 is trained to reduce the loss L0. Learning of the network 30 is then repeated until the loss L0 reaches a predetermined threshold value or until a predetermined number of learning rounds have been completed, thereby constructing a trained model. The trained model constructed in this manner is applied to the first derivation unit 24. The error map derived by the trained model is an evaluation map M0, and the value corresponding to each pixel of the pseudo-projection image in the evaluation map M0 becomes the evaluation value.

[0075] FIG. 15 is a diagram for explaining derivation of evaluation results in the fifth embodiment. As shown in FIG. 15, in the fifth embodiment, the first derivation unit 24 applies the trained model 31 constructed as described above. First, as in the first embodiment, the first derivation unit 24 projects a three-dimensional image V0 in the same direction as the capture direction of the perspective image T0 to derive a two-dimensional pseudo perspective image VT0. Then, the pseudo perspective image VT0 and the perspective image T0 are input to the trained model 31, which outputs an error map, i.e., an evaluation map M0.

[0076] In the fifth embodiment, the second derivation unit 25 refers to the evaluation value of the corresponding target position S1 corresponding to the target position S0 in the evaluation map M0 derived as described above, and derives a range A0 in which the target position may exist, having a size corresponding to the evaluation value, as in the first embodiment.

[0077] By deriving the evaluation map M0 in this manner, as in the first embodiment, the accuracy of the alignment between the perspective image T0 and the three-dimensional image V0 can be confirmed depending on the size of the range A0 superimposed on the perspective image T0.

[0078] In the above embodiments, the technology of the present disclosure is applied to aligning a three-dimensional image such as a CT image with a perspective image T0, but the present disclosure is not limited to this. For example, the technology of the present disclosure can also be applied to aligning a three-dimensional image with an endoscopic image captured by an endoscope inserted into a lumen of the human body. In this case, the endoscopic image is an example of a two-dimensional image of the present disclosure.

[0079] In addition, in each of the above embodiments, a lesion in the lung is used as the target position, but this is not limited thereto. For example, a branching position of the bronchus may be used as the target position. Furthermore, the target site is not limited to the lung, and any organ containing a lesion or the like to be treated can be used as the target.

[0080] Furthermore, in each of the above embodiments, the following various processors can be used as the hardware structure of a processing unit that executes various processes, such as the image acquisition unit 21, the target position identification unit 22, the alignment unit 23, the first derivation unit 24, the second derivation unit 25, and the display control unit 26. As described above, the various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as dedicated electrical circuits that are processors having a circuit configuration specifically designed to execute specific processes, such as a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and an ASIC (Application Specific Integrated Circuit).

[0081] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor.

[0082] Examples of configuring multiple processing units with a single processor include, first, a form in which one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units, as typified by computers such as client and server. Second, a form in which a processor is used to realize the functions of an entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by systems on chips (SoCs). In this way, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0083] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. [Explanation of symbols]

[0084] 1. Computer 2. 3D imaging device 3. Fluoroscopic imaging device 3A Arm 3B X-ray source 3C X-ray detector 4. Image storage server 5. Network 6. Endoscopic ultrasound equipment 6A Endoscope 10 Image processing device 11 CPU 12 Image Processing Programs 13. Storage 14 Display 15 Input Devices 16 memory 21 Image acquisition unit 22 Target position identification section 23 Alignment section 24 First derivation part 25 Second derivation part 26 Display control unit 30 Network 31 Pre-trained models A0 range E1~E3 Alignment error Em0,Es error map H0 deformation vector field M0 Evaluation Map R0 Relationship between the number of alignments and similarity R1~R3 ​​local solutions S0 target position S1 Corresponding target position SH0 standard deformation vector field T0 fluoroscopic image Tm1, Tm2 Deformed pseudo-projection images V0, Vm0 3D image V3 3D image of expiratory phase Vm1, Vm2 Deformed 3D images VT0,VT1 pseudo fluoroscopic image

Claims

1. at least one processor; The processor: Identifying a target position in a three-dimensional image obtained by photographing the subject before treatment; deriving a corresponding target position corresponding to the target position in the two-dimensional image by registering two-dimensional images sequentially acquired of the subject under treatment with the three-dimensional image; deriving an evaluation result representing the reliability of the registration at least at the target position of the three-dimensional image; deriving a range in the two-dimensional image that has a smaller size as the reliability represented by the evaluation result becomes higher as a range in the two-dimensional image where the corresponding target position may exist; An image processing device that displays the two-dimensional image with the range superimposed thereon.

2. The image processing device according to claim 1 , wherein the processor displays the corresponding target position superimposed on the two-dimensional image.

3. 3. The image processing device according to claim 1, wherein the processor derives the evaluation result by projecting a standard deformation amount between the first time phase and the second time phase of an organ including the target position, the standard deformation amount being derived in advance based on a group of multiple three-dimensional images including a three-dimensional image at a first time phase and a three-dimensional image at a second time phase, onto the three-dimensional image of the subject.

4. 4. The image processing apparatus according to claim 3, wherein the three-dimensional image in the first time phase is a three-dimensional image in an inhalation phase, and the three-dimensional image in the second time phase is a three-dimensional image in an exhalation phase.

5. the processor derives a pseudo two-dimensional image that simulates the movement of an organ including the target position from the acquired three-dimensional image; deriving a registration error representing a relative amount and direction of deformation between the pseudo two-dimensional image and the acquired three-dimensional image by aligning the pseudo two-dimensional image with the acquired three-dimensional image; The image processing device according to claim 1 , wherein the evaluation result is derived based on the registration error.

6. the processor repeatedly aligns the two-dimensional image with the three-dimensional image; deriving a relationship between the number of times the alignment is performed and a similarity between the aligned two-dimensional image and the aligned three-dimensional image, the similarity being derived each time the alignment is performed; deriving a plurality of local solutions in the relationship; The image processing apparatus according to claim 1 , wherein the evaluation result is derived based on statistics of registration errors between the two-dimensional image and the three-dimensional image when each of the plurality of local solutions is derived.

7. 3. The image processing device according to claim 1, wherein the processor derives the evaluation result using a trained model that has undergone machine learning to output the evaluation result when the processor receives a pseudo-two-dimensional image obtained by projecting the three-dimensional image in the direction in which the two-dimensional image was captured and the two-dimensional image.

8. 8. The image processing device according to claim 1, wherein the three-dimensional image is a CT image. 。

9. The image processing device according to claim 1 , wherein the target position is included in the lung of the subject.

10. The image processing device according to claim 9 , wherein the target position is a position where a lung lesion exists.

11. The image processing device according to claim 9 , wherein the target position is a branching position in a bronchus.

12. A computer identifies a target position in a three-dimensional image obtained by photographing a subject before treatment, deriving a corresponding target position corresponding to the target position in the two-dimensional image by registering two-dimensional images sequentially acquired of the subject under treatment with the three-dimensional image; deriving an evaluation result representing the reliability of the registration at least at the target position of the three-dimensional image; deriving a range in the two-dimensional image that has a smaller size as the reliability represented by the evaluation result becomes higher as a range in the two-dimensional image where the corresponding target position may exist; An image processing method for displaying the two-dimensional image on which the range is superimposed.

13. a step of identifying a target position in a three-dimensional image acquired by photographing a subject before treatment; a step of deriving a corresponding target position corresponding to the target position in the two-dimensional image by registering two-dimensional images sequentially acquired of the subject under treatment with the three-dimensional image; deriving an evaluation result representing the reliability of the registration at least at the target position of the three-dimensional image; a step of deriving a range in the two-dimensional image, the smaller the size of which is the higher the reliability represented by the evaluation result, as a range in the two-dimensional image where the corresponding target position may exist; and a procedure for displaying the two-dimensional image on which the range is superimposed.

Images