Image processing apparatus, method, and program

Abstract

To provide an image processing device, method, and program capable of speedily and accurately positioning a three-dimensional image and a two-dimensional radiation image.SOLUTION: A processor acquires a three-dimensional image of a subject, successively acquires a plurality of radiation images of the subject, performs first positioning for a reference radiation image among the radiation images and the three-dimensional image, deforms the three-dimensional image based on a result of the first positioning to derive a first deformed three-dimensional image, displays the first deformed three-dimensional image superimposed on the reference radiation image, and performs second positioning at calculation costs less than those of the first positioning, for the first deformed three-dimensional image and a new radiation image acquired after the reference radiation image.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present disclosure relates to an image processing apparatus, method, and program.

Background Art

[0002] An ultrasonic endoscope having an endoscopic observation unit and an ultrasonic observation unit at its tip is inserted into a lumen such as the digestive organ or bronchus of a subject, and endoscopic images within the lumen and ultrasonic images of sites such as lesions outside the lumen wall are captured. In addition, a biopsy is also performed to collect the tissue of a lesion outside the lumen wall using a treatment tool such as forceps.

[0003] When performing a procedure using such an ultrasonic endoscope, it is important to accurately reach the target position within the subject with the ultrasonic endoscope. For this reason, fluoroscopic imaging is performed in which radiation is continuously irradiated from a radiation source to the subject during the procedure, and the fluoroscopic image thus obtained is displayed in real time, thereby grasping the positional relationship between the ultrasonic endoscope and the human body structure.

[0004] Here, since the fluoroscopic image includes the anatomical structures such as organs, blood vessels, and bones within the subject overlapping each other, it is not easy to recognize the lumen and the lesion. For this reason, a three-dimensional image of the subject is acquired in advance before the procedure using a CT (Computed Tomography) apparatus, an MRI (Magnetic Resonance Imaging) apparatus, etc., the lesion position is specified in the three-dimensional image, and the lesion position is specified in the fluoroscopic image by performing alignment between the three-dimensional image and the fluoroscopic image (see, for example, Patent Documents 1 and 2).

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

[0006] Rigid body alignment requires less computation and can be performed relatively quickly, but its alignment accuracy is not very high. On the other hand, non-rigid body alignment offers high alignment accuracy but requires more computation and therefore takes more time to process.

[0007] This invention has been made in view of the above circumstances, and aims to enable high-speed and high-precision alignment of three-dimensional images and two-dimensional radiation images. [Means for solving the problem]

[0008] An image processing apparatus according to a first aspect of the present disclosure comprises at least one processor, the processor acquires a three-dimensional image of a subject, Multiple radiographic images of the subject are acquired sequentially. A first alignment is performed on the reference radiographic image and the 3D image from among multiple radiographic images. Based on the result of the first alignment, the 3D image is deformed to derive the first deformed 3D image. The first deformed 3D image is superimposed onto the reference radiation image. A second alignment is performed on the first deformed 3D image and a new radiation image acquired after the reference radiation image, with the second alignment being computationally less costly than the first alignment.

[0009] An image processing apparatus according to a second aspect of the present disclosure, in an image processing apparatus according to a first aspect of the present disclosure, the processor deforms the first deformed 3D image based on the result of the second alignment to derive a second deformed 3D image, We derived the alignment error between the second deformed 3D image and the new radiographic image. Determine whether the error is below a predetermined threshold. If the determination is affirmative, the second deformed 3D image may be superimposed onto the new radiographic image.

[0010] An image processing apparatus according to a third aspect of the present disclosure, in an image processing apparatus according to a second aspect of the present disclosure, if the determination is denied, the processor updates the reference radiation image to a new radiation image and performs a new first alignment with respect to the reference radiation image and the 3D image.

[0011] Based on the new first alignment result, the 3D image is deformed to derive a new first deformed 3D image. This may involve superimposing a new first deformed 3D image onto a new radiographic image.

[0012] The image processing apparatus according to the fourth aspect of the present disclosure may be an image processing apparatus according to the third aspect of the present disclosure in which the processor repeats the following each time a new radiographic image is acquired: second alignment, derivation of a second deformed 3D image, derivation of an error, superimposition of the second deformed 3D image onto the new radiographic image, determination, superimposition if the determination is affirmative, new first alignment if the determination is negative, derivation of a new first deformed 3D image, superimposition of the new first deformed 3D image, and updating of the reference radiographic image.

[0013] An image processing apparatus according to a fifth aspect of the present disclosure, in an image processing apparatus according to any one of the first to fourth aspects of the present disclosure, the reference radiographic image used for the first initial alignment may be a radiographic image of the respiratory phase that is closest to the respiratory phase of the three-dimensional image.

[0014] The image processing apparatus according to the sixth aspect of the present disclosure is an image processing apparatus according to any one of the first to fifth aspects of the present disclosure, wherein the three-dimensional image and the radiographic image include the chest of the subject. The first alignment includes rigid body alignment between a reference radiographic image and a 3D image, and non-rigid body alignment based on lung field regions extracted from the reference radiographic image and the 3D image, respectively. The second alignment may include non-rigid alignment based on the lung field region extracted from the first deformed three-dimensional image that has become two-dimensional during the first alignment and the lung field region extracted from a new radiographic image.

[0015] The image processing apparatus according to the seventh aspect of the present disclosure is the image processing apparatus according to the sixth aspect of the present disclosure, wherein the extraction of the lung field region from the new radiographic image in the second alignment may have a lower computational cost than the extraction of the lung field region from the reference radiographic image in the first alignment.

[0016] The image processing method according to the present disclosure acquires a three-dimensional image of a subject, sequentially acquires a plurality of radiographic images of the subject, performs a first alignment on a reference radiographic image and the three-dimensional image among the plurality of radiographic images, derives a first deformed three-dimensional image by deforming the three-dimensional image based on the result of the first alignment, superimposes and displays the first deformed three-dimensional image on the reference radiographic image, performs a second alignment, which has a lower computational cost than the first alignment, on the first deformed three-dimensional image and a new radiographic image acquired after the reference radiographic image.

[0017] The image processing program according to the present disclosure causes a computer to execute procedures for acquiring a three-dimensional image of a subject, sequentially acquiring a plurality of radiographic images of the subject, performing a first alignment on a reference radiographic image and the three-dimensional image among the plurality of radiographic images, deriving a first deformed three-dimensional image by deforming the three-dimensional image based on the result of the first alignment, superimposing and displaying the first deformed three-dimensional image on the reference radiographic image, and performing a second alignment, which has a lower computational cost than the first alignment, on the first deformed three-dimensional image and a new radiographic image acquired after the reference radiographic image.

Advantages of the Invention

[0018] According to the present disclosure, a three-dimensional image and a radiation image can be aligned quickly and accurately.

Brief Description of the Drawings

[0019] [Figure 1] FIG. showing a schematic configuration of a medical information system to which an image processing apparatus according to an embodiment of the present disclosure is applied [Figure 2] FIG. showing a schematic configuration of the image processing apparatus according to the present embodiment [Figure 3] Functional configuration diagram of the image processing apparatus according to the present embodiment [Figure 4] FIG. schematically showing the processing performed by the image processing apparatus according to the present embodiment [Figure 5] FIG. for explaining the detection of the difference in the position of the diaphragm [Figure 6] FIG. showing a display screen [Figure 7] Flowchart showing the processing performed in the present embodiment

Mode for Carrying Out the Invention

[0020] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. First, the configuration of a medical information system to which an image processing apparatus according to the present embodiment is applied will be described. FIG. 1 is a diagram showing a schematic configuration of the medical information system. The medical information system shown in FIG. 1 includes a computer 1 incorporating the image processing apparatus according to the present embodiment, a three-dimensional image capturing apparatus 2, a fluoroscopic image capturing apparatus 3, and an image storage server 4, which are connected in a communicable state via a network 5.

[0021] Computer 1 contains the image processing device according to this embodiment, and the image processing program of this embodiment is installed on it. Computer 1 is installed in the treatment room where treatment is performed on subject H, as described later. Computer 1 may be a workstation or personal computer directly operated by the medical professional performing the treatment, or it may be a server computer connected to them via a network. The image processing program is stored in a storage device of the server computer connected to the network, or in network storage, in a state that is accessible from the outside, and is downloaded and installed on computer 1 used by the physician as needed. Alternatively, it may be recorded on a recording medium such as a DVD (Digital Versatile Disc) or CD-ROM (Compact Disc Read Only Memory) and distributed, and then installed on computer 1 from that recording medium.

[0022] The 3D imaging device 2 is a device that generates a 3D image representing a specific area of ​​the subject H by imaging that area. Specifically, it is a CT scanner, an MRI scanner, or a PET (Positron Emission Tomography) scanner. The 3D image generated by the 3D imaging device 2, consisting of multiple tomographic images, is transmitted to the image storage server 4 and stored there. In this embodiment, the area of ​​the subject H to be treated is the lung, and the 3D imaging device 2 is a CT scanner. As will be described later, before treatment on the subject H, the chest of the subject H is imaged, and a CT image including the chest of the subject H is acquired in advance as a 3D image and stored in the image storage server 4.

[0023] The fluoroscopy imaging device 3 comprises a C-arm 3A, an X-ray source 3B, and an X-ray detector 3C. The X-ray source 3B and the X-ray detector 3C are attached to both ends of the C-arm 3A, respectively. In the fluoroscopy imaging device 3, the C-arm 3A is configured to be rotatable and movable so that the subject H can be photographed from any direction. Then, as will be described later, during treatment of the subject H, the fluoroscopy imaging device 3 continuously irradiates the subject H with X-rays at a predetermined frame rate and sequentially detects the X-rays that have passed through the subject H with the X-ray detector 3C, thereby sequentially acquiring X-ray images of the subject H. In the following description, the sequentially acquired X-ray images will be referred to as fluoroscopic images. A fluoroscopic image is an example of a radiographic image according to this disclosure.

[0024] The image storage server 4 is a computer that stores and manages various types of data, and is equipped with a large-capacity external storage device and database management software. The image storage server 4 communicates with other devices via a wired or wireless network 5 to send and receive image data, etc. Specifically, it acquires various types of data, including 3D images acquired by the 3D image acquisition device 2 and fluoroscopic images acquired by the fluoroscopic image acquisition device 3, via the network, and stores and manages them on a recording medium such as a large-capacity external storage device. The storage format of the image data and communication between each device via the network 5 are based on protocols such as DICOM (Digital Imaging and Communication in Medicine).

[0025] In this embodiment, while performing fluoroscopic imaging of subject H, a biopsy procedure is performed to examine the presence of disease in detail by taking a sample of a lesion such as a pulmonary nodule in the lung of subject H, which has been previously detected using a 3D image V0. For this reason, the fluoroscopic imaging device 3 is located in the treatment room where the procedure is performed. An endoscopic ultrasound device 6 is also installed in the treatment room. The endoscopic ultrasound device 6 is equipped with an endoscope 6A with an ultrasound probe and treatment instruments such as forceps attached to its tip. In this embodiment, in order to perform a biopsy of a lesion, the operator inserts the endoscope 6A into the bronchus of subject H, takes a fluoroscopic image of subject H using the fluoroscopic imaging device 3, and displays the captured fluoroscopic image and the endoscopic image taken by the endoscope 6A in real time. The operator confirms the position of the tip of the endoscope 6A within subject H in the fluoroscopic image and moves the tip of the endoscope 6A to the location of the target lesion.

[0026] Here, lung lesions such as pulmonary nodules occur outside the bronchi, not inside them. Therefore, after moving the tip of the endoscope 6A to the target position, the operator takes an ultrasound image of the outside of the bronchi using an ultrasound probe, displays the ultrasound image, and, while confirming the location of the lesion in the ultrasound image, performs a procedure to collect a portion of the lesion using instruments such as forceps.

[0027] Next, an image processing apparatus according to this embodiment will be described. Figure 2 is a diagram showing the hardware configuration of the image processing apparatus according to this embodiment. As shown in Figure 2, the image processing apparatus 10 includes a CPU (Central Processing Unit) 11, non-volatile storage 13, and memory 16 as a temporary storage area. The image processing apparatus 10 also includes a display 14 such as a liquid crystal display, input devices 15 such as a keyboard and mouse, and a network I / F (Interface) 17 connected to a network 5. The CPU 11, storage 13, display 14, input devices 15, memory 16, and network I / F 17 are connected to a bus 18. Note that the CPU 11 is an example of a processor in this disclosure.

[0028] The storage 13 is implemented using an HDD (Hard Disk Drive), SSD (Solid State Drive), flash memory, etc. The image processing program 12 is stored in the storage 13 as a storage medium. The CPU 11 reads the image processing program 12 from the storage 13, expands it into memory 16, and executes the expanded image processing program 12.

[0029] Next, the functional configuration of the image processing apparatus according to this embodiment will be described. Figure 3 is a diagram showing the functional configuration of the image processing apparatus according to this embodiment. Figure 4 is a diagram schematically showing the processing performed by the image processing apparatus according to this embodiment. As shown in Figure 3, the image processing apparatus 10 includes an image acquisition unit 21, a first alignment unit 22, a first derivation unit 23, a second alignment unit 24, a second derivation unit 25, an error derivation unit 26, a determination unit 27, and a display control unit 28. When the CPU 11 executes the image processing program 12, the CPU 11 functions as the image acquisition unit 21, the first alignment unit 22, the first derivation unit 23, the second alignment unit 24, the second derivation unit 25, the error derivation unit 26, the determination unit 27, and the display control unit 28.

[0030] The image acquisition unit 21 acquires a three-dimensional image V0 of the subject H from the image storage server 4 based on instructions from the operator via the input device 15. The image acquisition unit 21 also sequentially acquires fluoroscopic images T0 obtained by the fluoroscopic image acquisition device 3 during the treatment of the subject H.

[0031] The first alignment unit 22 performs a first alignment between the fluoroscopic image T0 and the three-dimensional image V0. Here, the three-dimensional image V0 is acquired by a CT scanner. When taking images with a CT scanner, images are taken while the patient is inhaling to make faint lesions easier to see. Therefore, the respiratory phase of the three-dimensional image V0 is the inspiratory phase. On the other hand, the fluoroscopic image T0 is acquired sequentially at a predetermined frame rate, but in order to improve the accuracy of the alignment, it is preferable that the respiratory phase of the fluoroscopic image T0 matches the respiratory phase of the three-dimensional image V0. Therefore, the first alignment unit 22 identifies the fluoroscopic image T0 in the inspiratory phase or a respiratory phase close to the inspiratory phase as the reference fluoroscopic image Tb. Then, the first alignment unit 22 performs a first alignment between the reference fluoroscopic image Tb and the three-dimensional image V0.

[0032] Here, the reference perspective image Tb is a two-dimensional image. Therefore, the first alignment unit 22 performs alignment between the two-dimensional image and the three-dimensional image. In this embodiment, the first alignment unit 22 first projects the three-dimensional image V0 in the same direction as the shooting direction of the reference perspective image Tb to derive a two-dimensional pseudo-perspective image VT0. Then, the first alignment unit 22 rigidly aligns the two-dimensional pseudo-perspective image VT0 with the perspective image T0. Any method can be used for rigid alignment, such as an affine transformation. For example, if an affine transformation is used, the first alignment unit 22 derives the amount of translation and rotation of the pseudo-perspective image VT0 relative to the perspective image T0 in order to match feature points such as the intersections of ribs contained in the pseudo-perspective image VT0 and the perspective image T0.

[0033] Furthermore, the first alignment unit 22 extracts the lung field region from the 3D image V0 and the reference fluoroscopic image Tb. The lung field region is the soft tissue of the lung included in the 3D image V0 and the reference fluoroscopic image Tb. In this embodiment, the first alignment unit 22 extracts the lung field region from the 3D image V0 and the reference fluoroscopic image Tb using a known computer-aided diagnosis (CAD) algorithm.

[0034] The first alignment unit 22 then performs non-rigid alignment between the 3D image V0 and the reference fluoroscopic image Tb. Specifically, it performs non-rigid alignment between the 3D image V0 and the reference fluoroscopic image Tb by deforming the 3D image V0 so that the lung field region extracted from the rigidly aligned 3D image V0 matches the lung field region extracted from the reference fluoroscopic image Tb, and derives the amount of deformation of the 3D image V0 relative to the reference fluoroscopic image Tb. In this case as well, the first alignment unit 22 projects the 3D image V0 in the same direction as the imaging direction of the reference fluoroscopic image Tb to derive a 2D pseudo-fluoroscopic image VT0, and performs non-rigid alignment based on the lung field region in the pseudo-fluoroscopic image VT0 and the lung field region in the reference fluoroscopic image Tb.

[0035] For non-rigid body alignment, one method can be used, but is not limited to this, which involves using functions such as B-splines and thin-plate splines to non-linearly transform the correspondence points between the lung field regions in the pseudo-fluoroscopic image VT0 and the lung field regions in the reference fluoroscopic image Tb, thereby deriving the deformation amount of the rigidly aligned pseudo-fluoroscopic image VT0 relative to the fluoroscopic image T0, and then expanding the derived deformation amount into three dimensions. Any method can be used, such as a method that performs non-rigid body alignment using a pre-trained model.

[0036] The first derivation unit 23 deforms the 3D image V0 based on the result of the first alignment to derive the first deformed 3D image V1. That is, it deforms the 3D image V0 based on the result of rigid body alignment, and then deforms the deformed 3D image V0 based on the result of non-rigid body alignment to derive the first deformed 3D image V1.

[0037] The second alignment unit 24 performs a second alignment on the first deformed 3D image V1 and a new fluoroscopic image T0 acquired after the reference fluoroscopic image Tb. In this embodiment, the second alignment has a lower computational cost than the first alignment performed by the first alignment unit 22. For the second alignment, the second alignment unit 24 extracts the lung field region from the first deformed 3D image V1 and performs only non-rigid alignment based on the lung field region extracted from the first deformed 3D image V1 and the lung field region extracted from the new fluoroscopic image T0. For this reason, the second alignment has a lower computational cost than the first alignment.

[0038] In the first alignment process, the lung field region extracted from the 3D image V0 is projected into 2D for non-rigid alignment. Therefore, the second alignment unit 24 may perform non-rigid alignment based on the 2D lung field region derived by the first alignment unit 22 and the lung field region extracted from the new fluoroscopic image T0, without extracting the lung field region from the 3D image V0. In this case as well, the second alignment will have a lower computational cost than the first alignment.

[0039] Here, the non-rigid alignment method may be the same as the non-rigid alignment method performed by the first alignment unit 22, or a non-rigid alignment method with less computational complexity may be used. Examples of non-rigid alignment methods with less computational complexity include those described in "Diffeomorphic Demons: Efficient Non-parametric Image Registration," Tom Vercauteren et al., December 2008 NeuroImage 45(1 Suppl):S61-72, DOI:10.1016 / j.neuroimage.2008.10.040 and "Image matching as a diffusion process: an analogy with Maxwell's demons," Jean-Philippe Thirion et al., Medical Image Analysis, Elsevier, 1998, 2 (3), pp.243-260.

[0040] Furthermore, the extraction of the lung field region from the new fluoroscopic image T0 in the second alignment unit 24 may use a method with lower computational cost than the extraction of the lung field region from the reference fluoroscopic image Tb in the first alignment unit 22. For example, the second alignment unit 24 may use a CAD with fewer parameters and less computation than the first alignment unit 22.

[0041] The second derivation unit 25 deforms the first deformed 3D image V1 based on the result of the second alignment to derive the second deformed 3D image V2. That is, the second deformed 3D image V2 is derived by deforming the first deformed 3D image V1 with the amount of deformation derived by the non-rigid alignment performed by the second alignment unit 24.

[0042] The error derivation unit 26 derives the alignment error D0 between the second deformed 3D image V2 and the new fluoroscopic image T0. To do this, the error derivation unit 26 derives the difference between the position of the diaphragm in the second deformed 3D image V2 in the axial direction of the subject H and the position of the diaphragm in the new fluoroscopic image T0 in the axial direction of the subject H.

[0043] Specifically, as shown in Figure 5, the error derivation unit 26 projects the second deformed 3D image V2 in the same direction as the imaging direction of the new fluoroscopic image T0 to derive a 2D pseudo-deformed fluoroscopic image VT2. Then, the error derivation unit 26 aligns the bones in the pseudo-deformed fluoroscopic image VT2 and the new fluoroscopic image T0 and derives the difference in the position of the diaphragm after alignment. The difference in the position of the diaphragm can be represented by a representative value of the difference between the position of the diaphragm 31 in the pseudo-deformed fluoroscopic image VT2 and the position of the diaphragm 32 in the new fluoroscopic image T0. Representative values ​​include the maximum value, minimum value, average value, and median value.

[0044] The error derivation unit 26 may also derive the difference in diaphragm position by using a trained model that derives the difference between the position of the diaphragm of subject H in the second deformed 3D image V2 in the body axis direction and the position of the diaphragm of subject H in the new fluoroscopic image T0 in the body axis direction. Such a trained model is constructed by training a neural network using 3D images and fluoroscopic images with known differences in diaphragm position in the body axis direction as training data.

[0045] Here, the area of ​​the lung field region differs between the expiratory phase and the inspiratory phase. For this reason, the error derivation unit 26 may extract the lung field region from the second deformed 3D image V2 and project it into 2D, extract the lung field region from the new fluoroscopic image T0, and derive the difference between the area of ​​the lung field region projected into 2D and the area of ​​the lung field region extracted from the new fluoroscopic image T0 as the error D0.

[0046] Furthermore, the error derivation unit 26 may use the deformation amount derived by the second alignment unit 24 as the error D0. Since the deformation amount is derived for each pixel of the rigidly aligned pseudo-perspective image VT0 and the new perspective image T0, a representative value of the deformation amount derived for each pixel can be used as the error D0. The representative value can be the mean, median, maximum, minimum, etc.

[0047] The determination unit 27 determines whether the error D0 derived by the error derivation unit 26 is less than a predetermined threshold Th1.

[0048] The display control unit 28 superimposes the first deformed 3D image V1 onto the reference perspective image Tb and displays it on the display 14. Alternatively, it superimposes the second deformed 3D image V2 onto the perspective image T0 and displays it on the display 14. Figure 6 is a diagram showing the display screen of the superimposed image. As shown in Figure 6, the display screen 40 shows a superimposed image 41, which is either the first deformed 3D image V1 superimposed onto the reference perspective image Tb, or the second deformed 3D image V2 superimposed onto the perspective image T0.

[0049] In this embodiment, if the determination by the determination unit 27 is affirmative, the display control unit 28 superimposes the second deformed 3D image V2 onto the new perspective image T0 and displays it on the display 14.

[0050] On the other hand, if the determination by the determination unit 27 is rejected, the first alignment unit 22 updates the reference perspective image Tb to a new perspective image T0 and performs a new first alignment on the updated reference perspective image Tb and the 3D image V0. Then, the first derivation unit 23 deforms the 3D image V0 based on the result of the new first alignment to derive a new first deformed 3D image V1. Then, the display control unit 28 superimposes and displays the new first deformed 3D image V1 on the new perspective image T0. After this, the reference fluoroscopic image Tb is updated to a new fluoroscopic image T0, and the second alignment, the derivation of the second deformed 3D image V2, the derivation of the alignment error D0, the superimposition of the second deformed 3D image V2 onto the new radiographic image T0, the judgment, the superimposition if the judgment is affirmative, a new first alignment if the judgment is negative, the derivation of a new first deformed 3D image V1, the superimposition of the new first deformed 3D image V1, and the update of the reference fluoroscopic image Tb are repeated.

[0051] Next, the processing performed in this embodiment will be described. Figure 7 is a flowchart showing the processing performed in this embodiment. First, the image acquisition unit 21 acquires a 3D image V0 from the image storage server 4 (step ST1), and then the image acquisition unit 21 sequentially acquires perspective images T0 (step ST2). Then, the first alignment unit 22 performs a first alignment between the reference perspective image Tb from the perspective images T0 and the 3D image V0 (step ST3). Next, the first derivation unit 23 deforms the 3D image V0 based on the result of the first alignment to derive a first deformed 3D image V1 (step ST4). Then, the display control unit 28 superimposes the first deformed 3D image V1 onto the reference perspective image Tb and displays it on the display 14 (first superimposed display: step ST5).

[0052] Next, the second alignment unit 24 performs a second alignment between the first deformed 3D image V1 and a new perspective image T0 acquired after the reference perspective image Tb (step ST6). Then, the second derivation unit 25 deforms the first deformed 3D image V1 based on the result of the second alignment to derive the second deformed 3D image V2 (step ST7). Subsequently, the error derivation unit 26 derives the alignment error D0 between the second deformed 3D image V2 and the new perspective image T0 (step ST8). Furthermore, the determination unit 27 determines whether the error D0 derived by the error derivation unit 26 is less than a predetermined threshold Th1 (step ST9).

[0053] If step ST9 is affirmed, the display control unit 28 superimposes the second deformed 3D image V2 onto the new perspective image T0 and displays it on the display 14 (second superimposed display: step ST10), returns to step ST6, and the processing from step ST6 onwards is repeated using the newly acquired perspective image T0. On the other hand, if step ST9 is denied, the first alignment unit 22 updates the reference perspective image Tb to the new perspective image T0 (step ST11), returns to step ST3, and the processing from step ST3 onwards is repeated.

[0054] In this embodiment, after performing a first alignment between the reference perspective image Tb and the 3D image V0, a ​​second alignment is performed between the first deformed 3D image V1 and the new perspective image T0, which has a lower computational cost than the first alignment. Furthermore, the alignment error D0 between the second deformed 3D image V2 and the new perspective image T0 is derived, and if the error is less than the threshold Th1, the second deformed 3D image V2 is superimposed on the new perspective image T0 and displayed on the display 14. As a result, as long as the error is less than the threshold Th1, the second alignment, which has a lower computational cost, is repeated. Therefore, compared to the case where only the first alignment is performed, the amount of computation is reduced and the alignment between the 3D image V0 and the perspective image T0 can be performed at high speed.

[0055] Furthermore, if the error exceeds the threshold Th1, the reference fluoroscopic image Tb is updated to a new fluoroscopic image T0, and the first alignment is performed. As a result, the alignment of the 3D image V0 and the fluoroscopic image T0 can be performed with greater accuracy compared to the case where only the second alignment is performed. Also, compared to the case where only the first alignment is performed, the amount of computation can be reduced, and the alignment of the 3D image V0 and the fluoroscopic image T0 can be performed at a higher speed. Therefore, according to this embodiment, the 3D image V0 and the fluoroscopic image T0 can be aligned quickly and accurately.

[0056] In the above embodiment, the process for collecting lung lesions using a bronchoscope inserted into the bronchi is described, but the invention is not limited to this. For example, the image processing device according to this embodiment can also be applied when inserting an ultrasound endoscope into a digestive organ such as the stomach to perform a biopsy of tissue such as the pancreas or liver.

[0057] Furthermore, in each of the above embodiments, the hardware structure of the Processing Unit that executes various processes such as the image acquisition unit 21, the first alignment unit 22, the first derivation unit 23, the second alignment unit 24, the second derivation unit 25, the error derivation unit 26, the determination unit 27, and the display control unit 28 can be the various processors shown below. As mentioned above, the various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as a Programmable Logic Device (PLD), which is a processor whose circuit configuration can be changed after manufacturing, such as an FPGA (Field Programmable Gate Array), and a dedicated electrical circuit, which is a processor with a circuit configuration specifically designed to execute a particular process, such as an ASIC (Application Specific Integrated Circuit).

[0058] A single processing unit may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA). Alternatively, multiple processing units may be composed of a single processor.

[0059] Examples of configuring multiple processing units with a single processor include, firstly, a configuration where one or more CPUs and software combine to form a single processor, as exemplified by client and server computers, and this processor functions as multiple processing units. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple processing units, on a single IC (Integrated Circuit) chip, as exemplified by System-on-a-Chip (SoC). Thus, various processing units are configured, in terms of hardware structure, using one or more of the above-mentioned processors.

[0060] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits (Circuitry) that combine circuit elements such as semiconductor elements. [Explanation of Symbols]

[0061] 1 Computer 2. 3D image acquisition device 3. Fluoroscopy imaging device 3A Arm 3B X-ray source 3C X-ray detector 4 Image storage server 5 Network 6. Ultrasound Endoscope 6A Endoscope 10 Image Processing Device 11 CPU 12 Image Processing Programs 13 Storage 14 displays 15 Input Devices 16 memory 21 Image acquisition unit 22 First alignment section 23 First derivation part 24 Second alignment section 25 Second derivation part 26 Error derivation part 27 Judgment section 28 Display Control Unit 31,32 Diaphragm 40 display screen 41 Superimposed images T0 fluoroscopic image Tb Reference fluoroscopic image V0 3D image V1, V2 Deformed 3D Images VT0 pseudo fluoroscopic image VT2 Pseudo-deformed fluoroscopic image

Claims

1. Equipped with at least one processor, The aforementioned processor, We acquire a 3D image of the subject, Multiple two-dimensional radiographic images of the subject are acquired sequentially. A first alignment is performed between the reference radiation image and the three-dimensional image from among the plurality of two-dimensional radiation images. Based on the result of the first alignment, the three-dimensional image is deformed to derive a first deformed three-dimensional image. The first deformed three-dimensional image is superimposed onto the reference radiation image. An image processing device that performs a second alignment, which has a lower computational cost than the first alignment, on a new two-dimensional radiation image acquired after the first deformed three-dimensional image and the reference radiation image.

2. The processor deforms the first deformed 3D image based on the result of the second alignment to derive a second deformed 3D image. The alignment error between the second deformed three-dimensional image and the new two-dimensional radiation image is derived. Determine whether the error is less than a predetermined threshold, If the above determination is affirmed, the image processing apparatus according to claim 1, which superimposes the second deformed three-dimensional image onto the new two-dimensional radiation image.

3. If the determination is rejected, the processor updates the reference radiation image to the new two-dimensional radiation image and performs a new first alignment between the reference radiation image and the three-dimensional image. Based on the results of the new first alignment, the three-dimensional image is deformed to derive a new first deformed three-dimensional image. The image processing apparatus according to claim 2, wherein the new first deformed three-dimensional image is superimposed on the new two-dimensional radiation image.

4. The image processing apparatus according to claim 3, wherein the processor repeats the second alignment, the derivation of the second deformed three-dimensional image, the derivation of the error, the superimposition of the second deformed three-dimensional image onto the new two-dimensional radiation image, the determination, the superimposition if the determination is affirmative, the new first alignment if the determination is negative, the derivation of the new first deformed three-dimensional image, the superimposition of the new first deformed three-dimensional image, and the updating of the reference radiation image each time the new two-dimensional radiation image is acquired.

5. The image processing apparatus according to any one of claims 1 to 4, wherein the reference radiographic image used for the first alignment performed initially is a two-dimensional radiographic image of the respiratory phase closest to the respiratory phase of the three-dimensional image.

6. The three-dimensional image and the two-dimensional radiographic image include the chest of the subject, The first alignment includes rigid alignment of the reference radiographic image and the three-dimensional image, and non-rigid alignment based on lung field regions extracted from the reference radiographic image and the three-dimensional image, respectively. The image processing apparatus according to claim 1, wherein the second alignment includes non-rigid alignment based on a lung field region extracted from a first deformed three-dimensional image which became two-dimensional during the first alignment, and a lung field region extracted from the new two-dimensional radiographic image.

7. The image processing apparatus according to claim 6, wherein the extraction of the lung field region from the two-dimensional radiographic image in the second alignment requires less computational cost than the extraction of the lung field region from the two-dimensional radiographic image in the first alignment.

8. We acquire a 3D image of the subject, Multiple two-dimensional radiographic images of the subject are acquired sequentially. A first alignment is performed between the reference radiation image and the three-dimensional image from among the plurality of two-dimensional radiation images. Based on the result of the first alignment, the three-dimensional image is deformed to derive a first deformed three-dimensional image. The first deformed three-dimensional image is superimposed onto the reference radiation image. An image processing method that performs a second alignment, which has a lower computational cost than the first alignment, on the first deformed three-dimensional image and a new two-dimensional radiation image acquired after the reference radiation image.

9. Procedure for acquiring a 3D image of the subject, A procedure for sequentially acquiring multiple two-dimensional radiographic images of the subject, A procedure for performing a first alignment between a reference radiation image and a three-dimensional image from among the plurality of two-dimensional radiation images, A procedure for deriving a first deformed 3D image by deforming the 3D image based on the result of the first alignment, A procedure for superimposing the first deformed three-dimensional image onto the reference radiation image, An image processing program that causes a computer to perform a procedure for performing a second alignment, which has a lower computational cost than the first alignment, on the first deformed three-dimensional image and a new two-dimensional radiation image acquired after the reference radiation image.

Images