Image processing apparatus, method, and program

Abstract

To provide an image processing device, method, and program capable of speedily and accurately positioning a three-dimensional image and a radiation image.SOLUTION: A processor acquires a three-dimensional image and a radiation image about an identical subject, performs rigid body positioning of the three-dimensional image and the radiation image, derives an error of a respiratory phase between the three-dimensional image and the radiation image subjected to the rigid body positioning, determines whether or not the error is less than a predetermined threshold, and when the determination is negative, performs non-rigid body positioning of the three-dimensional image and the radiation image.SELECTED DRAWING: Figure 3

Description

Technical Field

[0005]

[0001] The present disclosure relates to an image processing apparatus, method, and program.

Background Art

[0002] An ultrasonic endoscope having an endoscopic observation unit and an ultrasonic observation unit at its tip is inserted into a lumen such as the digestive organ or bronchus of a subject, and endoscopic images within the lumen and ultrasonic images of sites such as lesions outside the lumen wall are taken. Also, biopsies are performed to collect tissues of lesions outside the lumen wall using treatment tools such as forceps.

[0003] When performing a procedure using such an ultrasonic endoscope, it is important to accurately reach the target position within the subject. For this reason, fluoroscopic imaging is performed in which radiation is continuously irradiated from a radiation source to the subject during the procedure, and the fluoroscopic image thus obtained is displayed in real time, thereby grasping the positional relationship between the ultrasonic endoscope and the human body structure.

[0004] Here, since the fluoroscopic image includes anatomical structures such as organs, blood vessels, and bones within the subject overlapping each other, it is not easy to recognize the lumen and lesions. For this reason, a three-dimensional image of the subject is acquired in advance before the procedure using a CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, etc., the lesion position is specified in the three-dimensional image, and the lesion position is specified in the fluoroscopic image by aligning the three-dimensional image and the fluoroscopic image (see, for example, Patent Document 1). Also, various methods for performing rigid body alignment or non-rigid body alignment between the three-dimensional image and the fluoroscopic image have been proposed (see, for example, Patent Documents 2 and 3).

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

[0006] Rigid body alignment requires less computation and can be performed relatively quickly, but its alignment accuracy is not very high. On the other hand, non-rigid body alignment offers high alignment accuracy but requires more computation and therefore takes more time to process.

[0007] This invention has been made in view of the above circumstances, and aims to enable high-speed and high-precision alignment of three-dimensional images and two-dimensional radiation images. [Means for solving the problem]

[0008] An image processing apparatus according to a first aspect of the present disclosure comprises at least one processor, the processor acquires a three-dimensional image and a radiographic image of the same subject, By rigidly aligning the 3D image and the radiation image, We derived the respiratory phase error between a rigidly aligned 3D image and a radiographic image. Determine whether the error is below a predetermined threshold. If the determination is rejected, a non-rigid alignment is performed between the 3D image and the radiographic image.

[0009] In the image processing apparatus according to the second aspect of the present disclosure, the three-dimensional image and the radiographic image may include the lung field region of the subject, as in the image processing apparatus according to the first aspect.

[0010] An image processing apparatus according to a third aspect of this disclosure, in an image processing apparatus according to a second aspect, the processor extracts lung field regions from a three-dimensional image and a radiographic image, respectively. This method may also involve non-rigid alignment of the 3D image and the radiographic image based on the extracted lung field region.

[0011] The image processing apparatus according to the fourth aspect of this disclosure is an image processing apparatus according to the third aspect, wherein the processor extracts lung field regions from the three-dimensional image and the radiographic image respectively when the determination is denied.

[0012] The image processing apparatus according to the fifth aspect of this disclosure is an image processing apparatus according to the third aspect, in which the processor extracts lung field regions from the three-dimensional image and the radiographic image, respectively, before making a determination.

[0013] An image processing apparatus according to a sixth aspect of the present disclosure may be an image processing apparatus according to any one of the second to fifth aspects, wherein the processor derives a respiratory phase error based on the difference between the position of the diaphragm in the axial direction of the subject included in the three-dimensional image and the position of the diaphragm in the axial direction of the subject included in the radiographic image.

[0014] The image processing apparatus according to the seventh aspect of this disclosure is an image processing apparatus according to the sixth aspect, wherein the processor uses a trained model to derive the position of the diaphragm in the axial direction of the subject in the three-dimensional image and the position of the diaphragm in the axial direction of the subject in the radiographic image.

[0015] An image processing apparatus according to the eighth aspect of the present disclosure may be an image processing apparatus according to any one aspect of the second to fifth aspects, wherein the processor derives a respiratory phase error based on the difference between the area of ​​the lung field region included in the projected image obtained by projecting a three-dimensional image in the direction of radiographic image acquisition and the area of ​​the lung field region included in the radiographic image.

[0016] The image processing apparatus according to the ninth aspect of the present disclosure is the image processing apparatus according to any one of the first to eighth aspects, wherein when the determination is negative, the processor superimposes and displays a non-rigidly aligned three-dimensional image and a radiation image, and when the determination is positive, the processor may superimpose and display a rigidly aligned three-dimensional image and a radiation image.

[0017] The image processing method of the present disclosure acquires a three-dimensional image and a radiation image of the same subject, rigidly aligns the three-dimensional image and the radiation image, derives the respiratory phase error between the rigidly aligned three-dimensional image and the radiation image, determines whether the error is less than a predetermined threshold value, when the determination is negative, non-rigid alignment is performed between the three-dimensional image and the radiation image.

[0018] The image processing program of the present disclosure causes a computer to execute procedures for acquiring a three-dimensional image and a radiation image of the same subject, rigidly aligning the three-dimensional image and the radiation image, deriving the respiratory phase error between the rigidly aligned three-dimensional image and the radiation image, determining whether the error is less than a predetermined threshold value, and when the determination is negative, performing non-rigid alignment between the three-dimensional image and the radiation image.

Advantages of the Invention

[0019] According to the present disclosure, a three-dimensional image and a radiation image can be aligned quickly and accurately.

Brief Description of the Drawings

[0020] [Figure 1] Diagram showing the schematic configuration of a medical information system to which the image processing apparatus according to the first embodiment of the present disclosure is applied [Figure 2] Diagram showing the schematic configuration of the image processing apparatus according to the first embodiment [Figure 3]Functional configuration diagram of the image processing apparatus according to the first embodiment. [Figure 4] This diagram schematically shows the processing performed by the image processing device according to the first embodiment. [Figure 5] A diagram illustrating the detection of differences in diaphragm position. [Figure 6] Diagram showing the display screen [Figure 7] A flowchart illustrating the process performed in the first embodiment. [Figure 8] A flowchart illustrating the process performed in the second embodiment. [Modes for carrying out the invention]

[0021] Embodiments of this disclosure will be described below with reference to the drawings. First, the configuration of a medical information system to which the image processing device according to the first embodiment is applied will be described. Figure 1 is a diagram showing the schematic configuration of a medical information system. In the medical information system shown in Figure 1, a computer 1 containing the image processing device according to the first embodiment, a 3D image acquisition device 2, a fluoroscopy image acquisition device 3, and an image storage server 4 are connected via a network 5 in a state where they can communicate with each other.

[0022] Computer 1 contains the image processing device according to the first embodiment, and the image processing program according to the first embodiment is installed on it. Computer 1 is installed in the treatment room where treatment is performed on subject H, as described later. Computer 1 may be a workstation or personal computer directly operated by the medical professional performing the treatment, or it may be a server computer connected to them via a network. The image processing program is stored in a storage device of the server computer connected to the network, or in network storage, in a state that is accessible from the outside, and is downloaded and installed on computer 1 used by the physician as needed. Alternatively, it may be recorded on a recording medium such as a DVD (Digital Versatile Disc) or CD-ROM (Compact Disc Read Only Memory) and distributed, and then installed on computer 1 from that recording medium.

[0023] The 3D imaging device 2 is a device that generates a 3D image representing a specific area of ​​the subject H by imaging that area. Specifically, it is a CT scanner, an MRI scanner, or a PET (Positron Emission Tomography) scanner. The 3D image generated by the 3D imaging device 2, consisting of multiple tomographic images, is transmitted to the image storage server 4 and stored there. In this embodiment, the area of ​​the subject H to be treated is the lung, and the 3D imaging device 2 is a CT scanner. As will be described later, before treatment on the subject H, the chest of the subject H is imaged, and a CT image including the chest of the subject H is acquired in advance as a 3D image and stored in the image storage server 4.

[0024] The fluoroscopy imaging device 3 comprises a C-arm 3A, an X-ray source 3B, and an X-ray detector 3C. The X-ray source 3B and the X-ray detector 3C are attached to both ends of the C-arm 3A, respectively. In the fluoroscopy imaging device 3, the C-arm 3A is configured to be rotatable and movable so that the subject H can be photographed from any direction. Then, as will be described later, during treatment of the subject H, the fluoroscopy imaging device 3 continuously irradiates the subject H with X-rays at a predetermined frame rate and sequentially detects the X-rays that have passed through the subject H with the X-ray detector 3C, thereby sequentially acquiring X-ray images of the subject H. In the following description, the sequentially acquired X-ray images will be referred to as fluoroscopic images. A fluoroscopic image is an example of a radiographic image according to this disclosure.

[0025] The image storage server 4 is a computer that stores and manages various types of data, and is equipped with a large-capacity external storage device and database management software. The image storage server 4 communicates with other devices via a wired or wireless network 5 to send and receive image data, etc. Specifically, it acquires various types of data, including 3D images acquired by the 3D image acquisition device 2 and fluoroscopic images acquired by the fluoroscopic image acquisition device 3, via the network, and stores and manages them on a recording medium such as a large-capacity external storage device. The storage format of the image data and communication between each device via the network 5 are based on protocols such as DICOM (Digital Imaging and Communication in Medicine).

[0026] In this embodiment, while performing fluoroscopic imaging of subject H, a biopsy procedure is performed to examine the presence of disease in detail by taking a sample of a lesion such as a pulmonary nodule in the lung of subject H, which has been previously detected using a 3D image V0. For this reason, the fluoroscopic imaging device 3 is located in the treatment room where the procedure is performed. An endoscopic ultrasound device 6 is also installed in the treatment room. The endoscopic ultrasound device 6 is equipped with an endoscope 6A with an ultrasound probe and treatment instruments such as forceps attached to its tip. In this embodiment, in order to perform a biopsy of a lesion, the operator inserts the endoscope 6A into the bronchus of subject H, takes a fluoroscopic image of subject H using the fluoroscopic imaging device 3, and displays the captured fluoroscopic image and the endoscopic image taken by the endoscope 6A in real time. The operator confirms the position of the tip of the endoscope 6A within subject H in the fluoroscopic image and moves the tip of the endoscope 6A to the location of the target lesion.

[0027] Here, lung lesions such as pulmonary nodules occur outside the bronchi, not inside them. Therefore, after moving the tip of the endoscope 6A to the target position, the operator takes an ultrasound image of the outside of the bronchi using an ultrasound probe, displays the ultrasound image, and, while confirming the location of the lesion in the ultrasound image, performs a procedure to collect a portion of the lesion using instruments such as forceps.

[0028] Next, an image processing apparatus according to the first embodiment will be described. Figure 2 is a diagram showing the hardware configuration of the image processing apparatus according to the first embodiment. As shown in Figure 2, the image processing apparatus 10 includes a CPU (Central Processing Unit) 11, non-volatile storage 13, and memory 16 as a temporary storage area. The image processing apparatus 10 also includes a display 14 such as a liquid crystal display, input devices 15 such as a keyboard and mouse, and a network I / F (Interface) 17 connected to a network 5. The CPU 11, storage 13, display 14, input devices 15, memory 16, and network I / F 17 are connected to a bus 18. Note that the CPU 11 is an example of a processor in this disclosure.

[0029] The storage 13 is implemented using an HDD (Hard Disk Drive), SSD (Solid State Drive), flash memory, etc. The image processing program 12 is stored in the storage 13 as a storage medium. The CPU 11 reads the image processing program 12 from the storage 13, expands it into memory 16, and executes the expanded image processing program 12.

[0030] Next, the functional configuration of the image processing apparatus according to the first embodiment will be described. Figure 3 is a diagram showing the functional configuration of the image processing apparatus according to the first embodiment. Figure 4 is a diagram schematically showing the processing performed by the image processing apparatus according to the first embodiment. As shown in Figure 3, the image processing apparatus 10 includes an image acquisition unit 21, a first alignment unit 22, an error derivation unit 23, a determination unit 24, an extraction unit 25, a second alignment unit 26, and a display control unit 27. When the CPU 11 executes the image processing program 12, the CPU 11 functions as the image acquisition unit 21, the first alignment unit 22, the error derivation unit 23, the determination unit 24, the extraction unit 25, the second alignment unit 26, and the display control unit 27.

[0031] The image acquisition unit 21 acquires a three-dimensional image V0 of the subject H from the image storage server 4 based on instructions from the operator via the input device 15. The image acquisition unit 21 also sequentially acquires fluoroscopic images T0 obtained by the fluoroscopic image acquisition device 3 during the treatment of the subject H.

[0032] The first alignment unit 22 aligns the sequentially acquired fluoroscopic image T0 with the 3D image V0. Here, the fluoroscopic image T0 is a 2D radiographic image. Therefore, the first alignment unit 22 aligns the 2D image with the 3D image. In this embodiment, the first alignment unit 22 first projects the 3D image V0 in the same direction as the imaging direction of the fluoroscopic image T0 to derive a 2D pseudo-fluoroscopic image VT0. Then, the first alignment unit 22 rigidly aligns the 2D pseudo-fluoroscopic image VT0 with the fluoroscopic image T0. Any method can be used for rigid body alignment, such as an affine transformation. For example, when an affine transformation is used, the first alignment unit 22 derives the amount of translation and rotation of the pseudo-fluoroscopic image VT0 relative to the fluoroscopic image T0 as rigid body alignment information in order to match feature points such as the intersections of ribs contained in the pseudo-fluoroscopic image VT0 and the fluoroscopic image T0. The first alignment unit 22 then deforms the 3D image V0 using rigid body alignment information to derive a deformed 3D image V1.

[0033] The error derivation unit 23 derives the respiratory phase error D0 between the rigidly aligned 3D image V0, i.e., the deformed 3D image V1, and the fluoroscopic image T0. To this end, the error derivation unit 23 derives the difference between the position of the diaphragm in the axial direction of the subject H included in the deformed 3D image V1 derived by the first alignment unit 22, and the position of the diaphragm in the axial direction of the subject H included in the fluoroscopic image T0.

[0034] Specifically, as shown in Figure 5, the error derivation unit 23 projects the deformed 3D image V1 in the same direction as the imaging direction of the fluoroscopic image T0 to derive a 2D pseudo-deformed fluoroscopic image VT1. Then, the error derivation unit 23 aligns the bones in the pseudo-deformed fluoroscopic image VT1 and the fluoroscopic image T0 and derives the difference in the position of the diaphragm after alignment. The difference in the position of the diaphragm can be represented by a representative value of the difference between the position of the diaphragm 31 in the pseudo-deformed fluoroscopic image VT1 and the position of the diaphragm 32 in the fluoroscopic image T0. Representative values ​​include the maximum value, minimum value, average value, and median value.

[0035] The error derivation unit 23 may also derive the difference in diaphragm position by using a trained model that derives the difference between the position of the diaphragm of subject H in the axial direction included in the deformed 3D image V1 and the position of the diaphragm of subject H in the axial direction included in the fluoroscopic image T0. Such a trained model is constructed by training a neural network using 3D images and fluoroscopic images with known differences in diaphragm position in the axial direction as training data.

[0036] The determination unit 24 determines whether the error D0 derived by the error derivation unit 23 is less than a predetermined threshold Th1.

[0037] If the determination by the determination unit 24 is rejected, the extraction unit 25 extracts the lung region from the 3D image V0 and the fluoroscopic image T0. The lung region is the soft tissue of the lung included in the 3D image V0 and the fluoroscopic image T0. In this embodiment, the extraction unit 25 extracts the lung region from the 3D image V0 and the fluoroscopic image T0 using a known computer-aided diagnosis (CAD) algorithm.

[0038] If the determination by the determination unit 24 is rejected, the second alignment unit 26 performs non-rigid alignment between the 3D image V0 and the fluoroscopic image T0. Specifically, it performs non-rigid alignment between the 3D image V0 and the fluoroscopic image T0 by deforming the 3D image V0 so that the lung field region extracted from the 3D image V0 matches the lung field region extracted from the fluoroscopic image T0, and derives non-rigid alignment information that represents the amount of deformation of the 3D image V0 relative to the fluoroscopic image T0. Then, the second alignment unit 26 deforms the 3D image V0 using the non-rigid alignment information to derive a deformed 3D image V2.

[0039] For non-rigid body alignment, methods can be used, but are not limited to, such as using functions like B-splines and thin-plate splines to non-linearly transform the correspondence points between the lung field region in the pseudo-deformed fluoroscopic image VT1 and the lung field region in the fluoroscopic image T0 for the deformed 3D image V1. Any method, such as a method that performs non-rigid body alignment using a trained model, can be used.

[0040] If the determination by the determination unit 24 is negative, the display control unit 27 superimposes the deformed 3D image V2 onto the perspective image T0 and displays the superimposed image on the display 14. On the other hand, if the determination by the determination unit 24 is positive, that is, if the error D0 is less than the threshold Th1, the display control unit 27 superimposes the deformed 3D image V1 onto the perspective image T0 and displays the superimposed image on the display 14. In this case, the display control unit 27 projects the deformed 3D images V1 and V2 in the same direction as the shooting direction of the perspective image T0 to derive 2D pseudo-deformed perspective images VT1 and VT2, and superimposes the pseudo-deformed perspective images VT1 and VT2 onto the perspective image T0. Figure 4 shows that when the determination of whether or not the error D0 is less than the threshold Th1 is positive, a dashed arrow points to the deformed 3D image V1, indicating that the deformed 3D image V1 and the perspective image T0 are superimposed.

[0041] Figure 6 shows the display screen for the superimposed image. As shown in Figure 6, the display screen 40 shows a superimposed image 41, which is obtained by superimposing a deformed 3D image V1 or a deformed 3D image V2 onto a perspective image T0.

[0042] Next, the process performed in the first embodiment will be described. Figure 7 is a flowchart of the process performed in the first embodiment. First, the image acquisition unit 21 acquires a 3D image V0 from the image storage server 4 (step ST1), and then the image acquisition unit 21 acquires a perspective image T0 (step ST2). Then, the first alignment unit 22 performs rigid alignment of the 3D image V0 and the perspective image T0 (step ST3). Subsequently, the error derivation unit 23 derives the breathing phase error D0 between the rigidly aligned 3D image, i.e., the deformed 3D image V1 and the perspective image T0 (step ST4), and the determination unit 24 determines whether the error D0 is less than a predetermined threshold Th1 (step ST5).

[0043] If step ST5 is rejected, the extraction unit 25 extracts the lung field region from the 3D image V0 and the fluoroscopic image T0 (step ST6), and the second alignment unit 26 performs non-rigid alignment of the 3D image V0 and the fluoroscopic image T0 based on the lung field region (step ST7). Then, the display control unit 27 superimposes the non-rigid alignment of the 3D image, i.e., the deformed 3D image V2, and the fluoroscopic image T0 (step ST8), and returns to step ST2.

[0044] On the other hand, if step ST5 is affirmed, the display control unit 27 superimposes the rigidly aligned 3D image, i.e., the deformed 3D image V1 and the perspective image T0 (step ST9), and returns to step ST2.

[0045] Thus, in this embodiment, the alignment of the 3D image V0 and the 2D fluoroscopic image T0 is performed using rigid body alignment, which requires relatively little computation, and only when the respiratory phase error D0 is greater than or equal to the threshold Th1, the alignment of the 3D image V0 and the fluoroscopic image T0 is performed using non-rigid body alignment, which requires relatively more computation. Therefore, the alignment of the 3D image V0 and the fluoroscopic image T0 can be performed with greater accuracy compared to the case where only rigid body alignment is performed. Furthermore, the alignment of the 3D image V0 and the 2D fluoroscopic image T0 can be performed at high speed by reducing the amount of computation compared to the case where only non-rigid body alignment is performed. Consequently, the 3D image V0 and the 2D fluoroscopic image T0 can be aligned quickly and accurately.

[0046] Next, a second embodiment of the present disclosure will be described. Note that the functional configuration of the image processing apparatus according to the second embodiment is the same as that of the image processing apparatus according to the first embodiment shown in Figure 3, so a detailed explanation will be omitted here. In the first embodiment described above, if the determination by the determination unit 24 is rejected, the lung field region is extracted from the 3D image V0 and the fluoroscopic image T0 to perform non-rigid alignment. The second embodiment differs from the first embodiment in that the lung field region is extracted from the 3D image V0 and the fluoroscopic image T0 before the determination by the determination unit 24.

[0047] Next, the processing performed in the second embodiment will be described. Figure 8 is a flowchart showing the processing performed in the second embodiment. First, the image acquisition unit 21 acquires a 3D image V0 from the image storage server 4 (step ST11), and then the image acquisition unit 21 acquires a fluoroscopic image T0 (step ST12). Subsequently, the extraction unit 25 extracts the lung field region from the 3D image V0 and the fluoroscopic image T0 (step ST13). Then, the first alignment unit 22 performs rigid body alignment of the 3D image V0 and the fluoroscopic image T0 (step ST14). Subsequently, the error derivation unit 23 derives the respiratory phase error D0 between the rigidly aligned 3D image, i.e., the deformed 3D image V1, and the fluoroscopic image T0 (step ST15), and the determination unit 24 determines whether the error D0 is less than a predetermined threshold Th1 (step ST16). Note that the processing in step ST13 may be performed after the processing in step ST14 and before the processing in step ST16. Alternatively, the processing in steps ST14 and ST15 may be performed in parallel.

[0048] If step ST16 is rejected, the second alignment unit 26 performs non-rigid alignment of the 3D image V0 and the fluoroscopic image T0 based on the lung field region (step ST17). Then, the display control unit 27 superimposes the non-rigid alignment of the 3D image, i.e., the deformed 3D image V2 and the fluoroscopic image T0 (step ST18), and returns to step ST12.

[0049] On the other hand, if step ST16 is affirmed, the display control unit 27 superimposes the rigidly aligned 3D image, i.e., the deformed 3D image V1 and the perspective image T0 (step ST19), and returns to step ST12.

[0050] Thus, in the second embodiment, since the lung field region is extracted from the 3D image V0 and the fluoroscopic image T0 before the determination by the determination unit 24, if the determination by the determination unit 24 is rejected, non-rigid alignment can be performed immediately.

[0051] In the embodiments described above, the error derivation unit 23 derives the difference in the position of the diaphragm between the deformed 3D image V1 and the fluoroscopic image T0 as the error D0, but it is not limited to this. Here, the area of ​​the lung field differs between the expiratory phase and the inspiratory phase. For this reason, the error derivation unit 23 may extract the lung field from the deformed 3D image V1 and the fluoroscopic image T0, respectively, and derive the difference in the area of ​​the extracted lung field as the error D0.

[0052] Furthermore, while the above embodiments describe the process of collecting lung lesions using a bronchoscope inserted into the bronchi, the invention is not limited to this. For example, the image processing device according to this embodiment can also be applied when inserting an ultrasound endoscope into a digestive organ such as the stomach to perform a biopsy of tissue such as the pancreas or liver.

[0053] Furthermore, in each of the above embodiments, the hardware structure of the Processing Unit that executes various processes, such as the image acquisition unit 21, the first alignment unit 22, the error derivation unit 23, the determination unit 24, the extraction unit 25, the second alignment unit 26, and the display control unit 27, can be the following types of processors. As mentioned above, these types of processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as a Programmable Logic Device (PLD), which is a processor whose circuit configuration can be changed after manufacturing, such as an FPGA (Field Programmable Gate Array), and a dedicated electrical circuit, which is a processor with a circuit configuration specifically designed to execute a particular process, such as an ASIC (Application Specific Integrated Circuit).

[0054] A single processing unit may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA). Alternatively, multiple processing units may be composed of a single processor.

[0055] Examples of configuring multiple processing units with a single processor include, firstly, a configuration where one or more CPUs and software combine to form a single processor, as exemplified by client and server computers, and this processor functions as multiple processing units. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple processing units, on a single IC (Integrated Circuit) chip, as exemplified by System-on-a-Chip (SoC). Thus, various processing units are configured, in terms of hardware structure, using one or more of the above-mentioned processors.

[0056] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits (Circuitry) that combine circuit elements such as semiconductor devices. [Explanation of Symbols]

[0057] 1 Computer 2. 3D image acquisition device 3. Fluoroscopy imaging device 3A Arm 3B X-ray source 3C X-ray detector 4 Image storage server 5 Network 6. Ultrasound Endoscope 6A Endoscope 10 Image Processing Device 11 CPU 12 Image Processing Programs 13 Storage 14 displays 15 Input Devices 16 memory 21 Image acquisition unit 22 First alignment section 23 Error derivation part 24 Judgment section 25 Extraction part 26 Second alignment section 27 Display Control Unit 31,32 Diaphragm 40 display screen 41 Superimposed images T0 fluoroscopic image V0 3D image V1, V2 Deformed 3D Images VT0 pseudo fluoroscopic image VT1 Pseudo-deformed fluoroscopic image

Claims

1. Equipped with at least one processor, The aforementioned processor, By acquiring 3D images and radiographic images of the same subject, The three-dimensional image and the radiation image are rigidly aligned, The error in the respiratory phase between the rigidly aligned three-dimensional image and the radiation image is derived. Determine whether the error is less than a predetermined threshold, An image processing device that performs non-rigid alignment between the three-dimensional image and the radiation image when it is determined that the error is greater than or equal to the threshold.

2. The image processing apparatus according to claim 1, wherein the three-dimensional image and the radiographic image include the lung field region of the subject.

3. The processor extracts the lung region from the three-dimensional image and the radiographic image, The image processing apparatus according to claim 2, which performs non-rigid alignment of the three-dimensional image and the radiographic image based on the extracted lung field region.

4. The image processing apparatus according to claim 3, wherein the processor determines that the error is greater than or equal to the threshold, and extracts a lung field region from the three-dimensional image and the radiographic image, respectively.

5. The image processing apparatus according to claim 3, wherein the processor extracts lung field regions from the three-dimensional image and the radiographic image, respectively, before performing the determination.

6. The image processing apparatus according to any one of claims 2 to 5, wherein the processor derives the respiratory phase error based on the difference between the position of the diaphragm in the axial direction of the subject included in the three-dimensional image and the position of the diaphragm in the axial direction of the subject included in the radiographic image.

7. The image processing apparatus according to claim 6, wherein the processor uses a trained model to derive the position of the diaphragm included in the three-dimensional image in the axial direction of the subject and the position of the diaphragm included in the radiographic image in the axial direction of the subject.

8. The image processing apparatus according to any one of claims 2 to 5, wherein the processor derives the respiratory phase error based on the difference between the area of ​​the lung field region included in the projected image obtained by projecting the three-dimensional image in the direction of the imaging of the radiographic image and the area of ​​the lung field region included in the radiographic image.

9. The image processing apparatus according to claim 1, wherein the processor, when it is determined that the error is greater than or equal to the threshold, superimposes the non-rigid alignment three-dimensional image and the radiation image, and when it is determined that the error is less than the threshold, superimposes the rigid alignment three-dimensional image and the radiation image.

10. By acquiring 3D images and radiographic images of the same subject, The three-dimensional image and the radiation image are rigidly aligned, The error in the respiratory phase between the rigidly aligned three-dimensional image and the radiation image is derived. Determine whether the error is less than a predetermined threshold, An image processing method that performs non-rigid alignment between the three-dimensional image and the radiation image when it is determined that the error is greater than or equal to the threshold.

11. Procedures for acquiring 3D images and radiographic images of the same subject, A procedure for rigidly aligning the three-dimensional image and the radiation image, A procedure for deriving the respiratory phase error between the rigidly aligned three-dimensional image and the radiation image, A procedure for determining whether the error is less than a predetermined threshold, An image processing program that, when it is determined that the error is greater than or equal to the threshold, causes the computer to perform a procedure for non-rigid alignment between the three-dimensional image and the radiation image.

Images