Image processing apparatus, method, and program

Abstract

To provide an image processing device, method, and program capable of navigating an endoscope to a desired position in a subject without using a sensor.SOLUTION: A processor acquires a three-dimensional image of a subject; acquires a real endoscope image of a lumen structure of the subject imaged by an endoscope inserted into the lumen structure of the subject; derives a virtual viewpoint in the three-dimensional image of the endoscope using the real endoscope image and the three-dimensional image; acquires a radiation image of the subject in which the endoscope is inserted into the lumen structure; and executes positioning between the radiation image and the three-dimensional image. On the basis of the virtual viewpoint and the result of the positioning, at least part of the lumen structure included in the three-dimensional image is displayed on the radiation image in a superimposed manner.SELECTED DRAWING: Figure 3

Description

Technical Field

[0005] ,

[0004] , , , , ,

[0006]

[0001] The present disclosure relates to an image processing apparatus, method, and program. [[ID=�]]

Background Art

[0002] An endoscope having an endoscopic observation unit and an ultrasonic observation unit at its tip is inserted into a lumen structure such as the digestive organ or bronchus of a subject, and an endoscopic image within the lumen structure and an ultrasonic image of a site such as a lesion outside the outer wall of the lumen structure are captured. Also, a biopsy is performed to collect a tissue sample of the lesion using a treatment tool such as forceps.

[0003] When performing a procedure using such an endoscope, it is important to accurately reach the target position within the subject. For this reason, fluoroscopic imaging is performed by continuously irradiating the subject with radiation from a radiation source during the procedure and displaying the fluoroscopic image obtained thereby in real time, so as to grasp the positional relationship between the endoscope and the human body structure.

[0004] Here, since the fluoroscopic image includes the anatomical structures such as organs, blood vessels, and bones within the subject overlapping each other, it is not easy to recognize the lumen and the lesion. For this reason, a three-dimensional image of the subject is acquired in advance before the procedure using a CT (Computed Tomography) apparatus, an MRI (Magnetic Resonance Imaging) apparatus, etc., and the insertion path of the endoscope and the position of the lesion, etc. are simulated in the three-dimensional image in advance.

[0005] Also, in Patent Document 1, a virtual endoscopic image inside the bronchus is generated from a three-dimensional image, the tip position of the endoscope is detected using a position sensor during the procedure, and the virtual endoscopic image is displayed together with the real endoscopic image captured by the endoscope to perform navigation of the insertion of the endoscope into the bronchus.

[0006] Furthermore, Patent Document 2 proposes a method in which a position sensor is attached to the tip of the endoscope to detect the tip position of the endoscope, a grid-like marker is used to detect the orientation of the imaging device that takes fluoroscopic images, a 3D image is reconstructed from multiple acquired fluoroscopic images, and the reconstructed 3D image is aligned with a 3D image such as a CT image acquired in advance. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2009-056239 [Patent Document 2] Japanese Patent Publication No. 2021-030073 [Overview of the project] [Problems that the invention aims to solve]

[0008] However, the methods described in Patent Documents 1 and 2 require sensors to be installed on the endoscope to detect its position. To avoid using sensors, it is conceivable to detect the position of the endoscope from the image of the endoscope captured in the fluoroscopic image. However, since the position in the depth direction perpendicular to the fluoroscopic image cannot be determined, it is not possible to detect the three-dimensional position of the endoscope from the fluoroscopic image. For this reason, it is not possible to accurately navigate the endoscope to the desired position within the subject.

[0009] This invention has been made in view of the above circumstances, and aims to enable the navigation of an endoscope to a desired position within a subject without the use of sensors. [Means for solving the problem]

[0010] An image processing apparatus according to a first aspect of the present disclosure comprises at least one processor, the processor acquires a three-dimensional image of a subject, By inserting an endoscope into the tubular structure of the subject, we obtain actual endoscopic images of the tubular structure of the subject. Using actual endoscopic images and 3D images, we derive a virtual viewpoint within the 3D image of the endoscope. An endoscope is inserted into a tubular structure to obtain radiographic images of the subject. Align the radiographic image with the 3D image,

[0011] Based on the virtual viewpoint and alignment results, at least a portion of the tubular structures included in the 3D image are superimposed onto the radiographic image.

[0012] The image processing apparatus according to the second aspect of the present disclosure is an image processing apparatus according to the first aspect of the present disclosure, wherein the processor detects the position of the endoscope from the radiographic image, Alignment may also be performed by setting the projection conditions of the 3D image so that the projection position when the virtual viewpoint is projected onto the radiographic image coincides with the position of the endoscope.

[0013] "To match" includes not only a perfect match, but also positions that are close enough to be practically identical.

[0014] An image processing apparatus according to a third aspect of the present disclosure is an image processing apparatus according to a second aspect of the present disclosure, wherein the processor further sets projection conditions such that a specific anatomical structure contained in a radiographic image matches a specific anatomical structure contained in a projected image derived by projecting a three-dimensional image onto a radiographic image.

[0015] In the third embodiment, “to coincide” includes not only cases where they coincide perfectly, but also cases where their positions are close enough to coincide substantially.

[0016] An image processing apparatus according to a fourth aspect of the present disclosure may, in an image processing apparatus according to a second or third aspect of the present disclosure, further include non-rigid alignment.

[0017] The image processing apparatus according to the fifth aspect of the present disclosure is the image processing apparatus according to any one of the first to fourth aspects of the present disclosure, wherein the processor identifies at least a part to be projected onto the radiation image for the luminal structure included in the three-dimensional image based on the position and orientation of the virtual viewpoint. At least a part of the identified luminal structure may be superimposed and displayed on the radiation image.

[0018] The image processing method according to the present disclosure includes acquiring a three-dimensional image of a subject. Acquiring a real endoscopic image within the luminal structure of the subject captured by an endoscope inserted into the luminal structure of the subject. Deriving a virtual viewpoint within the three-dimensional image of the endoscope using the real endoscopic image and the three-dimensional image. Acquiring a radiation image of the subject with the endoscope inserted into the luminal structure. Performing alignment between the radiation image and the three-dimensional image. Based on the virtual viewpoint and the result of alignment, at least a part of the luminal structure included in the three-dimensional image is superimposed and displayed on the radiation image.

[0019] The image processing program according to the present disclosure includes a procedure for acquiring a three-dimensional image of a subject, a procedure for acquiring a real endoscopic image within the luminal structure of the subject captured by an endoscope inserted into the luminal structure of the subject, a procedure for deriving a virtual viewpoint within the three-dimensional image of the endoscope using the real endoscopic image and the three-dimensional image, a procedure for acquiring a radiation image of the subject with the endoscope inserted into the luminal structure, a procedure for performing alignment between the radiation image and the three-dimensional image, and a procedure for causing a computer to superimpose and display at least a part of the luminal structure included in the three-dimensional image on the radiation image based on the virtual viewpoint and the result of alignment. [Advantages of the Invention]

[0020] According to the present disclosure, without using a sensor, the endoscope can be navigated to a desired position within the subject.

Brief Description of the Drawings

[0021] [Figure 1] Figure showing the schematic configuration of a medical information system to which an image processing apparatus according to an embodiment of the present disclosure is applied [Figure 2] Figure showing the schematic configuration of an image processing apparatus according to the present embodiment [Figure 3] Functional configuration diagram of the image processing apparatus according to the present embodiment [Figure 4] Figure showing an information generation screen [Figure 5] Figure for explaining the adjustment of a virtual viewpoint using the method of Zhou et al. [Figure 6] FIG. 6 is a diagram schematically showing the derivation of a virtual viewpoint [Figure 7] Figure showing a perspective image [Figure 8] Figure showing a navigation screen [Figure 9] Figure showing a navigation screen [Figure 10] Flowchart showing the processing performed in the present embodiment

Mode for Carrying Out the Invention

[0022] [[ID=4,2]]Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. First, the configuration of a medical information system to which an image processing apparatus according to the present embodiment is applied will be described. FIG. 1 is a diagram showing the schematic configuration of the medical information system. The medical information system shown in FIG. 1 includes a computer 1 incorporating the image processing apparatus according to the present embodiment, a three-dimensional image capturing apparatus 2, a fluoroscopic image capturing apparatus 3, and an image storage server 4, which are connected in a communicable state via a network 5.

[0023] Computer 1 contains the image processing device according to this embodiment, and the image processing program of this embodiment is installed on it. Computer 1 is installed in a treatment room where treatment is performed on a subject, as will be described later. Computer 1 may be a workstation or personal computer directly operated by the medical professional performing the treatment, or it may be a server computer connected to them via a network. The image processing program is stored in a storage device of the server computer connected to the network, or in network storage, in a state that is accessible from the outside, and is downloaded and installed on Computer 1 used by the physician as needed. Alternatively, it may be recorded on a recording medium such as a DVD (Digital Versatile Disc) or CD-ROM (Compact Disc Read Only Memory) and distributed, and then installed on Computer 1 from that recording medium.

[0024] The 3D imaging device 2 is a device that generates a 3D image representing a part of the subject H by imaging that part of the subject H that is the subject of diagnosis. Specifically, it is a CT scanner, an MRI scanner, or a PET (Positron Emission Tomography) scanner. The 3D image, consisting of multiple tomographic images, generated by the 3D imaging device 2 is transmitted to the image storage server 4 and stored there. In this embodiment, the part of the subject H to be treated is the lung, and the 3D imaging device 2 is a CT scanner. As will be described later, before treatment on subject H, the chest of subject H is imaged, and a CT image including the chest of subject H is acquired in advance as a 3D image and stored in the image storage server 4.

[0025] The fluoroscopy imaging device 3 includes a C-arm 3A, an X-ray source 3B, and an X-ray detector 3C. The X-ray source 3B and the X-ray detector 3C are attached to both ends of the C-arm 3A, respectively. In the fluoroscopy imaging device 3, the C-arm 3A is configured to be rotatable and movable so that the subject H can be photographed from any direction. The fluoroscopy imaging device 3 acquires an X-ray image of the subject H by performing fluoroscopy during treatment of the subject H, as will be described later, by irradiating the subject H with X-rays and detecting the X-rays that have passed through the subject H with the X-ray detector 3C. In the following description, the acquired X-ray image will be referred to as a fluoroscopy image. The fluoroscopy image is an example of a radiographic image according to this disclosure. The fluoroscopy image T0 may be acquired by continuously irradiating the subject H with X-rays at a predetermined frame rate, or, as will be described later, by irradiating the subject H with X-rays at a predetermined timing such that the endoscope 7 reaches the branching of the bronchi.

[0026] The image storage server 4 is a computer that stores and manages various types of data, and is equipped with a large-capacity external storage device and database management software. The image storage server 4 communicates with other devices via a wired or wireless network 5 to send and receive image data, etc. Specifically, it acquires various types of data, including 3D images acquired by the 3D image acquisition device 2 and fluoroscopic images acquired by the fluoroscopic image acquisition device 3, via the network, and stores and manages them on a recording medium such as a large-capacity external storage device. The storage format of the image data and communication between each device via the network 5 are based on protocols such as DICOM (Digital Imaging and Communication in Medicine).

[0027] In this embodiment, a biopsy is performed to examine the presence of disease in detail by excising a portion of a lesion such as a pulmonary nodule present in the lungs of subject H while performing fluoroscopic imaging of the subject H. For this reason, the fluoroscopic imaging device 3 is located in the treatment room for performing the biopsy. An endoscopic ultrasound device 6 is also installed in the treatment room. The endoscopic ultrasound device 6 is equipped with an endoscope 7 with an ultrasound probe and treatment instruments such as forceps attached to its tip. In this embodiment, in order to perform a biopsy of the lesion, the operator inserts the endoscope 7 into the bronchus of subject H, and while taking endoscopic images of the inside of the bronchus with the endoscope 7, a fluoroscopic image of subject H is taken with the fluoroscopic imaging device 3. Then, while displaying the captured fluoroscopic image in real time, the operator confirms the position of the endoscope 7 inside subject H in the fluoroscopic image and moves the tip of the endoscope 7 to the location of the target lesion. The bronchus is an example of a tubular structure in this disclosure.

[0028] Endoscopic images are acquired continuously at a predetermined frame rate. If fluoroscopic images T0 are acquired at a predetermined frame rate, the frame rate at which endoscopic images are acquired may be the same as the frame rate at which fluoroscopic images T0 are acquired. Furthermore, even if fluoroscopic images T0 are acquired at any arbitrary timing, endoscopic images are acquired at a predetermined frame rate.

[0029] Here, lung lesions such as pulmonary nodules occur outside the bronchi, not inside them. Therefore, after moving the endoscope 7 to the target position, the operator takes an ultrasound image of the outside of the bronchi using an ultrasound probe, displays the ultrasound image, and, while confirming the location of the lesion in the ultrasound image, performs a procedure to collect a portion of the lesion using instruments such as forceps.

[0030] Next, an image processing apparatus according to this embodiment will be described. Figure 2 is a diagram showing the hardware configuration of the image processing apparatus according to this embodiment. As shown in Figure 2, the image processing apparatus 10 includes a CPU (Central Processing Unit) 11, non-volatile storage 13, and memory 16 as a temporary storage area. The image processing apparatus 10 also includes a display 14 such as a liquid crystal display, input devices 15 such as a keyboard and mouse, and a network I / F (Interface) 17 connected to a network 5. The CPU 11, storage 13, display 14, input devices 15, memory 16, and network I / F 17 are connected to a bus 18. Note that the CPU 11 is an example of a processor in this disclosure.

[0031] The storage 13 is implemented using an HDD (Hard Disk Drive), SSD (Solid State Drive), flash memory, etc. The image processing program 12 is stored in the storage 13 as a storage medium. The CPU 11 reads the image processing program 12 from the storage 13, expands it into memory 16, and executes the expanded image processing program 12.

[0032] Next, the functional configuration of the image processing apparatus according to this embodiment will be described. Figure 3 is a diagram showing the functional configuration of the image processing apparatus according to this embodiment. As shown in Figure 3, the image processing apparatus 10 includes an image acquisition unit 21, a virtual viewpoint derivation unit 22, a positioning unit 23, and a display control unit 24. When the CPU 11 executes the image processing program 12, the CPU 11 functions as the image acquisition unit 21, the virtual viewpoint derivation unit 22, the positioning unit 23, and the display control unit 24. In this embodiment, a simulation program, which will be described later, is installed on the computer 1. The CPU 11 functions as a simulation unit by executing the simulation program. In Figure 3, the simulation unit 26 is shown.

[0033] The image acquisition unit 21 acquires a 3D image V0 of subject H from the image storage server 4 based on instructions from the operator via the input device 15. The acquired 3D image V0 is taken before the procedure on subject H. The image acquisition unit 21 also acquires a fluoroscopic image T0 taken by the fluoroscopic image acquisition device 3 during the procedure on subject H. Furthermore, the image acquisition unit 21 acquires an endoscopic image R0 taken by the endoscope 7 during the procedure on subject H. The endoscopic image acquired by the endoscope 7 is obtained by actually photographing the inside of subject H's bronchi with the endoscope 7. For this reason, in the following explanation, the endoscopic image acquired by the endoscope 7 will be referred to as the actual endoscopic image R0. The actual endoscopic image R0 is acquired at a predetermined frame rate, regardless of how the fluoroscopic image T0 is acquired. Therefore, the actual endoscopic image R0 is acquired at a timing close to the timing when the fluoroscopic image T0 is acquired. Consequently, there is an actual endoscopic image R0 that corresponds to the acquisition timing of the fluoroscopic image T0.

[0034] In this embodiment, the simulation unit 26 performs a simulation of the procedure using the endoscope 7 with a 3D image V0 before the procedure, and uses the simulation results to guide the endoscope 7 to the target point after insertion into the bronchus. The simulation will be described below. During the simulation, the simulation unit 26 extracts the bronchial region from the 3D image V0. In this embodiment, the simulation unit 26 extracts the bronchial region from the 3D image V0 using a known computer-aided diagnosis (CAD) algorithm. Alternatively, the simulation unit 26 may extract the bronchial region included in the 3D image V0 using any method described, for example, in Japanese Patent Application Publication No. 2010-220742.

[0035] The simulation unit 26 then displays the extracted bronchial region on the display 14. The operator sets a path from the entrance of the bronchus to the target point where the lesion is located within the displayed bronchial region. It is preferable to set a starting point for the navigation. For example, the first branching point of the bronchus can be used as the starting point for the navigation. Then, the points where the bronchus branches along the path from the starting point to the target point are set as waypoints. The simulation unit 26 then generates virtual endoscopic images at the starting point, waypoints, and target point.

[0036] A virtual endoscopic image is a virtual image that depicts the inner wall of the bronchus as seen from each viewpoint. In this embodiment, the virtual endoscopic image is derived using, for example, the method described in Japanese Patent Application Publication No. 2020-010735. Specifically, a projected image is generated by performing a central projection, which projects a plurality of three-dimensional images V0 extending radially in the direction of the endoscope's line of sight from the viewpoints at the starting point, passing point, and target point onto a predetermined projection plane. This projected image becomes a virtual endoscopic image that is virtually generated as if it were taken at the position of the endoscope 7. As for the specific method of central projection, for example, known volume rendering methods can be used. Furthermore, the vertical direction of the virtual endoscopic image VG0 can be determined based on the orientation of the viewpoint corresponding to each position (i.e., the twist of the endoscope 7). Therefore, the simulation unit 26 identifies the viewpoint of the virtual endoscopic image at the starting point, passing point, and target point, i.e., the position and orientation of the virtual endoscope, and generates a virtual endoscopic image according to the position and orientation of the virtual endoscope.

[0037] The simulation unit 26 displays an information generation screen on the display 14 for generating navigation information when inserting the endoscope 7 into the bronchus. Figure 4 shows the information generation screen. As shown in Figure 4, the information generation screen 30 includes a display area 31 for bronchial region images and a display area 32 for virtual endoscopic image VG0. On the information generation screen 30, the operator sets the insertion path 35 of the endoscope using the input device 15, and sets the navigation start point P0, passing points P1, P2, and target point Pt in the insertion path 35. The target point Pt is a location in the bronchus where a lesion 37 exists laterally. The virtual endoscopic image VG0 displayed in the display area 32 is a virtual endoscopic image generated at the start point P0, passing points P1, P2, and target point Pt, and the virtual endoscopic images of the start point P0, passing points P1, P2, and target point Pt can be switched by instructions from the input device 15.

[0038] Here, the insertion direction of the endoscope 7 into the bronchus is from the mouth or nose toward the end of the bronchus. Therefore, at each position within the bronchial region in the 3D image V0, the orientation of the endoscope 7 at that position, i.e., the direction of the viewpoint, can be determined. Furthermore, the method of inserting the endoscope 7 into the subject H is predetermined. For example, at the start of insertion of the endoscope 7, the method of insertion is predetermined so that the ventral side of the subject H is on the upper side of the actual endoscopic image. Therefore, at the derived viewpoint position, the degree to which the endoscope 7 is twisted around its long axis can be derived in advance by simulation based on the shape of the bronchial region. Accordingly, the simulation unit 26 derives the viewpoint orientation (i.e., the direction of the line of sight and the twist) at the starting point P0, passing points P1, P2, and target point Pt.

[0039] The simulation unit 26 saves the results of the simulation performed as described above to the storage 13. Then, when the procedure is performed with the endoscope 7, the simulation unit 26 reads the simulation results saved in the storage 13 and displays the navigation screen, which will be described later, on the display 14.

[0040] The virtual viewpoint derivation unit 22 uses the actual endoscope image R0 and the 3D image V0 to derive a virtual viewpoint in the 3D image V0 that corresponds to the viewpoint in the actual endoscope image R0. To this end, the virtual viewpoint derivation unit 22 determines whether the endoscope 7 has reached the simulation starting point P0 based on the actual endoscope image R0 and the virtual endoscope image VG0 of the starting point P0. In making this determination, the virtual viewpoint derivation unit 22 derives a depth map DM1 of the actual endoscope image R0 and a depth map DM2 of the virtual endoscope image VG0 of the starting point P0, and derives the similarity between depth map DM1 and depth map DM2.

[0041] For deriving the depth map, for example, the method described in "Unsupervised Learning of Depth and Ego-Motion from Video," Tinghui Zhou et al., April 2017, can be used. Figure 5 is a diagram illustrating the method of Zhou et al. As shown in Figure 5, the Zhou et al. paper discloses a method for training a first pre-trained model 41 for deriving the depth map and a second pre-trained model 42 for deriving the change in viewpoint. The virtual viewpoint derivation unit 22 derives the depth map using the first pre-trained model 41 trained by the method described in the Zhou et al. paper. The first pre-trained model 41 is constructed by machine learning a neural network to derive a depth map representing the distribution of distances in the depth direction of a single frame that makes up a video.

[0042] The second pre-trained model 42 is constructed by machine learning a neural network to derive the change in viewpoint between two frames from two frames that make up a video. The change in viewpoint is the amount of translation t of the viewpoint between frames and the amount of change in posture, i.e., rotation K.

[0043] In Zhou et al.'s method, the first pre-trained model 41 and the second pre-trained model 42 are trained simultaneously without using training data, based on the relationship that the change in viewpoint and the depth map must satisfy across multiple frames. Alternatively, the first pre-trained model 41 may be constructed using a large amount of training data consisting of training images and depth maps as ground truth data for those training images, without using Zhou et al.'s method. Furthermore, the second pre-trained model 42 may be constructed using a large amount of training data consisting of combinations of two training images and the change in viewpoint of those two ground truth images.

[0044] The virtual viewpoint derivation unit 22 then uses the first trained model 41 to derive a first depth map DM1 representing the distribution of distances in the depth direction of the actual endoscope image R0, and derives a second depth map DM2 representing the distribution of distances in the depth direction of the virtual endoscope image VG0 from the virtual endoscope image VG0 at the starting point P0. Note that for the second depth map DM2, the simulation unit 26 may pre-derive a depth map of the bronchial structure when the viewpoint is set to the starting point P0, using the bronchial region extracted from the 3D image V0, and use this derived depth map as the second depth map DM2. Next, the virtual viewpoint derivation unit 22 derives the similarity between the first depth map DM1 and the second depth map DM2. For example, the correlation value between the first depth map DM1 and the second depth map DM2 can be used as the similarity. The virtual viewpoint derivation unit 22 then determines that the endoscope 7 has reached the starting point P0 when the similarity becomes equal to or greater than a predetermined threshold Th1. At the moment the endoscope reaches the starting point P0, the viewpoint of the starting point P0 becomes the virtual viewpoint VP0 of the 3D image V0, which corresponds to the actual endoscope image R0.

[0045] The starting point P0 is the location where the first bronchial branch appears, and until the endoscope 7 reaches the vicinity of the starting point P0, there is only one hole in the actual endoscopic image R0. On the other hand, the virtual endoscopic image VG0 of the starting point P0 has two holes. For this reason, the virtual viewpoint derivation unit 22 may derive the similarity between the actual endoscopic image R0 and the virtual endoscopic image VG0 each time the actual endoscopic image R0 is acquired, and determine that the endoscope 7 has reached the starting point P0 when the similarity becomes equal to or greater than a predetermined threshold Th1.

[0046] After the endoscope 7 reaches the starting point P0, the virtual viewpoint deriving unit 22 derives a virtual viewpoint of the endoscope 7 within the 3D image V0 using the actual endoscope image R0 and the 3D image V0. The virtual viewpoint is a virtual viewpoint that corresponds to the viewpoint of the actual endoscope image R0 acquired within the 3D image V0.

[0047] Figure 6 schematically illustrates the derivation of the virtual viewpoint. As shown in Figure 6, first, the virtual viewpoint derivation unit 22 uses the actual endoscopic image R0 and the newly acquired actual endoscopic image R1 to derive the change in viewpoint from the actual endoscopic image R0 to the new actual endoscopic image R1.

[0048] The virtual viewpoint derivation unit 22 uses the second trained model 42 described in the literature by Zhou et al. shown in Figure 5 above to derive the change in viewpoint between the actual endoscopic image R0 and the new actual endoscopic image R1. The change in viewpoint is derived as the amount of translation t and rotation K of the viewpoint from the actual endoscopic image R0 to the new actual endoscopic image R1.

[0049] The virtual viewpoint derivation unit 22 then uses the derived viewpoint change to convert the virtual viewpoint VP0 corresponding to the actual endoscope image R0, thereby deriving a new virtual viewpoint VP1. In this embodiment, deriving the virtual viewpoint VP0 means deriving the three-dimensional position of the virtual viewpoint in the three-dimensional image V0, and the orientation of the viewpoint (i.e., the direction and twist of the line of sight). When a new actual endoscope image R1 is acquired, the virtual viewpoint derivation unit 22 uses the new actual endoscope image R1 and the previously acquired actual endoscope image R0 to derive a new virtual viewpoint for the new actual endoscope image R1. The virtual viewpoint derivation unit 22 updates the virtual viewpoint until the endoscope 7 reaches the target point Pt. Whether or not the endoscope 7 has reached the target point Pt can be determined by determining whether or not the similarity between the depth map of the actual endoscope image R0 and the depth map of the virtual endoscope image VGt at the target point Pt is equal to or greater than the threshold Th1, similar to how the determination of whether or not the actual endoscope image R0 has reached the starting point P0 is made.

[0050] The simulation unit 26 derives a virtual endoscopic image at the virtual viewpoint derived by the virtual viewpoint derivation unit 22 in conjunction with the update of the virtual viewpoint. The display control unit 24 displays the virtual endoscopic image derived in conjunction with the update of the virtual viewpoint on the navigation screen, which will be described later.

[0051] The alignment unit 23 aligns the acquired fluoroscopic image T0 with the 3D image V0. The alignment of the fluoroscopic image T0 and the 3D image V0 is also performed using the fluoroscopic image T0 acquired after the endoscope 7 reaches the starting point P0. Furthermore, the fluoroscopic image T0 that the alignment unit 23 aligns is acquired at a timing corresponding to the actual endoscope image R0 used by the virtual viewpoint derivation unit 22 when deriving the virtual viewpoint.

[0052] First, the alignment unit 23 detects the position of the endoscope 7 from the fluoroscopic image T0. Figure 7 is a diagram showing the fluoroscopic image. As shown in Figure 7, the fluoroscopic image T0 includes an image 45 of the endoscope 7. The alignment unit 23 detects the tip 46 of the endoscope image 45 from the fluoroscopic image T0 using a trained model that has been trained to detect the tip 46 of the endoscope image 45 from the fluoroscopic image T0. Note that the detection of the tip 46 of the endoscope image 45 from the fluoroscopic image T0 is not limited to this. Any method, such as a method using template matching, can be used. The tip 46 of the endoscope image 45 detected in this way is the position of the endoscope 7 and becomes the viewpoint position of the actual endoscope image R0 corresponding to the fluoroscopic image T0.

[0053] Here, the fluoroscopic image T0 is a two-dimensional image. Therefore, the alignment unit 23 aligns the two-dimensional fluoroscopic image T0 with the three-dimensional image V0. In this embodiment, the alignment unit 23 first projects the virtual viewpoint VP0 derived from the three-dimensional image V0 according to provisional fluoroscopic projection conditions to derive a projected virtual viewpoint VTP0. The provisional fluoroscopic projection conditions include the fluoroscopic projection viewpoint, the projection direction, and the distance from the fluoroscopic projection viewpoint to the projection plane. Here, the shooting direction when acquiring the fluoroscopic image T0 is predetermined in the fluoroscopic image acquisition device 3 as from the ventral side to the dorsal side. Also, the distance between the X-ray source 3B and the X-ray detector 3C in the fluoroscopic image acquisition device 3 is fixed. Therefore, the projection direction in the provisional fluoroscopic projection conditions is set to coincide with the shooting direction of the fluoroscopic image T0. Also, the distance from the fluoroscopic projection viewpoint to the projection plane in the provisional fluoroscopic projection conditions is set to the distance between the X-ray source 3B and the X-ray detector 3C in the fluoroscopic image acquisition device 3. The projection plane will be the plane on which the perspective image T0 exists.

[0054] The fluoroscopic projection viewpoint among the provisional fluoroscopic projection conditions is set so that the position where the virtual viewpoint VP0 derived by the virtual viewpoint derivation unit 22 in the 3D image V0 is projected onto the projection plane coincides with the position of the endoscope 7 in the fluoroscopic image T0 (i.e., the tip 46 of the endoscope image 45). At this time, the alignment unit 23 changes the fluoroscopic projection viewpoint in a 2D manner in a plane orthogonal to the projection direction, projects the virtual viewpoint VP0 from the provisional fluoroscopic projection viewpoint onto the projection plane, i.e., the fluoroscopic image T0, and sets the provisional fluoroscopic projection viewpoint where the position of the projected virtual viewpoint coincides with the position of the endoscope 7 in the fluoroscopic image T0 as the fluoroscopic projection viewpoint. Note that "the position where the virtual viewpoint VP0 is projected onto the projection plane coincides with the position of the endoscope 7 in the fluoroscopic image T0" does not mean that they coincide perfectly, but rather that they are close enough that they can be considered to coincide with the position where the virtual viewpoint VP0 is projected onto the projection plane and the position of the endoscope 7 in the fluoroscopic image T0.

[0055] The alignment unit 23 adjusts the provisional perspective projection conditions so that the pseudo-perspective image VT0 derived by projecting the 3D image V0 onto the projection surface according to the set provisional perspective projection conditions matches the perspective image T0. At this time, the alignment unit 23 extracts specific anatomical structures such as bones from the 3D image V0 and the perspective image T0, projects the specific anatomical structures extracted from the 3D image V0 according to the provisional perspective projection conditions, and adjusts the provisional perspective projection conditions so that the projected anatomical structures match the anatomical structures in the perspective image T0. Then, the alignment unit 23 sets the adjusted provisional perspective projection conditions as the final perspective projection conditions.

[0056] Furthermore, the alignment unit 23 may perform non-rigid alignment between the pseudo-fluoroscopic image VT0 and the fluoroscopic image T0 when matching the position of the virtual viewpoint VP0 projected onto the projection plane with the position of the endoscope 7 in the fluoroscopic image T0, and / or when matching the projected anatomical structure with the anatomical structure in the fluoroscopic image T0. In this case, the alignment unit 23 extracts the lung field region from the pseudo-fluoroscopic image VT0 and the fluoroscopic image T0 derived from the determined fluoroscopic projection conditions, derives the amount of deformation of the pseudo-fluoroscopic image VT0 relative to the fluoroscopic image T0 so that the lung field region extracted from the pseudo-fluoroscopic image VT0 matches the lung field region extracted from the fluoroscopic image T0, and performs non-rigid alignment by deforming the pseudo-fluoroscopic image VT0 based on the amount of deformation.

[0057] The display control unit 24 displays the navigation screen on the display 14. Figure 8 shows the navigation screen. As shown in Figure 8, the navigation screen 50 displays an image of the bronchial region 51, a fluoroscopic image T0, a real endoscopy image R0, and a virtual endoscopy image VG0. The image of the bronchial region 51 displays a path 52 for navigating the endoscope 7 to the target point Pt. The current position 53 of the endoscope 7 is also shown on the path 52. In Figure 8, paths 52 that the endoscope 7 has already traversed are shown with a solid line, and paths 52 that have not yet been traversed are shown with a dashed line.

[0058] In this embodiment, the display control unit 24 identifies at least a portion of the bronchi included in the 3D image V0 to be projected onto the fluoroscopic image T0, based on the position and orientation of the virtual viewpoint derived by the virtual viewpoint derivation unit 22. Then, it superimposes the identified portion of the bronchi onto the fluoroscopic image T0. In this embodiment, the display control unit 24 identifies the bronchi in the 3D image V0 that are on the side of the direction of travel of the endoscope 7, i.e., the direction the virtual viewpoint is facing, from the position of the virtual viewpoint, i.e., the current position 53 of the endoscope 7, as a portion of the bronchi to be projected onto the fluoroscopic image T0. In Figure 8, only the bronchi in the 3D image V0 that are on the side of the direction of travel of the endoscope 7, i.e., from the position of the virtual viewpoint, i.e., the current position 53 of the endoscope 7, are superimposed onto the fluoroscopic image T0. Alternatively, as shown in Figure 9, only the bronchi that the endoscope 7 passes through until it reaches the lesion 37 in the direction of travel may be superimposed onto the fluoroscopic image T0. In Figures 8 and 9, the superimposed display of the bronchi is shown by adding hatching to the bronchi included in the fluoroscopic image T0.

[0059] Next, the processing performed in this embodiment will be described. Figure 10 is a flowchart of the processing performed in this embodiment. It is assumed that the 3D image V0 is acquired by the image acquisition unit 21 and stored in the storage 13, and that the navigation information of the endoscope 7 is generated by the simulation unit 26 and stored in the storage 13. First, the image acquisition unit 21 acquires the fluoroscopic image T0 and the actual endoscope image R0 (image acquisition: step ST1). Next, the virtual viewpoint derivation unit 22 uses the actual endoscope image R0 and the 3D image V0 to derive the virtual viewpoint VP0 within the 3D image V0 of the endoscope 7 (step ST2).

[0060] Next, the alignment unit 23 aligns the acquired fluoroscopic image T0 with the 3D image V0 (step ST3). Then, the display control unit 24 superimposes at least a portion of the bronchi included in the 3D image V0 onto the fluoroscopic image T0 based on the virtual viewpoint and the alignment result (step ST4), and returns to step ST1.

[0061] Thus, in this embodiment, a virtual viewpoint VP0 within the 3D image V0 of the endoscope 7 is derived using the actual endoscopic image R0 and the 3D image V0. The fluoroscopic image T0 and the 3D image V0 are then aligned, and at least a portion of the bronchi included in the 3D image V0 are superimposed onto the fluoroscopic image T0 based on the virtual viewpoint and alignment results. Therefore, even without detecting the tip of the endoscope 7 using a sensor, the endoscope 7 can be navigated to a desired position within the subject H based on the 3D image V0 superimposed on the fluoroscopic image T0.

[0062] In the above embodiments, the application of the image processing device of the present disclosure to the observation of bronchi was described, but the present disclosure is not limited to this, and can also be applied to the observation of tubular structures such as the stomach, large intestine, and blood vessels using an endoscope.

[0063] Furthermore, in the above embodiment, the hardware structure of the Processing Unit, which executes various processes such as the image acquisition unit 21, the virtual viewpoint derivation unit 22, the alignment unit 23, and the display control unit 24, can be the various processors shown below. As mentioned above, these various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as a Programmable Logic Device (PLD), which is a processor whose circuit configuration can be changed after manufacturing, such as an FPGA (Field Programmable Gate Array), and a dedicated electrical circuit, which is a processor with a circuit configuration specifically designed to execute a particular process, such as an ASIC (Application Specific Integrated Circuit).

[0064] A single processing unit may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA). Alternatively, multiple processing units may be composed of a single processor.

[0065] Examples of configuring multiple processing units with a single processor include, firstly, a configuration where one or more CPUs and software combine to form a single processor, as exemplified by client and server computers, and this processor functions as multiple processing units. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple processing units, on a single IC (Integrated Circuit) chip, as exemplified by System-on-a-Chip (SoC). Thus, various processing units are configured, in terms of hardware structure, using one or more of the above-mentioned processors.

[0066] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits (Circuitry) that combine circuit elements such as semiconductor devices. [Explanation of Symbols]

[0067] 1 Computer 2. 3D image acquisition device 3. Fluoroscopy imaging device 3A Arm 3B X-ray source 3C X-ray detector 4 Image storage server 5 Network 6. Ultrasound Endoscope 7 Endoscopy 10 Image Processing Device 11 CPU 12 Image Processing Programs 13 Storage 14 displays 15 Input Devices 16 memory 21 Image acquisition unit 22 Virtual viewpoint derivation unit 23 Alignment section 24 Display Control Unit 26 Simulation Department 30 Information generation screen 31,32,33 Display area 41. First pre-trained model 42. Second pre-trained model 45 Endoscopic images 46 Tip 50 Navigation screen 51 Images of the bronchial region 52 routes 53 positions R0, R1 Endoscopic Images T0 fluoroscopic image VG0, VG1 virtual endoscopic images

Claims

1. Equipped with at least one processor, The aforementioned processor, We acquire a 3D image of the subject, An actual endoscopic image of the tubular structure of the subject is obtained by taking an image of the tubular structure of the subject using an endoscope inserted into the tubular structure of the subject. Using the actual endoscope image and the three-dimensional image, a virtual viewpoint of the endoscope within the three-dimensional image is derived. The endoscope is inserted into the tubular structure and acquires radiographic images of the subject. The aforementioned radiation image and the aforementioned three-dimensional image are aligned, An image processing device that superimposes at least a portion of the tubular structure included in the three-dimensional image onto the radiographic image based on the virtual viewpoint and the alignment results.

2. The processor detects the position of the endoscope from the radiographic image, The image processing apparatus according to claim 1, which performs the alignment by setting the projection conditions of the three-dimensional image such that the projection position when the virtual viewpoint is projected onto the radiation image coincides with the position of the endoscope.

3. The image processing apparatus according to claim 2, wherein the processor further sets the projection conditions such that a specific anatomical structure included in the radiographic image matches the specific anatomical structure included in the projected image derived by projecting the three-dimensional image onto the radiographic image.

4. The image processing apparatus according to claim 2 or 3, wherein the alignment further includes non-rigid alignment.

5. The processor identifies at least a portion of the tubular structures included in the three-dimensional image to be projected onto the radiographic image, based on the position and orientation of the virtual viewpoint. The image processing apparatus according to claim 1, which superimposes at least a portion of the identified tubular structure onto the radiographic image.

6. We acquire a 3D image of the subject, An actual endoscopic image of the tubular structure of the subject is obtained by taking an image of the tubular structure of the subject using an endoscope inserted into the tubular structure of the subject. Using the actual endoscope image and the three-dimensional image, a virtual viewpoint of the endoscope within the three-dimensional image is derived. The endoscope is inserted into the tubular structure and acquires radiographic images of the subject. The aforementioned radiation image and the aforementioned three-dimensional image are aligned, An image processing method that superimposes at least a portion of the tubular structure included in the three-dimensional image onto the radiographic image based on the virtual viewpoint and the alignment results.

7. Procedure for acquiring a 3D image of the subject, A procedure for obtaining a real endoscopic image of the tubular structure of the subject, taken by an endoscope inserted into the tubular structure of the subject, A procedure for deriving a virtual viewpoint of the endoscope within the three-dimensional image using the actual endoscope image and the three-dimensional image, A procedure for acquiring radiographic images of the subject with the endoscope inserted into a tubular structure, A procedure for aligning the aforementioned radiographic image with the aforementioned three-dimensional image, An image processing program that causes a computer to perform the steps of superimposing at least a portion of the tubular structure included in the three-dimensional image onto the radiographic image based on the virtual viewpoint and the results of the alignment.

Images