Image processing device, method of operating the image processing device, and image processing program

Abstract

The image processing device, method, and program are designed to facilitate the understanding of the positional relationship between the tip of the endoscope and the lesion. [Solution] The processor sequentially acquires multiple radiographic images of a subject to which an ultrasound endoscope, to which an ultrasound imaging device and radiopaque markers are attached, is inserted into the body cavity. It sequentially acquires multiple two-dimensional ultrasound images corresponding to each of the multiple radiographic images acquired by the ultrasound imaging device. Based on the markers contained in each of the multiple radiographic images, the processor recognizes the position and orientation of the ultrasound endoscope within the body cavity. Based on the position and orientation of the ultrasound endoscope recognized with respect to the multiple radiographic images, it derives a three-dimensional ultrasound image from the multiple two-dimensional ultrasound images.

Description

Technical Field

[0001] The present disclosure relates to an image processing apparatus, an operating method of the image processing apparatus, and an image processing program.

Background Art

[0002] An endoscope is inserted into a lumen such as a bronchus or a digestive organ of a subject, and an endoscopic image of the lumen is acquired to observe the inside of the lumen. In addition, a biopsy procedure is also performed in which tissue at a site where a lesion discovered in the endoscopic image is suspected is collected by a treatment tool such as forceps attached to the tip of the endoscope. When performing a procedure using such an endoscope, it is important to accurately reach the target position in the subject. For this reason, fluoroscopic imaging is performed in which radiation is continuously irradiated from a radiation source to the subject during the procedure, and the fluoroscopic image thus obtained is displayed in real time, so as to grasp the positional relationship between the endoscope and the human body structure. However, it is difficult to grasp the depth inside the subject from the fluoroscopic image. In addition, when the lesion is small, it may be difficult to see in the endoscopic image, so the success rate of collecting the tissue of the lesion is low.

[0003] For this reason, a small ultrasonic observation device is attached to the tip of the endoscope, and a lesion outside the wall is confirmed by ultrasonic waves from the inside of the bronchus, and the tissue is collected while confirming whether the treatment tool for collecting the tissue hits the lesion. However, even when using such an endoscope, since the positional relationship between the treatment tool and the endoscope is confirmed using a fluoroscopic image, it is difficult to completely grasp these positional relationships and collect the tissue.

[0004] In order to solve such problems, a marker made of a material that does not transmit radiation is provided at the tip of the endoscope, and the position and orientation of the endoscope are grasped using the marker image included in the fluoroscopic image (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0005] [Patent Document 1] Special Publication No. 2010-522597 [Overview of the project] [Problems that the invention aims to solve]

[0006] In the method described in Patent Document 1, it is easy to determine the position and orientation of the endoscope in the fluoroscopic image, but the relationship between the position of the lesion and the position of the endoscope remains unclear.

[0007] This disclosure is made in view of the above circumstances and aims to facilitate understanding of the positional relationship between the tip of the endoscope and the lesion. [Means for solving the problem]

[0008] The image processing apparatus according to this disclosure comprises at least one processor, which sequentially acquires multiple radiographic images of a subject to which an ultrasound endoscope, to which an ultrasound imaging device and radiopaque markers are attached, is inserted into a body cavity. Multiple 2D ultrasound images corresponding to each of the multiple radiographic images obtained by the ultrasound imaging device are acquired sequentially. Based on markers contained in each of multiple radiographic images, the position and orientation of the ultrasound endoscope within the body cavity are recognized. Based on the recognized position and orientation of the endoscope in multiple radiographic images, a three-dimensional ultrasound image is derived from multiple two-dimensional ultrasound images.

[0009] Furthermore, in the image processing apparatus according to this disclosure, the processor aligns the radiographic image and the 3D ultrasound image, This may involve superimposing a aligned 3D ultrasound image onto a radiographic image.

[0010] Furthermore, in the image processing apparatus according to this disclosure, the processor extracts the body cavity into which the ultrasound endoscope is inserted from a previously acquired 3D image of the subject, The position and orientation of the ultrasound endoscope are corrected according to the shape of the extracted body cavity. A three-dimensional ultrasound image may be derived from multiple two-dimensional ultrasound images based on the corrected position and orientation.

[0011] The image processing method disclosed herein sequentially acquires multiple radiographic images of a subject in which an ultrasound endoscope, to which an ultrasound imaging device and a radiopaque marker are attached, is inserted into a body cavity. Multiple 2D ultrasound images corresponding to each of the multiple radiographic images obtained by the ultrasound imaging device are acquired sequentially. Based on markers contained in each of multiple radiographic images, the position and orientation of the ultrasound endoscope within the body cavity are recognized. Based on the recognized position and orientation of the endoscope in multiple radiographic images, a three-dimensional ultrasound image is derived from multiple two-dimensional ultrasound images.

[0012] The image processing program disclosed herein includes a procedure for sequentially acquiring multiple radiographic images of a subject in which an ultrasound endoscope, to which an ultrasound imaging device and a radiopaque marker are attached, is inserted into a body cavity, and A procedure for sequentially acquiring multiple 2D ultrasound images corresponding to multiple radiographic images obtained by an ultrasound imaging device, A procedure for recognizing the position and orientation of an endoscope in a body cavity based on markers contained in each of multiple radiographic images, The computer is instructed to perform a procedure to derive a 3D ultrasound image from multiple 2D ultrasound images, based on the recognized position and orientation of the ultrasound endoscope in relation to multiple radiographic images. [Effects of the Invention]

[0013] According to this disclosure, the location of lesions included in radiographic images can be easily confirmed by using three-dimensional ultrasound images. [Brief explanation of the drawing]

[0014] [Figure 1] Figure showing the schematic configuration of a medical information system to which an image processing apparatus according to the first embodiment of the present disclosure is applied [Figure 2] Figure showing the tip portion of an endoscope according to the present embodiment [Figure 3] Exploded view of a radiation-opaque marker [Figure 4] Figure showing a state where a radiation-opaque marker is attached [Figure 5] Figure showing changes in an annular marker [Figure 6] Figure showing the schematic configuration of an image processing apparatus according to the first embodiment [Figure 7] [[ID=2(]]Functional configuration diagram of an image processing apparatus according to the first embodiment [Figure 8] Figure for explaining the derivation of a three-dimensional ultrasonic image [Figure 9] Figure for explaining the derivation of the spatial positional relationship of corresponding pixels between ultrasonic images [Figure 10] Figure for explaining the derivation of a three-dimensional ultrasonic image [Figure 11] Figure showing a display screen [Figure 12] Flowchart showing the processes performed in the first embodiment [Figure 13] Functional configuration diagram of an image processing apparatus according to the second embodiment [Figure 14] Flowchart showing the processes performed in the second embodiment [Figure 15] Figure showing another example of the tip portion of an endoscope according to the present embodiment

Embodiments for Carrying Out the Invention

[0015] Embodiments of this disclosure will be described below with reference to the drawings. First, the configuration of a medical information system to which the image processing device according to the first embodiment is applied will be described. Figure 1 is a diagram showing the schematic configuration of a medical information system. In the medical information system shown in Figure 1, a computer 1 containing the image processing device according to the first embodiment, a 3D image acquisition device 2, a fluoroscopy image acquisition device 3, and an image storage server 4 are connected via a network 5 in a state where they can communicate with each other.

[0016] Computer 1 contains the image processing device according to the first embodiment, and the image processing program according to the first embodiment is installed. Computer 1 is installed in a treatment room where treatment is performed on a subject, as described later. Computer 1 may be a workstation or personal computer directly operated by the medical professional performing the treatment, or it may be a server computer connected to them via a network. The image processing program is stored in a storage device of the server computer connected to the network, or in network storage, in a state that is accessible from the outside, and is downloaded and installed on Computer 1 used by the physician upon request. Alternatively, it may be recorded on a recording medium such as a DVD (Digital Versatile Disc) or CD-ROM (Compact Disc Read Only Memory) and distributed, and then installed on Computer 1 from that recording medium.

[0017] The 3D imaging device 2 is a device that generates a 3D image representing a part of the subject H by imaging that part of the subject H that is the subject of diagnosis. Specifically, it is a CT scanner, an MRI scanner, or a PET (Positron Emission Tomography) scanner. The 3D image, consisting of multiple tomographic images, generated by the 3D imaging device 2 is transmitted to the image storage server 4 and stored there. In this embodiment, the part of the subject H to be treated is the lung, and the 3D imaging device 2 is a CT scanner. As will be described later, before treatment on subject H, the chest of subject H is imaged, and a CT image including the chest of subject H is acquired in advance as a 3D image and stored in the image storage server 4.

[0018] The fluoroscopy imaging device 3 comprises a C-arm 3A, an X-ray source 3B, and an X-ray detector 3C. The X-ray source 3B and the X-ray detector 3C are attached to both ends of the C-arm 3A, respectively. In the fluoroscopy imaging device 3, the C-arm 3A is configured to rotate and move so that the subject H can be photographed from any direction. Then, as will be described later, during treatment of the subject H, the fluoroscopy imaging device 3 continuously irradiates the subject H with X-rays at a predetermined frame rate and sequentially detects the X-rays that have passed through the subject H with the X-ray detector 3C, thereby sequentially acquiring X-ray images of the subject H. In the following description, the sequentially acquired X-ray images will be referred to as fluoroscopic images. A fluoroscopic image is an example of a radiation image according to this disclosure. Also, X-rays are an example of radiation according to this disclosure.

[0019] The image storage server 4 is a computer that stores and manages various types of data, and is equipped with a large-capacity external storage device and database management software. The image storage server 4 communicates with other devices via a wired or wireless network 5 to send and receive image data, etc. Specifically, it acquires various types of data via the network, including 3D images acquired by the 3D image acquisition device 2, fluoroscopic images acquired by the fluoroscopy image acquisition device 3, and ultrasound images acquired by the ultrasound endoscopic device 6 (described later), and stores and manages them on a recording medium such as a large-capacity external storage device. The storage format of the image data and communication between each device via the network 5 are based on protocols such as DICOM (Digital Imaging and Communication in Medicine).

[0020] In this embodiment, a biopsy is performed to examine the presence of disease in detail by taking a sample of a lesion such as a pulmonary nodule in the lung of subject H while simultaneously performing fluoroscopic imaging of the subject H. For this reason, the fluoroscopic imaging device 3 is located in the treatment room for performing the biopsy. An endoscopic ultrasound device 6 is also installed in the treatment room. The endoscopic ultrasound device 6 is equipped with an endoscope 7 with an ultrasound probe and a puncture needle and other treatment instruments attached to its tip.

[0021] Figure 2 shows the tip portion of the endoscope 7 according to this embodiment. As shown in Figure 2, a channel 7A is formed at the tip of the endoscope 7 through which a treatment instrument (not shown), such as a puncture needle, enters and exits, and an optical system 7B for acquiring endoscopic images is attached near the exit of channel 7A. Furthermore, an ultrasonic probe 7C is attached at a position further forward than channel 7A. In addition, a radiopaque marker 8 is attached to the tip of the endoscope 7. The ultrasonic endoscope device 6 acquires an ultrasonic image of a cross-section perpendicular to the long axis of the endoscope 7 in the direction that the ultrasonic probe 7C is pointing. The range in which an ultrasonic image can be taken is a predetermined trapezoidal range in which the ultrasonic waves spread from the ultrasonic probe 7C.

[0022] Figure 3 is an unfolded diagram of the radiopaque marker. As shown in Figure 3, the radiopaque marker 8 includes a linear marker 8A and a chessboard marker 8B. By attaching such a marker 8 to the tip of the endoscope 7, the linear marker 8A becomes an annular marker 8C with a portion cut out, as shown in Figure 4. Note that in Figure 4, the white chessboard marker 8B is shown facing the wrong way.

[0023] In this embodiment, in order to perform a biopsy of a lesion, the operator inserts the endoscope 7 into the bronchus of the subject H, captures a fluoroscopic image of the subject H using the fluoroscopic imaging device 3, displays the captured fluoroscopic image in real time, confirms the position of the tip of the endoscope 7 within the subject H in the fluoroscopic image, and moves the tip of the endoscope 7 to the location of the target lesion.

[0024] Here, lung lesions such as pulmonary nodules occur outside the bronchi, not inside them. Therefore, after moving the tip of the endoscope 7 to the target position, the operator uses an ultrasound probe to take ultrasound images from the inside to the outside of the bronchi and displays the ultrasound images. While confirming the location of the lesion in the ultrasound image, the operator uses a treatment instrument to take a sample of the lesion.

[0025] In this case, the position and orientation of the tip of the endoscope 7 can be recognized by how the marker 8 attached to the tip of the endoscope 7 appears in the fluoroscopic image. Regarding orientation, when three spatial axes are set as shown in Figure 4, the annular marker 8C changes as shown in the upper part of Figure 5, "around the y axis," due to the change in orientation caused by the rotation of the tip of the endoscope 7 around the y axis (i.e., in the direction of arrow A1). Also, the annular marker 8C changes as shown in the middle part of Figure 5, "around the x axis," due to the change in orientation caused by the rotation of the tip of the endoscope 7 around the x axis (i.e., in the direction of arrow A2). Furthermore, the annular marker 8C changes as shown in the lower part of Figure 5, "around the z axis," due to the change in orientation caused by the rotation of the tip of the endoscope 7 around the z axis (i.e., in the direction of arrow A3). In addition, by using the chessboard marker 8B as an auxiliary, the orientation of the tip of the endoscope 7 can be recognized with greater accuracy.

[0026] Therefore, when acquiring ultrasound images, the operator can determine the position and orientation of the tip of the endoscope 7 when the lesion is included in the ultrasound image based on the position and shape of the marker 8 included in the fluoroscopic image, and by maintaining that position and bringing the treatment instrument to the lesion, the lesion can be reliably collected.

[0027] On the other hand, when using an endoscope without a treatment instrument attached, after confirming the location of the lesion, the endoscope with the treatment instrument attached is inserted to collect lesion tissue. In this case, if the endoscope with the treatment instrument attached is also marked with the same marker 8, the operator can easily remember the location of the lesion by relying on the marker 8 included in the fluoroscopic image, thus enabling them to reliably collect lesion tissue by inserting the endoscope with the treatment instrument attached to the location of the lesion.

[0028] Next, an image processing apparatus according to the first embodiment will be described. Figure 6 is a diagram showing the hardware configuration of the image processing apparatus according to this embodiment. As shown in Figure 6, the image processing apparatus 10 includes a CPU (Central Processing Unit) 11, non-volatile storage 13, and memory 16 as a temporary storage area. The image processing apparatus 10 also includes a display 14 such as a liquid crystal display, input devices 15 such as a keyboard and mouse, and a network I / F (Interface) 17 connected to a network 5. The CPU 11, storage 13, display 14, input devices 15, memory 16, and network I / F 17 are connected to a bus 18. Note that the CPU 11 is an example of a processor in this disclosure.

[0029] The storage 13 is implemented using an HDD (Hard Disk Drive), SSD (Solid State Drive), flash memory, etc. The image processing program 12 is stored in the storage 13 as a storage medium. The CPU 11 reads the image processing program 12 from the storage 13, expands it into memory 16, and executes the expanded image processing program 12.

[0030] Next, the functional configuration of the image processing apparatus according to the first embodiment will be described. Figure 7 is a diagram showing the functional configuration of the image processing apparatus according to the first embodiment. As shown in Figure 7, the image processing apparatus 10 includes an image acquisition unit 21, a recognition unit 22, an output unit 23, an alignment unit 24, and a display control unit 25. The CPU 11 executes the image processing program 12, and the CPU 11 functions as the image acquisition unit 21, the recognition unit 22, the output unit 23, the alignment unit 24, and the display control unit 25.

[0031] The image acquisition unit 21 sequentially acquires multiple fluoroscopic images T0 acquired by the fluoroscopic image acquisition device 3 during the treatment of the subject H at a predetermined frame rate. The image acquisition unit 21 also sequentially acquires multiple ultrasound images corresponding to each of the multiple fluoroscopic images T0 acquired by the ultrasound endoscopic device 6 at a predetermined frame rate. The ultrasound images acquired by the ultrasound endoscopic device 6 are an example of the two-dimensional ultrasound images of this disclosure. In the following description, unless otherwise specified, the term "ultrasound image" refers to a two-dimensional ultrasound image.

[0032] The recognition unit 22 recognizes the position and orientation of the endoscope 7 within the bronchus based on the image of the marker 8 (hereinafter referred to as the marker image) contained in the fluoroscopic image T0. Since the marker 8 is radiopaque, the marker image appears as a high-brightness (low-density) region in the fluoroscopic image T0. Therefore, it can be detected from the fluoroscopic image T0 using thresholding or a trained model. Here, as shown in Figures 3 to 5, the orientation of the endoscope 7 based on its rotation around three axes within the subject can be recognized based on the annular marker 8C. The position of the annular marker 8C in the fluoroscopic image T0 corresponds to the position of the tip of the endoscope 7. The size of the marker 8 corresponds to the position of the endoscope 7 in the direction perpendicular to the fluoroscopic image T0, i.e., in the depth direction.

[0033] The recognition unit 22 sets one of the sequentially acquired fluoroscopic images T0 as a reference fluoroscopic image Tb, detects a marker image from the reference fluoroscopic image Tb, and recognizes the position and orientation of the marker image. The position and orientation of the marker image in the reference fluoroscopic image Tb are referred to as the reference position and orientation. The reference position can be determined by the operator using the input device 15 to specify, for example, the position of the first bronchial branch or a position near a lesion.

[0034] The recognition unit 22 acquires a reference fluoroscopic image Tb, and then recognizes the position and orientation of the marker image in the sequentially acquired fluoroscopic images T0. As a result, the position and orientation of the endoscope 7 relative to the reference position are sequentially recognized in the sequentially acquired fluoroscopic images T0. The recognition unit 22 may also use a chessboard marker 8B in addition to the annular marker 8C to auxiliaryly recognize the position and orientation of the marker image.

[0035] The derivation unit 23 derives a three-dimensional ultrasound image UV0 from multiple ultrasound images U0 based on the position and orientation of the endoscope 7 recognized with respect to multiple fluoroscopic images T0. Figure 8 is a diagram illustrating the derivation of the three-dimensional ultrasound image UV0. In Figure 8, the dashed line shows the path taken by the endoscope 7 within the bronchus. Figure 8 shows the state in which five ultrasound images U1 to U5 were acquired at predetermined time intervals along the path 30 taken by the endoscope 7. Note that the path 20 of the endoscope in Figure 8 is for illustrative purposes only and does not represent the actual path. Also, the intervals between ultrasound images U1 to U5 are for illustrative purposes only and do not represent the actual intervals. Furthermore, in Figure 8, annular markers 8C included in the fluoroscopic images T1 to T5 acquired when each of the ultrasound images U1 to U5 was acquired are shown in correspondence with the ultrasound images U1 to U5.

[0036] As shown in Figure 8, the position and orientation of the endoscope 7 change as it moves through the bronchus along the path 30. Consequently, the position and orientation of the cross-section within the subject H represented by the ultrasound image change. The position and orientation of the cross-section represented by the ultrasound image correspond to the position and orientation of the marker 8 included in the fluoroscopic image T0. Therefore, the derivation unit 23 derives the spatial positional relationship of corresponding pixels in ultrasound image U1 and ultrasound image U2 from the position and orientation of the endoscope 7 when ultrasound image U1 was acquired and the position and orientation of the endoscope 7 when ultrasound image U2 was acquired, for two ultrasound images (let's call them U1 and U2) acquired at adjacent times.

[0037] Figure 9 is a diagram illustrating the derivation of the spatial positional relationship of corresponding pixels between ultrasound images. As shown in Figure 9, the derivation unit 23 derives the spatial relationship of where each pixel in ultrasound image U1 has moved to in ultrasound image U2, based on the changes in the position and orientation of the endoscope 7 between the acquisition of ultrasound image U1 and the acquisition of ultrasound image U2. In Figure 9, the changes of five pixels in ultrasound image U1 are shown by vectors pointing from ultrasound image U1 to ultrasound image U2.

[0038] Then, the derivation unit 23 derives a three-dimensional ultrasound image UV12, as shown in Figure 10, by interpolating the corresponding pixels of ultrasound image U1 and ultrasound image U2 based on the derived positional relationship.

[0039] The derivation unit 23 derives the 3D ultrasound image UV0 by repeating the above process for ultrasound images acquired at adjacent times.

[0040] The alignment unit 24 aligns the 3D ultrasound image UV0 derived by the derivation unit 23 with the fluoroscopic image T0. To do this, the alignment unit 24 derives a 2D projected ultrasound image UT0 by projecting the 3D ultrasound image UV0 derived from the previously acquired ultrasound image U0 in the direction of the latest fluoroscopic image T0. Any projection method, such as maximum value projection or minimum value projection, can be used.

[0041] The alignment unit 24 then aligns the two-dimensional projected ultrasound image UT0 with the fluoroscopic image T0. Any method can be used for alignment, such as rigid body alignment or non-rigid body alignment.

[0042] The display control unit 25 superimposes the aligned two-dimensional projected ultrasound image UT0 onto the fluoroscopic image T0 and displays it on the display 14. Figure 11 shows the display screen. As shown in Figure 11, the fluoroscopic image T0 is displayed on the display screen 40. The fluoroscopic image T0 includes an image of the endoscope 7. The two-dimensional projected ultrasound image UT0 is superimposed near the tip of the endoscope 7 in the fluoroscopic image T0. It can also be seen that the two-dimensional projected ultrasound image UT0 includes a lesion 41.

[0043] Next, the process performed in the first embodiment will be described. Figure 12 is a flowchart showing the process performed in the first embodiment. First, the image acquisition unit 21 acquires a fluoroscopic image T0 and an ultrasound image U0 (image acquisition: step ST1). Next, the recognition unit 22 recognizes the position and orientation of the endoscope 7 in the bronchus based on the marker image contained in the fluoroscopic image T0 (step ST2). Subsequently, the output unit 23 derives a three-dimensional ultrasound image UV0 from a plurality of ultrasound images U0 based on the position and orientation of the endoscope 7 recognized with respect to a plurality of fluoroscopic images T0 (step ST3).

[0044] Then, the alignment unit 24 aligns the 3D ultrasound image UV0 with the latest fluoroscopic image T0 (step ST4), and the display control unit 25 superimposes the aligned 3D ultrasound image UV0, i.e., the 2D projected ultrasound image UT0, onto the fluoroscopic image T0 (step ST5), and returns to step ST1.

[0045] Thus, in this embodiment, a three-dimensional ultrasound image UV0 is derived from multiple ultrasound images U0 based on the position and orientation of the endoscope 7 recognized in relation to multiple fluoroscopic images T0. By using such a three-dimensional ultrasound image UV0, the position of lesions included in the fluoroscopic images T0 can be easily confirmed.

[0046] In particular, by superimposing the 3D ultrasound image UV0 onto the fluoroscopic image T0, the positional relationship between the tip of the endoscope 7 included in the fluoroscopic image T0 and the lesion included in the 3D ultrasound image UV0 can be easily grasped. Therefore, when collecting tissue from a lesion for biopsy, the accuracy of tissue collection from the lesion can be improved based on the positional relationship between the tip of the endoscope 7 included in the fluoroscopic image T0 and the lesion included in the 3D ultrasound image UV0.

[0047] Next, a second embodiment of the present disclosure will be described. Figure 13 is a diagram showing the functional configuration of an image processing apparatus according to the second embodiment of the present disclosure. In Figure 13, the same reference numerals are used for components identical to those in Figure 7, and detailed explanations are omitted. As shown in Figure 13, the image processing apparatus 10A according to the second embodiment differs from the first embodiment in that it further includes an extraction unit 26 and a correction unit 27.

[0048] In the second embodiment, the image acquisition unit 21 acquires a three-dimensional image V0 of the subject H from the image storage server 4 based on instructions from the operator via the input device 15 before the procedure.

[0049] The extraction unit 26 extracts the body cavity into which the endoscope 7 is inserted from the 3D image V0. In the second embodiment, since the endoscope 7 is inserted into the bronchi, the extraction unit 26 extracts the bronchi from the 3D image V0. For this purpose, the extraction unit 26 extracts the lung region from the 3D image V0. Any method can be used to extract the lung region, such as a method that histograms the signal values ​​for each pixel in the 3D image V0 and extracts the lungs by thresholding, or a region growing method based on seed points representing the lungs. In addition, a classifier that has been machine-learned to extract the lung region may be used.

[0050] The extraction unit 26 then extracts the graph structure of the bronchial region included in the lung region extracted from the 3D image V0 as a 3D bronchial region. As a method for extracting the bronchial region, for example, a method can be used in which the graph structure of the bronchi is extracted using a Hessian matrix, the extracted graph structure is classified into start points, endpoints, branching points and edges, and the bronchial region is extracted by connecting the start points, endpoints and branching points with edges, as described in Japanese Patent Application Publication No. 2010-220742. However, the method for extracting the bronchial region is not limited to this.

[0051] The correction unit 27 corrects the position and orientation of the endoscope 7 according to the shape of the extracted bronchus. To do this, the correction unit 27 performs a process to match the coordinate system of the 3D image V0 with the coordinate system of the tip position of the endoscope 7. For example, the coordinate system of the 3D image V0 and the coordinate system of the tip position of the endoscope 7 are matched by performing a coordinate transformation on the coordinates (3D) of the tip position of the endoscope 7 so that the coordinate system of the endoscope 7 matches the coordinate system of the 3D image V0.

[0052] The correction unit 27 then determines whether the tip of the endoscope 7 is located inside the bronchi. If the tip of the endoscope 7 is not located inside the bronchi, it corrects the recognized position and orientation of the endoscope 7 so that the tip of the endoscope 7 is located inside the bronchi. On the other hand, if the tip of the endoscope 7 is located inside the bronchi, the correction unit 27 does not correct the position and orientation of the endoscope 7.

[0053] If the position and orientation of the endoscope 7 are corrected, the output unit 23 outputs the 3D ultrasound image UV0 based on the corrected position and orientation of the endoscope. If the position and orientation of the endoscope 7 are not corrected, the recognition unit 22 outputs the 3D ultrasound image UV0 based on the position and orientation of the endoscope recognized by the recognition unit 22.

[0054] Next, the processing performed in the second embodiment will be described. Figure 14 is a flowchart showing the processing performed in the second embodiment. First, the image acquisition unit 21 acquires a fluoroscopic image T0 and an ultrasound image U0 in addition to a three-dimensional image V0 (image acquisition: step ST11). Then, the extraction unit 26 extracts the bronchial region from the three-dimensional image V0 (step ST12). Next, the recognition unit 22 recognizes the position and orientation of the endoscope 7 within the bronchi based on the marker image included in the fluoroscopic image T0 (step ST13). Subsequently, the correction unit 27 performs a process to match the coordinate system of the three-dimensional image V0 with the coordinate system of the endoscope's position (step ST14), and determines whether the tip of the endoscope 7 is inside the bronchi or not (step ST15).

[0055] If step ST15 is rejected, the correction unit 27 corrects the recognized position and orientation of the endoscope 7 (step ST16). Subsequently, the output unit 23 derives a three-dimensional ultrasound image UV0 from multiple ultrasound images U0 based on the corrected position and orientation of the endoscope 7 with respect to multiple fluoroscopic images T0 (step ST17).

[0056] If step ST15 is affirmed, the process proceeds to step ST17, in which the derivation unit 23 derives a three-dimensional ultrasound image UV0 from a plurality of ultrasound images U0 based on the position and orientation of the endoscope 7 recognized with respect to a plurality of fluoroscopic images T0.

[0057] Then, the alignment unit 24 aligns the 3D ultrasound image UV0 with the latest fluoroscopic image T0 (step ST18), and the display control unit 25 overlays the aligned 3D ultrasound image UV0, i.e., the 2D projected ultrasound image UT0, onto the fluoroscopic image T0 (step ST19), and returns to step ST11.

[0058] Thus, in the second embodiment, the position and orientation of the endoscope are corrected when the endoscope is not located within the bronchi, thereby improving the accuracy of endoscope position recognition. Consequently, the positional relationship between the tip of the endoscope 7 included in the fluoroscopic image T0 and the lesion included in the 3D ultrasound image UV0 can be accurately determined, and as a result, the accuracy of tissue sampling from the lesion can be improved.

[0059] In addition, in each of the above embodiments, an endoscope having an ultrasound probe 7D capable of capturing ultrasound images over the entire circumference may be used, as shown in Figure 15. When an endoscope 7 having such an ultrasound probe 7D is used, a circular ultrasound image U10 is acquired, as shown in Figure 15. As a result, a three-dimensional ultrasound image UV0, which has a three-dimensional shape like a deformed cylinder, is derived.

[0060] Furthermore, while the above embodiments describe the process for collecting lung lesions using a bronchoscope, the invention is not limited to this. For example, the image processing apparatus according to this embodiment can also be applied when inserting an ultrasound endoscope into a digestive organ such as the stomach to perform a biopsy of tissue such as the pancreas or liver.

[0061] Furthermore, in each of the above embodiments, the hardware structure of the Processing Unit that performs various processes such as the image acquisition unit 21, recognition unit 22, output unit 23, alignment unit 24, display control unit 25, extraction unit 26, and correction unit 27 can be the various processors shown below. As mentioned above, these various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as a Programmable Logic Device (PLD), which is a processor whose circuit configuration can be changed after manufacturing, such as an FPGA (Field Programmable Gate Array), and a dedicated electrical circuit, which is a processor with a circuit configuration specifically designed to perform a particular process, such as an ASIC (Application Specific Integrated Circuit).

[0062] A single processing unit may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA). Alternatively, multiple processing units may be composed of a single processor.

[0063] Examples of configuring multiple processing units with a single processor include, firstly, a configuration where one or more CPUs and software combine to form a single processor, as exemplified by client and server computers, and this processor functions as multiple processing units. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple processing units, on a single IC (Integrated Circuit) chip, as exemplified by System-on-a-Chip (SoC). Thus, various processing units are configured, in terms of hardware structure, using one or more of the above-mentioned processors.

[0064] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits (Circuitry) that combine circuit elements such as semiconductor elements.

[0065] The following are additional notes to this disclosure. (Additional note 1) Equipped with at least one processor, The aforementioned processor, Multiple radiographic images are sequentially acquired of a subject in which an ultrasound endoscope, equipped with an ultrasound imaging device and a radiopaque marker, is inserted into the body cavity. Multiple two-dimensional ultrasound images corresponding to each of the multiple radiographic images obtained by the ultrasound imaging device are acquired sequentially. Based on the markers included in each of the plurality of radiographic images, the position and orientation of the ultrasound endoscope within the body cavity are recognized. An image processing device that derives a three-dimensional ultrasound image from a plurality of two-dimensional ultrasound images based on the position and orientation of the ultrasound endoscope recognized with respect to the plurality of radiographic images. (Additional note 2) The processor aligns the radiographic image and the three-dimensional ultrasound image, The image processing apparatus according to Appendix 1, which superimposes the aligned three-dimensional ultrasound image onto the radiation image. (Additional note 3) The processor extracts the body cavity into which the ultrasound endoscope is inserted from a previously acquired three-dimensional image of the subject, The position and orientation of the ultrasound endoscope are corrected according to the shape of the extracted body cavity. The image processing apparatus according to appendix 1 or 2, which derives a three-dimensional ultrasound image from a plurality of two-dimensional ultrasound images based on corrected position and orientation. (Additional note 4) Multiple radiographic images are sequentially acquired of a subject in which an ultrasound endoscope, equipped with an ultrasound imaging device and a radiopaque marker, is inserted into the body cavity. Multiple two-dimensional ultrasound images corresponding to each of the multiple radiographic images obtained by the ultrasound imaging device are acquired sequentially. Based on the markers included in each of the plurality of radiographic images, the position and orientation of the ultrasound endoscope within the body cavity are recognized. An image processing method for deriving a three-dimensional ultrasound image from a plurality of two-dimensional ultrasound images based on the position and orientation of the ultrasound endoscope recognized with respect to the plurality of radiographic images. (Additional note 5) A procedure for sequentially acquiring multiple radiographic images of a subject in which an ultrasound endoscope, equipped with an ultrasound imaging device and a radiopaque marker, is inserted into a body cavity, and A procedure for sequentially acquiring multiple two-dimensional ultrasound images corresponding to each of the multiple radiographic images acquired by the ultrasound imaging device, A procedure for recognizing the position and orientation of the ultrasound endoscope within the body cavity based on the markers included in each of the plurality of radiographic images, An image processing program that causes a computer to perform a procedure for deriving a three-dimensional ultrasound image from a plurality of two-dimensional ultrasound images based on the position and orientation of the ultrasound endoscope recognized with respect to the plurality of radiographic images. [Explanation of symbols]

[0066] 1 Computer 2. 3D image acquisition device 3. Fluoroscopy imaging device 3A Arm 3B X-ray source 3C X-ray detector 4 Image storage server 5 Network 6. Ultrasound Endoscope 7 Endoscopy 7A channel 7B Optical system 7C Ultrasound Probe 8 Markers 8A Linear Marker 8B Chessboard Markers 8C ring marker 10,10A Image Processing Device 11 CPU 12 Image Processing Programs 13 Storage 14 displays 15 Input Devices 16 memory 21 Image acquisition unit 22 Recognition part 23 Derivation part 24 Alignment section 25 Display Control Unit 26 Extraction part 27 Correction section 30 routes 40 display screen 41 lesions T0, T1-T5 fluoroscopic images U0, U1~U5, U10 Ultrasound Images UT0 2D Projected Ultrasound Image UV0, UV12 3D ultrasound imaging

Claims

1. Equipped with at least one processor, The aforementioned processor, Multiple radiographic images are sequentially acquired of a subject in whom an ultrasound endoscope, equipped with an ultrasound imaging device and a radiopaque marker, is inserted into the bronchus. Multiple two-dimensional ultrasound images corresponding to each of the multiple radiographic images obtained by the ultrasound imaging device are acquired sequentially. Based on the markers included in each of the plurality of radiographic images, the position and orientation of the ultrasound endoscope in the bronchus are recognized. Based on the position and orientation of the ultrasound endoscope recognized with respect to the plurality of radiographic images, a three-dimensional ultrasound image is derived from the plurality of two-dimensional ultrasound images. The aforementioned radiographic image and the aforementioned three-dimensional ultrasound image are aligned, An image processing device that superimposes a two-dimensional projected ultrasonic image, obtained by projecting the three-dimensional ultrasonic image in the direction of acquisition of the radiographic image, onto the radiographic image.

2. A method for operating an image processing device having at least one processor, The aforementioned processor, Multiple radiographic images are sequentially acquired of a subject in whom an ultrasound endoscope, equipped with an ultrasound imaging device and a radiopaque marker, is inserted into the bronchus. Multiple two-dimensional ultrasound images corresponding to each of the multiple radiographic images obtained by the ultrasound imaging device are acquired sequentially. Based on the markers included in each of the plurality of radiographic images, the position and orientation of the ultrasound endoscope in the bronchus are recognized. Based on the position and orientation of the ultrasound endoscope recognized with respect to the plurality of radiographic images, a three-dimensional ultrasound image is derived from the plurality of two-dimensional ultrasound images. The aforementioned radiographic image and the aforementioned three-dimensional ultrasound image are aligned, A method for operating an image processing device that superimposes a two-dimensional projected ultrasound image, obtained by projecting the three-dimensional ultrasound image in the direction of acquisition of the radiation image, onto the radiation image.

3. A procedure for sequentially acquiring multiple radiographic images of a subject in which an ultrasound endoscope, equipped with an ultrasound imaging device and a radiopaque marker, is inserted into the bronchus, and A procedure for sequentially acquiring multiple two-dimensional ultrasound images corresponding to each of the multiple radiographic images acquired by the ultrasound imaging device, A procedure for recognizing the position and orientation of the ultrasound endoscope in the bronchus based on the markers included in each of the plurality of radiographic images, A procedure for deriving a three-dimensional ultrasound image from a plurality of two-dimensional ultrasound images based on the position and orientation of the ultrasound endoscope recognized with respect to the plurality of radiographic images, A procedure for aligning the aforementioned radiographic image and the aforementioned three-dimensional ultrasound image, An image processing program that causes a computer to perform the steps of superimposing a two-dimensional projected ultrasound image, obtained by projecting the three-dimensional ultrasound image in the direction of the radiation image, onto the radiation image.

Images