Image processing device, image processing method, and image processing program

Abstract

To provide an image processing device, an image processing method, and image processing program capable of suppressing deterioration in accuracy of positioning among images while suppressing an increase in a burden on a user.SOLUTION: An image processing device acquires a first image and a second image; derives a first deformation parameter by performing positioning between the first image and the second image; performs control to display a first composite image superimposing the first image onto the second image using the first deformation parameter; acquires a second deformation parameter different from the first deformation parameter when user input indicates that the first composite image is not appropriate; and performs control to display a second composite image compositing a plurality of images using one or more of the first deformation parameter and the second deformation parameter.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, an image processing method, and an image processing program. [Background technology]

[0002] Patent Document 1 discloses a technique for performing a first alignment relating to alignment between a first image data set and a second image data set, and a second alignment relating to alignment different from the first alignment between the first image data set and the second image data set. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-171867 Summary of the Invention [Problem to be solved by the invention]

[0004] However, there are cases where registration between multiple images cannot be performed efficiently. For example, in the medical field, transbronchial lung biopsy is performed by using 3D and 2D medical images to support the procedure. The problem of aligning 3D and 2D images generally involves a multi-peak objective function, and the alignment result may fall into a local optimum. In this case, the accuracy of image registration decreases. In response to this, requiring the user to perform image registration increases the burden on the user.

[0005] The present disclosure has been made in consideration of the above circumstances, and aims to provide an image processing device, an image processing method, and an image processing program that can suppress a decrease in the accuracy of alignment between images while suppressing an increase in the burden on the user. [Means for solving the problem]

[0006] A first aspect of the image processing device is an image processing device having at least one processor, wherein the processor acquires a first image and a second image whose shooting range is different from the first image and which at least partially overlaps with the shooting range of the first image, derives first deformation parameters that project the first image into the image space of the second image by aligning the first image and the second image, controls the display of a first composite image in which the first image is superimposed on the second image using the first deformation parameters, accepts user input indicating the validity of the first composite image, and if the user input indicates that the first composite image is invalid, acquires second deformation parameters that are different from the first deformation parameters, and controls the display of a second composite image in which multiple images are synthesized using one or more of the first deformation parameters and the second deformation parameters.

[0007] The image processing device of the second aspect is the image processing device of the first aspect, in which the processor generates a first composite image by superimposing one or more regions of interest contained in the first image onto the second image using first transformation parameters.

[0008] The image processing device of the third aspect is the image processing device of the second aspect, wherein the processor performs control to display the region of interest in the first composite image in a distinguishable display mode.

[0009] An image processing device of a fourth aspect is an image processing device of any one of the first to third aspects, in which the processor derives multiple sets of first deformation parameters by aligning a first image and a second image, selects one first deformation parameter from the derived multiple sets of first deformation parameters, and controls the display of a first composite image in which the first image is superimposed on the second image using the selected one first deformation parameter, and if user input indicates that the first composite image is invalid, obtains a second deformation parameter different from the first deformation parameter from the multiple sets of first deformation parameters.

[0010] In the image processing device of the fifth aspect, when user input indicates that the first composite image is invalid, the image processing device of the fourth aspect controls the display of a third composite image in which the first image is superimposed on the second image using one or more transformation parameters other than the first transformation parameters used to generate the first composite image from among multiple sets of first transformation parameters, and obtains the transformation parameters used to generate the third composite image selected by the user as second transformation parameters.

[0011] In the image processing device of the sixth aspect, in the image processing device of the fifth aspect, the processor controls the display of the third composite image by controlling the display of information representing the difference between the first composite image and the third composite image.

[0012] The image processing device of a seventh aspect is the image processing device of the sixth aspect, wherein the information indicating the difference is the difference between the first composite image and the third composite image in pixel units.

[0013] An eighth aspect of the image processing device is the image processing device of the sixth aspect, in which the processor acquires landmarks set on the first image and performs control to display, as information representing the difference, information representing the difference between landmarks superimposed on the first composite image using the first deformation parameters and landmarks superimposed on the third composite image using one or more deformation parameters.

[0014] In a ninth aspect of the image processing device, in the image processing device of the fourth aspect, the processor clusters all of the first deformation parameters derived by aligning the first image and the second image, and derives deformation parameters representative of each cluster as multiple sets of first deformation parameters.

[0015] An image processing device of a tenth aspect is the image processing device of the fourth aspect, in which the processor derives multiple sets of first deformation parameters by aligning the first image and the second image using multiple different techniques.

[0016] An eleventh aspect of the image processing device is the image processing device of the fourth aspect, in which the processor sets different sets of initial deformation parameters and search ranges, and derives multiple sets of first deformation parameters by repeatedly aligning the first image and the second image.

[0017] An image processing device of a twelfth aspect is the image processing device of any one of the first to eleventh aspects, wherein the first image is a three-dimensional medical image captured before surgery.

[0018] An image processing device of a thirteenth aspect is the image processing device of any one of the first to twelfth aspects, wherein the second image is a two-dimensional medical image captured during surgery.

[0019] The image processing device of a fourteenth aspect is the image processing device of the second or third aspect, wherein the region of interest is a region of the main bronchus, a region of the pulmonary artery, a lung field contour, or a region of a pulmonary nodule.

[0020] A fifteenth aspect of the image processing method involves acquiring a first image and a second image whose shooting range is different from the first image and which at least partially overlaps with the shooting range of the first image, deriving first deformation parameters that project the first image into the image space of the second image by aligning the first image and the second image, controlling the display of a first composite image in which the first image is superimposed on the second image using the first deformation parameters, accepting user input that indicates the validity of the first composite image, and if the user input indicates that the first composite image is invalid, acquiring second deformation parameters that are different from the first deformation parameters, and controlling the display of a second composite image in which multiple images are combined using one or more of the first deformation parameters and the second deformation parameters, all of which are executed by a processor provided in the image processing device.

[0021] The image processing program of a 16th aspect causes a processor provided in an image processing device to execute the following process: acquire a first image and a second image whose shooting range is different from the first image and which at least partially overlaps with the shooting range of the first image; derive first deformation parameters that project the first image into the image space of the second image by aligning the first image and the second image; control the display of a first composite image in which the first image is superimposed on the second image using the first deformation parameters; accept user input that indicates the validity of the first composite image; and, if the user input indicates that the first composite image is invalid, acquire second deformation parameters that are different from the first deformation parameters; and control the display of a second composite image in which multiple images are combined using one or more of the first deformation parameters and the second deformation parameters. [Effects of the Invention]

[0022] According to the present disclosure, it is possible to suppress a decrease in the accuracy of alignment between images while suppressing an increase in the burden on the user. [Brief explanation of the drawings]

[0023] [Figure 1] FIG. 1 is a diagram illustrating an example of a medical information system. [Figure 2] FIG. 2 is a block diagram illustrating an example of a hardware configuration of the image processing apparatus. [Figure 3] FIG. 2 is a block diagram illustrating an example of a functional configuration of the image processing apparatus. [Figure 4] FIG. 10 is a diagram for explaining clustering of deformation parameters. [Figure 5] FIG. 10 is a diagram showing an example of a display screen of a first composite image. [Figure 6] FIG. 10 is a diagram showing an example of a display screen for a third composite image. [Figure 7] FIG. 10 is a diagram showing an example of a display screen for a third composite image. [Figure 8] FIG. 10 is a diagram showing an example of a display screen for a third composite image. [Figure 9] FIG. 10 is a diagram showing an example of a display screen for a second composite image. [Figure 10] 10 is a flowchart illustrating an example of an insertion support process. DETAILED DESCRIPTION OF THE INVENTION

[0024] Hereinafter, examples of embodiments for carrying out the technology of the present disclosure will be described in detail with reference to the drawings.

[0025] First, the configuration of a medical information system 10 will be described with reference to Fig. 1. As shown in Fig. 1, the medical information system 10 includes an image processing device 1, a three-dimensional image capturing device 2, a fluoroscopic image capturing device 3, and an image storage server 4. The image processing device 1, the three-dimensional image capturing device 2, the fluoroscopic image capturing device 3, and the image storage server 4 are each connected via a network 5 in a state where they can communicate with each other.

[0026] The three-dimensional imaging device 2 captures an image of a diagnostic target region of the subject H, thereby generating a three-dimensional medical image representing the region. Examples of the three-dimensional imaging device 2 include a CT device, an MRI (Magnetic Resonance Imaging) device, and a PET (Positron Emission Tomography) device. The three-dimensional medical image, consisting of a plurality of tomographic images, generated by the three-dimensional imaging device 2 is transmitted to and stored in the image storage server 4. In this embodiment, the target region of the subject H is the lung, and the three-dimensional imaging device 2 is a CT device. That is, the three-dimensional medical image according to this embodiment is a CT image. In addition, in this embodiment, it is assumed that the three-dimensional imaging device 2 captures an image of the subject H's chest before a procedure on the subject H (i.e., before surgery), thereby obtaining a three-dimensional medical image including the subject H's chest. The three-dimensional medical image is an example of a first image according to the disclosed technology.

[0027] The fluoroscopic imaging device 3 has a C-arm 3A, an X-ray source 3B, and an X-ray detector 3C. The X-ray source 3B and the X-ray detector 3C are attached to both ends of the C-arm 3A, respectively. In the fluoroscopic imaging device 3, the C-arm 3A is configured to be rotatable and movable so that the subject H can be imaged from any direction. During a treatment on the subject H, the fluoroscopic imaging device 3 performs fluoroscopic imaging by continuously irradiating the subject H with X-rays, as an example of radiation, at a predetermined frame rate and sequentially detecting the X-rays that have passed through the subject H with the X-ray detector 3C, thereby sequentially acquiring radiographic images of the subject H. In the following description, the radiographic images of each sequentially acquired frame will be referred to as a fluoroscopic image. The fluoroscopic image acquired by radiographic imaging using the fluoroscopic imaging device 3 is an example of a two-dimensional medical image according to the disclosed technology.

[0028] The image storage server 4 is a computer that stores and manages various data, and is equipped with a large-capacity external storage device and database management software. The image storage server 4 communicates with other devices via a wired or wireless network 5, sending and receiving image data and the like. Specifically, the image storage server 4 acquires various data via the network 5, including image data representing 3D medical images acquired by the 3D imaging device 2 and fluoroscopic images acquired by the fluoroscopic imaging device 3, and stores and manages the data on a recording medium such as a large-capacity external storage device. The storage format of the image data and communication between devices via the network 5 are based on protocols such as DICOM (Digital Imaging and Communication in Medicine).

[0029] In this embodiment, an example will be described in which a biopsy procedure is performed by excising a portion of a lesion, such as a pulmonary nodule, present in the lungs of the subject H while performing fluoroscopic imaging of the subject H to examine the presence of the disease in detail. For this purpose, the fluoroscopic imaging device 3 is disposed in a treatment room for the biopsy. That is, fluoroscopic images are taken during the procedure. The fluoroscopic images are an example of a second image according to the disclosed technology. An ultrasound endoscope device 6 is also installed in the treatment room. The ultrasound endoscope device 6 includes an endoscope 6A having an ultrasound probe and treatment tools, such as forceps, attached to its tip. In this embodiment, a user, such as a doctor, inserts the endoscope 6A into the bronchi of the subject H to perform a biopsy of the lesion. During the biopsy, the fluoroscopic imaging device 3 takes fluoroscopic images of the subject H and displays the taken fluoroscopic images in real time. The user confirms the position of the tip of the endoscope 6A within the subject H in the fluoroscopic images and moves the tip of the endoscope 6A to the location of the target lesion.

[0030] Since lung lesions such as pulmonary nodules occur outside the bronchi, not inside, the user moves the tip of the endoscope 6A to the target position and then performs a procedure to extract part of the lesion using a treatment tool such as forceps while checking the location of the lesion in an ultrasound image obtained by capturing an image of the outside of the bronchi with the ultrasound probe.

[0031] In this embodiment, the three-dimensional medical image and the fluoroscopic image are different images, specifically, images captured by different modalities. Furthermore, the three-dimensional medical image and the fluoroscopic image are obtained by capturing an imaging range including a region to be diagnosed of the same subject H. That is, the imaging range of the three-dimensional medical image captured by the three-dimensional imaging device 2 and the imaging range of the fluoroscopic image captured by the fluoroscopic imaging device 3 at least partially overlap.

[0032] Next, with reference to FIG. 2, a hardware configuration of an image processing device 1 according to this embodiment will be described. The image processing device 1 is a device that assists in inserting an endoscope 6A into a bronchi, which is an example of a tubular structure. As shown in FIG. 2, the image processing device 1 includes a CPU (Central Processing Unit) 20, a memory 21 as a temporary storage area, and a non-volatile storage unit 22. The image processing device 1 also includes a display 23 such as a liquid crystal display, an input device 24 such as a keyboard and a mouse, and a network I / F (Interface) 25 connected to a network 5. The input device 24 may be a touch panel integrated with the display 23. The CPU 20, the memory 21, the storage unit 22, the display 23, the input device 24, and the network I / F 25 are connected to a bus 27. The CPU 20 is an example of a processor according to the disclosed technology. An example of the image processing device 1 is a computer, such as a personal computer or a server computer.

[0033] The storage unit 22 is realized by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or the like. The storage unit 22 serving as a storage medium stores an image processing program 30. The CPU 20 reads the image processing program 30 from the storage unit 22, expands it in the memory 21, and executes the expanded image processing program 30.

[0034] The storage unit 22 also stores three-dimensional medical images 32 and three-dimensional path data 34. The three-dimensional medical images 32 include a group of tomographic images obtained by capturing images of a subject H who is the target of biopsy. The three-dimensional medical images 32 are acquired from the image storage server 4 via the network 5 prior to the biopsy.

[0035] The three-dimensional path data 34 is data generated based on the three-dimensional medical image 32. Specifically, it is a collection of voxel data generated by performing 3D (three-dimensional) modeling that numerically describes the three-dimensional shape of the body of the subject H based on the three-dimensional medical image 32. Voxel data is a unit of pixel in three-dimensional space, and has three-dimensional coordinate information and pixel values. In this embodiment, the imaging range of the three-dimensional medical image 32 includes the bronchi. That is, the three-dimensional path data 34 includes a three-dimensional path, which is three-dimensional information of the bronchial lumen, which is the path through which the endoscope 6A passes. This three-dimensional path is a collection of voxel data corresponding to the bronchial lumen. That is, the three-dimensional path is also a three-dimensional image.

[0036] Next, the functional configuration of the image processing device 1 will be described with reference to Fig. 3. As shown in Fig. 3, the image processing device 1 includes a first acquisition unit 40, a derivation unit 42, a selection unit 44, a first generation unit 46, a first display control unit 48, a reception unit 50, a second generation unit 52, a second display control unit 54, a second acquisition unit 56, and a third display control unit 58. When the CPU 20 executes the image processing program 30, the image processing device 1 functions as the first acquisition unit 40, the derivation unit 42, the selection unit 44, the first generation unit 46, the first display control unit 48, the reception unit 50, the second generation unit 52, the second display control unit 54, the second acquisition unit 56, and the third display control unit 58.

[0037] The first acquisition unit 40 acquires the three-dimensional medical image 32 from the storage unit 22. The first acquisition unit 40 also sequentially acquires, via the network I / F 25, fluoroscopic images captured by the fluoroscopic image capturing device 3 at a predetermined frame rate.

[0038] The derivation unit 42 derives first deformation parameters for projecting the three-dimensional medical image 32 into the image space of the perspective image by aligning the three-dimensional medical image 32 acquired by the first acquisition unit 40 with the perspective image.

[0039] In this embodiment, the derivation unit 42 derives multiple sets of first deformation parameters by aligning the 3D medical image 32 and the fluoroscopic image using multiple different methods. Specifically, the derivation unit 42 derives multiple sets of first deformation parameters by sequentially performing three methods: alignment using rigid transformation, alignment using affine transformation, and alignment using nonlinear nonrigid transformation. Affine transformation is an example of linear nonrigid transformation. Note that the derivation unit 42 may derive multiple sets of first deformation parameters by individually performing multiple alignment methods rather than sequentially. The deformation parameters of rigid transformation include multiple parameters such as the amount of rotation and the amount of translation. The deformation parameters of affine transformation and nonlinear nonrigid transformation also include multiple parameters. In affine transformation and nonlinear nonrigid transformation, the deformation parameters include parameters for deforming the image.

[0040] First, the derivation unit 42 projects the three-dimensional medical image 32 onto a two-dimensional plane in the same direction as the imaging direction of the fluoroscopic image, and derives rigid transformation parameters as first deformation parameters so that the positions of known areas such as the bronchi, pulmonary arteries, and lesion areas match the positions of those areas in the fluoroscopic image.

[0041] Next, the derivation unit 42 projects the three-dimensional medical image 32 after deformation by the rigid transformation, similar to the rigid transformation, and derives parameters of the affine transformation as first deformation parameters so that the position of the known region coincides with the position of the region in the perspective image. Furthermore, the derivation unit 42 projects the three-dimensional medical image 32 after deformation by the rigid transformation and the affine transformation, similar to the rigid transformation, and derives parameters of the nonlinear non-rigid transformation as first deformation parameters so that the position of the known region coincides with the position of the region in the perspective image.

[0042] The derivation unit 42 may set different sets of initial deformation parameters and search ranges. In this case, the derivation unit 42 may derive the first deformation parameters for each set of initial deformation parameters and search ranges by repeatedly aligning the three-dimensional medical image 32 with the perspective image while varying the deformation parameters within the search range based on the initial deformation parameters. In this case, the derivation unit 42 may determine, as the first deformation parameters, the deformation parameters used to generate a two-dimensional image having the highest similarity to the perspective image among two-dimensional images obtained by projecting the three-dimensional medical image 32 onto a two-dimensional plane using the deformation parameters for each set.

[0043] Furthermore, the derivation unit 42 may cluster all deformation parameters derived by aligning the three-dimensional medical image 32 with the perspective image. FIG. 4 shows an example of a clustering result of deformation parameters using two indices: the amount of deformation due to the deformation parameter and the similarity between the two-dimensional image and the perspective image obtained using the deformation parameter. Each circle in FIG. 4 represents one deformation parameter. FIG. 4 shows an example in which 16 deformation parameters are classified into three clusters. In this case, the derivation unit 42 may derive a deformation parameter representing each cluster as multiple sets of first deformation parameters. The deformation parameter representing each cluster may be the deformation parameter with the highest similarity or the deformation parameter with the smallest amount of deformation.

[0044] The selector 44 selects one first deformation parameter from the plurality of sets of first deformation parameters derived by the deriver 42. For example, the selector 44 selects, from the plurality of sets of first deformation parameters, the first deformation parameter that provides the highest similarity between the perspective image and a two-dimensional image obtained by projecting the three-dimensional medical image 32 onto a two-dimensional plane using the first deformation parameter. The similarity between images can be determined by using a known index value such as the sum of differences between corresponding pixels or cosine similarity.

[0045] The first generator 46 generates a composite image (hereinafter referred to as the "first composite image") by superimposing the three-dimensional medical image 32 on the perspective image using one first deformation parameter selected by the selector 44. In this embodiment, the first composite image is generated by superimposing one or more regions of interest included in the three-dimensional medical image 32 on the perspective image using the first deformation parameter. In this embodiment, an example will be described in which the regions of the main bronchus, the region of the pulmonary artery, and the contours of the lung field are applied as the regions of interest. Note that a lesion region, such as a pulmonary nodule region, may also be applied as the region of interest.

[0046] The first display control unit 48 controls the display of the first composite image generated by the generation unit 46 on the display 23. Fig. 5 shows an example of a display screen of the first composite image displayed on the display 23. Fig. 5 shows an example in which the contours of the main bronchi, the pulmonary artery, and the lung field are superimposed on a fluoroscopic image. In Fig. 5, the solid line L1 indicates the main bronchi, the solid line L2 indicates the pulmonary artery, and the dashed line L3 indicates the contour of the lung field.

[0047] The first display control unit 48 may control the display of the regions of interest in the first composite image in a distinguishable display manner. For example, the first display control unit 48 may control the display of the regions of interest by filling them with a color other than that used in the fluoroscopic image, such as by displaying the main bronchial tube in green and the pulmonary artery in red. This makes the regions of interest distinguishable. Furthermore, the first display control unit 48 may control the display of the regions of interest in a distinguishable display manner by varying not only the color but also the type of line (e.g., dashed line), the line thickness, or the transparency for each region of interest.

[0048] The user checks the first composite image displayed on the display 23 and inputs information indicating the validity of the first composite image. For example, the user inputs information indicating whether the alignment in the first composite image is valid or not via the input device 24. The receiving unit 50 receives the user input indicating the validity of the first composite image.

[0049] When the user input received by the receiving unit 50 indicates that the first composite image is invalid, the second generating unit 52 extracts one or more transformation parameters other than the one transformation parameter used to generate the first composite image from the multiple sets of transformation parameters derived by the derivation unit 42. For example, when the one transformation parameter used to generate the first composite image is an affine transformation parameter, the second generating unit 52 extracts a rigid transformation parameter and a nonlinear rigid transformation parameter as the transformation parameters.

[0050] Then, the second generating unit 52 generates a composite image by superimposing the 3D medical image 32 on the perspective image using one or more extracted deformation parameters, similar to the first generating unit 46. The composite image generated by the second generating unit 52 is referred to as a "third composite image" to distinguish it from the first composite image described above and the second composite image described below. When two or more sets of deformation parameters are extracted, the second generating unit 52 generates multiple third composite images by separately using each of the deformation parameters.

[0051] The second display control unit 54 performs control to display the third composite image generated by the second generation unit 52 on the display 23. Fig. 6 shows an example of a display screen of the third composite image displayed on the display 23. As shown in Fig. 6, in this embodiment, the second display control unit 54 performs control to display the first composite image on the display 23 together with the third composite image for comparison with the third composite image. Fig. 6 shows an example in which there are two third composite images.

[0052] In addition, in controlling the display of the third composite image, the second display control unit 54 may also perform control to display information representing the difference between the first composite image and the third composite image. In this case, as shown in Fig. 7, the second display control unit 54 may perform control to display the difference between the first composite image and the third composite image in pixel units as information representing the difference. Fig. 7 shows an example in which a difference image between the first composite image and the third composite image is displayed below each of the third composite images.

[0053] As shown in FIG. 8 , the second display control unit 54 may display, as information representing the difference, information representing the difference between a landmark P superimposed on the first composite image using the first deformation parameter and a landmark P superimposed on the third composite image using one or more deformation parameters. In this case, the landmark P may be set in advance on the three-dimensional medical image 32, and the second display control unit 54 may acquire the landmark P set on the three-dimensional medical image 32. In the example of FIG. 8 , six landmarks P are superimposed on the first composite image and the third composite image, and the difference in position between each landmark P is represented by an arrow Y in the third composite image. In the example of FIG. 8 , the starting point of the arrow represents the position of the landmark P in the first composite image, and the ending point of the arrow represents the position of the landmark P in the third image. In other words, by looking at the arrow, the user can grasp the direction and extent of the deviation of the landmark P. Examples of landmarks P include the bifurcation of the bronchi and the anterior superior lobe branch.

[0054] The user checks the third composite images displayed on the display 23 and selects the third composite image that is considered most appropriate via the input device 24. The second acquisition unit 56 acquires the transformation parameters used to generate the third composite image selected by the user as the second transformation parameters. That is, the second acquisition unit 56 acquires the second transformation parameters, which are different from the first transformation parameters, from multiple sets of first transformation parameters.

[0055] The third display control unit 58 uses the second deformation parameters acquired by the second acquisition unit 56 to perform control to display on the display 23 a composite image (hereinafter referred to as the "second composite image") obtained by combining multiple images. In this embodiment, the third display control unit 58 uses the second deformation parameters to perform control to display on the display 23 a second composite image obtained by superimposing a three-dimensional path included in the three-dimensional path data 34 on a perspective image. FIG. 9 shows an example of a display screen for the second composite image. The solid line L4 in FIG. 9 represents the three-dimensional path along which the endoscope 6A passes. This display screen is used as a screen for navigating the operation of the endoscope 6A by the user.

[0056] In addition, when the user input received by the receiving unit 50 indicates that the first composite image is appropriate, the third display control unit 58 controls the display 23 to display a second composite image in which the three-dimensional route included in the three-dimensional route data 34 is superimposed on the perspective image using the first deformation parameters.

[0057] In addition, when the user selects the first composite image on the display screen of the third composite image, the third display control unit 58 may use the first deformation parameters to control the display 23 to display a second composite image in which the three-dimensional path is superimposed on the perspective image.

[0058] Next, the operation of the image processing device 1 will be described with reference to Fig. 10. The CPU 20 executes the image processing program 30, thereby executing the insertion support process shown in Fig. 10. The insertion support process is executed, for example, when a command to start execution is input by the user.

[0059] 10, the first acquisition unit 40 acquires the three-dimensional medical image 32 from the storage unit 22. In step S12, the first acquisition unit 40 acquires the fluoroscopic image captured by the fluoroscopic image capturing device 3 via the network I / F 25. In step S14, the derivation unit 42 derives multiple sets of first deformation parameters by aligning the three-dimensional medical image 32 and the fluoroscopic image using multiple different methods, as described above.

[0060] In step S16, the selection unit 44 selects one first deformation parameter from the sets of first deformation parameters derived in step S14, as described above. In step S18, the first generation unit 46 generates a first composite image by superimposing the three-dimensional medical image 32 on the perspective image using the one first deformation parameter selected in step S16, as described above. In step S20, the first display control unit 48 controls the display 23 to display the first composite image generated in step S18.

[0061] In step S22, the receiving unit 50 receives a user input indicating the validity of the first composite image displayed on the display 23 in step S20. In step S24, the second generation unit 52 determines whether the user input received in step S22 indicates that the first composite image is valid. If this determination is negative, that is, if the user input indicates that the first composite image is not valid, the process proceeds to step S26.

[0062] In step S26, the second generator 52 extracts one or more deformation parameters other than the one deformation parameter used to generate the first composite image from the plurality of sets of deformation parameters derived in step S14, as described above. Then, the second generator 52 generates a third composite image by superimposing the three-dimensional medical image 32 on the perspective image using the one or more extracted deformation parameters.

[0063] In step S28, the second display control unit 54 performs control to display the third composite image generated in step S26 on the display 23. In step S30, the second acquisition unit 56 acquires, as second transformation parameters, the transformation parameters used to generate the third composite image selected by the user. In step S32, the third display control unit 58 performs control to display, on the display 23, a second composite image in which the three-dimensional route included in the three-dimensional route data 34 is superimposed on the perspective image, using the second transformation parameters acquired in step S30. When the processing of step S32 ends, the insertion support processing ends.

[0064] On the other hand, if the determination in step S24 is positive, the process proceeds to step S34. In step S34, the third display control unit 58 performs control to display a second composite image in which the three-dimensional path is superimposed on the perspective image, using the first deformation parameter, on the display 23. When the process of step S34 ends, the insertion support process ends.

[0065] As described above, according to this embodiment, it is possible to suppress a decrease in the accuracy of alignment between images while suppressing an increase in the burden on the user.

[0066] In the above embodiment, an endoscope is used as the medical instrument, and the bronchial lumen is used as the path through which the medical instrument passes, but the disclosed technology is not limited to this. For example, the lumen of a blood vessel may be used as the path through which the medical instrument passes.

[0067] In the above embodiment, a case where a perspective image is used as a two-dimensional medical image is described, but the disclosed technology is not limited to this. For example, an endoscopic image may be used as a two-dimensional medical image.

[0068] Furthermore, in the above embodiment, for example, the following various processors can be used as the hardware structure of a processing unit that executes various processes such as each functional unit of the image processing device 1. As described above, the various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as dedicated electrical circuits that are processors having a circuit configuration specifically designed to execute specific processes, such as a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and an ASIC (Application Specific Integrated Circuit).

[0069] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor.

[0070] Examples of configuring multiple processing units with a single processor include, first, a form in which one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units, as typified by computers such as client and server. Second, a form in which a processor is used to realize the functions of the entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by systems on chips (SoCs). In this way, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0071] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0072] In the above embodiment, the image processing program 30 is pre-stored (installed) in the storage unit 22, but the present invention is not limited to this. The image processing program 30 may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a USB (Universal Serial Bus) memory. The image processing program 30 may also be downloaded from an external device via a network. [Explanation of symbols]

[0073] 1. Image processing device 2. 3D imaging device 3. Fluoroscopic imaging device 3A C-arm 3B X-ray source 3C X-ray detector 4. Image storage server 5. Network 6. Endoscopic ultrasound equipment 6A Endoscope 10 Medical Information Systems 20 CPU 21 Memory 22 Memory section 23 Display 24 Input Devices 25 Network I / F 27 Bus 30 Image Processing Programs 32 3D Medical Imaging 34 3D route data 40 First acquisition part 42 Derivation part 44 Selection section 46 1st generation part 48 First display control unit 50 Reception 52 Second generation part 54 Second display control unit 56 Second acquisition part 58 Third display control unit H Subject L1, L2, L4 solid lines L3 dashed line P Landmark Y arrow

Claims

1. An image processing device comprising at least one processor, The processor: acquiring a first image and a second image of a shooting range different from the first image and at least partially overlapping with a shooting range of the first image; deriving first transformation parameters that project the first image into an image space of the second image by registering the first image and the second image; performing control to display a first composite image in which the first image is superimposed on the second image using the first transformation parameters; accepting user input indicating the validity of the first composite image; if the user input indicates that the first composite image is invalid, obtaining second transformation parameters different from the first transformation parameters; and performing control to display a second composite image obtained by combining a plurality of images using one or more of the first transformation parameter and the second transformation parameter. Image processing device.

2. The processor: generating a first composite image by superimposing one or more regions of interest included in the first image on the second image using the first transformation parameters; The image processing device according to claim 1 .

3. The processor: Control is performed to display the region of interest in the first composite image in a distinguishable display mode. The image processing device according to claim 2 .

4. The processor: deriving a plurality of sets of the first deformation parameters by registering the first image and the second image; selecting one of the first transformation parameters from the derived sets of first transformation parameters; performing control to display the first composite image obtained by superimposing the first image on the second image using the selected one of the first transformation parameters; If the user input indicates that the first composite image is not valid, a second transformation parameter different from the first transformation parameter is obtained from a plurality of sets of the first transformation parameters. The image processing device according to any one of claims 1 to 3.

5. When the user input indicates that the first composite image is invalid, control is performed to display a third composite image in which the first image is superimposed on the second image using one or more transformation parameters other than the first transformation parameters used to generate the first composite image, from among a plurality of sets of the first transformation parameters; The transformation parameters used to generate the third composite image selected by the user are acquired as the second transformation parameters. The image processing device according to claim 4 .

6. The processor: In the control of displaying the third composite image, control is performed to display information representing a difference between the first composite image and the third composite image. The image processing device according to claim 5 .

7. The information representing the difference is a difference between the first composite image and the third composite image in pixel units. The image processing device according to claim 6 .

8. The processor: acquiring landmarks set on the first image; and performing control to display, as information representing the difference, information representing a difference between the landmark superimposed on the first composite image using the first transformation parameter and the landmark superimposed on the third composite image using the one or more transformation parameters. The image processing device according to claim 6 .

9. The processor: clustering all of the first deformation parameters derived by aligning the first image and the second image; Derivation of a plurality of sets of first deformation parameters representing each cluster. The image processing device according to claim 4 .

10. The processor: deriving a plurality of sets of the first deformation parameters by aligning the first image and the second image using a plurality of different techniques; The image processing device according to claim 4 .

11. The processor: A plurality of different sets of initial deformation parameters and search ranges are set, and the first image and the second image are aligned repeatedly to derive a plurality of sets of the first deformation parameters. The image processing device according to claim 4 .

12. The first image is a three-dimensional medical image taken before surgery. The image processing device according to any one of claims 1 to 3.

13. The second image is a two-dimensional medical image taken during surgery. The image processing device according to any one of claims 1 to 3.

14. The region of interest is a region of the main bronchus, a region of the pulmonary artery, a lung field contour, or a region of a pulmonary nodule.

4. The image processing device according to claim 2 or 3.

15. acquiring a first image and a second image of a shooting range different from the first image and at least partially overlapping with a shooting range of the first image; deriving first transformation parameters that project the first image into an image space of the second image by registering the first image and the second image; performing control to display a first composite image in which the first image is superimposed on the second image using the first transformation parameters; accepting user input indicating the validity of the first composite image; if the user input indicates that the first composite image is invalid, obtaining second transformation parameters different from the first transformation parameters; and performing control to display a second composite image obtained by combining a plurality of images using one or more of the first transformation parameter and the second transformation parameter. An image processing method in which processing is performed by a processor provided in an image processing device.

16. acquiring a first image and a second image of a shooting range different from the first image and at least partially overlapping with a shooting range of the first image; deriving first transformation parameters that project the first image into an image space of the second image by registering the first image and the second image; performing control to display a first composite image in which the first image is superimposed on the second image using the first transformation parameters; accepting user input indicating the validity of the first composite image; if the user input indicates that the first composite image is invalid, obtaining second transformation parameters different from the first transformation parameters; and performing control to display a second composite image obtained by combining a plurality of images using one or more of the first transformation parameter and the second transformation parameter. An image processing program for causing a processor included in the image processing device to execute the processing.

Images