Medical image processing system and medical image processing method

Abstract

To display a tubular tissue included in a MR image on an X-ray image by accurately aligning and superimposing a MR image photographed in advance on an X-ray image photographed during an operation.SOLUTION: A MR image and a CT image, which are photographed in advance, are aligned with each other to determine a coordinate correspondence. Next, the CT image and an X-ray image are aligned with each other to determine a coordinate correspondence. The X-ray image and the MR image are aligned by using the coordinate correspondence between the MR image and the CT image and the coordinate correspondence between the CT image and the X-ray image. On the other hand, feature points indicating positions of predetermined sites of a subject in the MR image and the CT image are extracted. Feature points indicating the positions of the corresponding sites in the X-ray image are extracted. After the X-ray image and the MR image are further aligned by using the coordinates of at least one of the feature points of the MR image and the CT image and the coordinates of the feature points of the X-ray image, the X-ray image and the MR image are superimposed and displayed as a guide image.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a system for processing images captured by an X-ray imaging device. [Background technology]

[0002] Patent Document 1 discloses a technique for aligning and superimposing a projected image, which is a three-dimensional X-ray CT image, on an X-ray image captured by an X-ray imaging device, and displaying the superimposed image.

[0003] On the other hand, Patent Document 2 discloses a technique for displaying a 3D X-ray CT image (blood vessel image data) and a 2D X-ray fluoroscopic image in an easy-to-view manner. This technique extracts a point of interest in an X-ray fluoroscopic image, estimates a corresponding point on the 3D blood vessel image data that corresponds to the point of interest, and generates and displays a projection image including the corresponding point in the 3D blood vessel image data. At this time, a fluoroscopic image for the viewer's left eye and a fluoroscopic image for the viewer's right eye are generated to enable stereoscopic viewing. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2016-041166 A [Patent Document 2] JP 2013-059609 A Summary of the Invention [Problem to be solved by the invention]

[0005] The 3D X-ray CT images taken in advance show the bone structure of the subject's spine and other parts of the body in high resolution, but do not show tubular tissues such as blood vessels and the pancreatic duct. X-ray images show blood vessels and the pancreatic duct if a contrast agent is used, but do not show them if a contrast agent is not used.

[0006] In recent years, there has been a demand for the ability to grasp tubular tissues such as blood vessels and pancreatic ducts using pre-contrast X-ray images during surgery.

[0007] On the other hand, MR images taken by MRI (Magnetic Resonance Imaging) devices can depict tubular tissues depending on the imaging method. For example, MR cholangiopancreatography (MRCP) can depict the bile duct and pancreatic duct using MRI without using contrast agents.

[0008] However, MR images depict protons in tissue, while X-ray images depict high and low X-ray absorption rates, so the subjects depicted in the two images are different, and it is not easy to accurately align MR and X-ray images.

[0009] An object of the present invention is to accurately align and superimpose an MR image taken in advance on an X-ray image taken during surgery, and to display tubular tissue contained in the MR image on the X-ray image. [Means for solving the problem]

[0010] In order to achieve the above object, the medical image processing system of the present invention includes a pre-image acquisition unit, an intraoperative image acquisition unit, an MR-CT image registration unit, a CT-X-ray image registration unit, a pre-image feature point extraction unit, an intraoperative image feature point extraction unit, an MR-X-ray image registration unit, and an image superimposition unit. The pre-image acquisition unit captures an MR image of a subject captured in advance by a magnetic resonance imaging device and a CT image of the subject captured in advance by an X-ray CT device. The intraoperative image acquisition unit captures an X-ray image of the subject captured during surgery. The MR-CT image registration unit registers the MR image and the CT image, and determines the correspondence between the coordinates of the MR image and the CT image. The CT-X-ray image registration unit registers the CT image and the X-ray image, and determines the correspondence between the coordinates of the CT image and the X-ray image. The pre-image feature point extraction unit extracts feature points indicating the positions of predetermined parts of the subject in the MR image and the CT image, respectively. The intraoperative image feature point extraction unit extracts feature points indicating the position of the subject's part in the X-ray image. The MR-X-ray image alignment unit aligns the X-ray image and the MR image. The image superimposition unit superimposes the X-ray image and the MR image aligned by the MR-X-ray image alignment unit to generate a superimposed image and display the superimposed image as a guide image. The MR-X-ray image alignment unit aligns the X-ray image and the MR image using the corresponding relationship between the coordinates of the MR image and the CT image and the corresponding relationship between the coordinates of the CT image and the X-ray image, and then further aligns the X-ray image and the MR image at least in the translation direction using the coordinates of at least one of the feature points of the MR image and the feature points of the CT image and the coordinates of the feature points in the X-ray image. Effect of the Invention

[0011] According to the present invention, an MR image taken in advance can be precisely aligned and superimposed on an X-ray image taken during surgery, thereby making it possible to display tubular tissue contained in the MR image on the X-ray image. [Brief description of the drawings]

[0012] [Figure 1] 1 is a functional block diagram showing the configuration of a medical image processing system 100 according to a first embodiment of the present invention. [Diagram 2]4 is a flowchart showing the operation of the medical image processing system 100 of the first embodiment. [Diagram 3] 1(a) to 1(f) are diagrams showing an overview of image processing in the medical image processing system 100 of the first embodiment. [Figure 4] FIG. 11 is a functional block diagram showing the configuration of a medical image processing system 200 according to a second embodiment of the present invention. [Diagram 5] 4 is a flowchart showing the operation of the medical image processing system 200 of the first embodiment. [Figure 6] 13(a) to 13(f) are diagrams showing an outline of an area where feature points exist and alignment in the medical image processing system 100 of the second embodiment. [Figure 7] 13 is a diagram showing the relationship between the imaging (respiration) conditions of the medical image processing system 200 of the second embodiment and the shape of the existence region. FIG. [Figure 8] 11(a) to 11(c) are diagrams showing a part of image processing in a medical image processing system 200 according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0014] <<Embodiment 1>> The configuration of a medical image processing system 100 according to the first embodiment will be described.

[0015] The medical image processing system 100 generates a guide image by superimposing an MR image captured in advance on an X-ray image of a subject undergoing surgery without an injected contrast agent. As a result, the guide image displays tubular structures such as blood vessels and pancreatic ducts depicted in the MR image superimposed on the X-ray image, allowing the surgeon to grasp, on the X-ray image, the tubular structures into which a treatment tool will be inserted.

[0016] Fig. 1 is a diagram showing the configuration of a medical image processing system 100. Fig. 2 is a flowchart showing the operation of the medical image processing system 100. Fig. 3(a) to Fig. 3(f) are diagrams showing an overview of image processing.

[0017] As shown in FIG. 1, the medical image processing system 100 includes an intraoperative two-dimensional image acquisition unit 10, a preoperative three-dimensional image acquisition unit 20, an image processing unit 150, an input unit 30, and a display unit .

[0018] An X-ray imaging device 2 is connected to the intraoperative two-dimensional image acquisition unit 10. A medical image server 3 is connected to the pre-operative three-dimensional image acquisition unit 20.

[0019] The medical image server 3 stores an MR image 202 obtained by previously capturing an image of a subject 1 using an MRI (Magnetic Resonance Imaging) device, and a CT image 201 obtained by previously capturing an image of the same subject 1 using an X-ray CT device (FIG. 3(a)). Here, both the MR image 202 and the CT image 201 are three-dimensional images.

[0020] The image processing unit 150 includes an MR-CT image alignment unit 50, a pre-image feature extraction unit 51, a projection calculation unit 52, an intraoperative image feature extraction unit 53, a CT-X-ray image alignment unit 54, a pre-image feature extraction unit 51, an MR-X-ray image alignment unit 55, and an image superposition unit 56.

[0021] The preliminary three-dimensional image acquisition unit 20 acquires an MR image 202 and a CT image 201 of the subject 1 from the medical image server 3 .

[0022] As is known, the X-ray CT image 201 depicts in detail the morphology (shape) 211a of the bones, contours, etc. of the subject 1, but the tubular structures, such as blood vessels and pancreatic ducts, in the subject 1 are difficult to depict without the injection of a contrast agent. On the other hand, the MR image 202 depicts the tubular tissues 212a, such as the bile duct and pancreatic duct, of the subject, even without the injection of a contrast agent, but the morphology (shape) of the bones, contours, etc. are difficult to depict.

[0023] On the other hand, the intraoperative two-dimensional image acquisition unit 10 captures an X-ray image 203 of the subject 1 captured by the X-ray imaging device 2 during the operation (FIG. 3(d)). The captured X-ray image 203 may be one frame of fluoroscopic imaging (moving image) captured during the operation, or may be an X-ray image (still image) captured during the operation. It is generally known that the X-ray image 203 precisely depicts the morphology (shape) 203a of the bones, contours, etc. of the subject 1, similarly to the CT image 201. Furthermore, during surgery, when a treatment tool 301 such as an endoscope or a vascular catheter is inserted into the subject 1, the image of the treatment tool is also depicted in the X-ray image 203. However, in the X-ray image 203, it is difficult to depict tubular structures such as the bile duct and pancreatic duct in the subject 1 unless a contrast agent is injected.

[0024] The MR-CT image registration unit 50 aligns the MR image 202 and the CT image 201 acquired by the preliminary 3D image acquisition unit 20, and obtains a correspondence relationship between the coordinates of the MR image 202 and the CT image 201. Since the MR image 202 and the CT image 201 are 3D images in this case, the MR-CT image registration unit 50 aligns the MR image 202 and the CT image 201 in the 3D space. A known method is used as the alignment method. For example, a conventional alignment method using pixel values, an alignment method using feature points as landmarks, an alignment method using mutual information, or the like is used.

[0025] The preliminary image feature point extraction unit 51 extracts feature points 251, 252 indicating the positions of predetermined parts of the subject 1 in the CT image 201 and the MR image 202, respectively (FIGS. 3(a) and (b)). For example, the feature point is the "duodenal papilla." For example, the preliminary image feature point extraction unit 51 receives designation of positions in the three-dimensional CT image 201 and the three-dimensional MR image 202 from the user using a pointer or the like of the input unit 30, and sets the received positions as the feature points 251, 252. Alternatively, the preliminary image feature point extraction unit 51 may automatically extract the feature points 251, 252 by performing image processing on the CT image 201 and the MR image 202.

[0026] The projection calculation unit 52 projects the three-dimensional MR image 202 and CT image 201 in a predetermined direction to generate a two-dimensional MR projection image 212 and a CT projection image 211. The projection direction is set to the direction of the optical axis of the X-rays irradiated to the subject 1 by the X-ray imaging device 2. As a projection method, a ray-sum method in which the sum of pixel values ​​arranged along a ray is used as the pixel value of the projected image, a maximum intensity projection (MIP) method, or the like is used.

[0027] The MR-CT image alignment unit 50 can generate a superimposed image 220 of the two-dimensional MR projection image 212 and the CT projection image 211 after projection, and display it together with a guide image (superimposed image 230) described later (FIG. 3(b)).

[0028] In this superimposed image, the morphology 211a of the bones, etc. of the subject 1 is depicted by the CT projection image 211, and the tubular tissue 212a of the subject 1 is depicted by the MR projection image 212, resulting in an image simulating an X-ray image taken by the X-ray imaging device 2 while contrasting the tubular tissue with a contrast agent during surgery.

[0029] On the other hand, the intraoperative image feature point extraction unit 53 extracts a feature point 253 indicating the position of a predetermined part of the subject 1 from the captured X-ray image 203 (FIG. 3(d)). For example, in the X-ray image 203, the user is asked to place the tip of the treatment tool 301 at a predetermined part (e.g., the papilla), and the intraoperative image feature point extraction unit 53 extracts the insertion position (e.g., the tip) of the treatment tool 301 by image processing, and sets the extracted position as the feature point 253. The position of the feature point is the same part (e.g., the "duodenal papilla") as the feature point extracted by the pre-image feature point extraction unit 51. Similarly to the pre-image feature point extraction unit 51, the intraoperative image feature point extraction unit 53 may also receive a position designation from the user in the X-ray image 203 by a pointer or the like of the input unit 30, and set this position as the feature point 253.

[0030] The CT-X-ray image alignment unit 54 aligns the two-dimensional CT projection image 211 after projection with the X-ray image 203, and obtains the coordinate correspondence (parameters) between the CT projection image 211 and the X-ray image 203 (FIGS. 3(c) and 3(d)). In this alignment, alignment is performed in the parallel translation direction and the rotation direction so that a portion where the contrast of the CT projection image 211 is higher than a predetermined value (morphology 211a such as bones or body contours) coincides with a portion where the contrast of the X-ray image 203 is higher than a predetermined value (morphology 203a such as bones or body contours), and obtains the coordinate correspondence (parameters) between the parallel translation direction and the rotation direction.

[0031] The MR-X-ray image registration unit 55 registers the MR projection image 212 and the X-ray image 203 using the parameters of the translation direction and rotation direction obtained by the CT-X-ray image registration unit 54 .

[0032] Furthermore, the MR-X-ray image alignment unit 55 aligns the X-ray image 203 and the MR projection image 212 at least in the parallel movement direction so that the feature points 253 of the X-ray image 203 and the feature points 252 of the MR projection image 212 coincide or have a predetermined positional relationship (Figures 3(e) and (d)).

[0033] The MR projection image 212 has been registered with the CT projection image 211 in the three-dimensional space before projection by the MR-CT image registration unit 50, and the CT projection image 211 has been registered with the X-ray image 203 in the translation direction and rotation direction by the CT-X-ray image registration unit 54. Therefore, the MR projection image 212 and the X-ray image 203 have also been registered with each other in the translation direction and rotation direction. The MR-X-ray image registration unit 55 registers the MR projection image 212 and the X-ray image 203 with each other in the translation direction using the feature points 252 and 253, thereby registering them more accurately.

[0034] When aligning the MR projection image 212 and the X-ray image 203 using the feature points 252, 253 of the two images, the MR-X-ray image alignment unit 55 may display the feature point 251 of the CT projection image 211 on the MR projection image 212 and perform alignment using the feature point 251 together with or instead of the feature point 252. Since the MR projection image 212 and the CT projection image 211 have already been aligned, the positional relationship between the feature point 252 and the feature point 252 is clear, and therefore the feature point 251 of the CT projection image 211 can be displayed on the MR projection image 212.

[0035] The image superimposition unit 56 superimposes the X-ray image 203 aligned by the MR-X-ray image alignment unit 55 and the MR projection image 212 to generate a superimposed image 230, and displays the superimposed image 230 on the display unit 40 as a guide image (Figure 3(f)).

[0036] Thereby, by viewing the superimposed image 230 on the display unit 40, the surgeon can confirm the tubular tissue 212a such as the bile duct or pancreatic duct that is not depicted in the intraoperative pre-contrast X-ray image 203 in a registered state with the X-ray image 203. Therefore, the surgeon can grasp the shape of the tubular tissue 212a and the positional relationship between the treatment tool (e.g., endoscope) 301 depicted in the X-ray image 203 and the tubular tissue 212a. Even if the tubular tissue 212a is not depicted in the X-ray image 203, the surgeon can grasp the tubular tissue 212a on the X-ray image 203 and proceed with the procedure using the treatment tool 301.

[0037] Next, the operation of each unit of the medical image processing system 100 will be described with reference to the flow of FIG.

[0038] The image processing unit 150 includes a CPU and a memory, and the CPU executes a program stored in advance in the memory, thereby executing the operation of each unit (50-56) of the image processing unit 150 by software. Note that it is also possible to realize a part or the whole of the image processing unit 150 by hardware. For example, a circuit design may be performed to realize the function of each unit using a custom IC such as an ASIC (Application Specific Integrated Circuit) or a programmable IC such as an FPGA (Field-Programmable Gate Array).

[0039] <Step 101> First, the preliminary three-dimensional image acquisition unit 20 acquires, from the medical image server 3, a three-dimensional CT image 201 obtained by capturing an image of the subject 1 in advance.

[0040] <Step 102> Next, the preliminary three-dimensional image acquisition unit 20 acquires, from the medical image server 3, a three-dimensional MR image 202 of the subject 1 captured in advance.

[0041] <Step 103> The MR-CT image registration unit 50 performs registration in a three-dimensional space between the MR image 202 and the CT image 201. As a registration method, for example, a known image registration method using mutual information is used.

[0042] <Step 104> The preliminary image feature point extraction unit 51 extracts feature points 251 indicating the position of a predetermined part of the subject 1 in the CT image 201. For example, the preliminary image feature point extraction unit 51 causes the display unit 40 to display the three-dimensional CT image 201, prompts the user to operate the pointer or the like of the input unit 30 to point a cursor at a predetermined part (for example, the duodenal papilla) on the image, and captures the position pointed by the user as the feature point 251.

[0043] <Step 105> Next, the pre-image feature point extracting unit 51 extracts, in the MR image 202 in the same manner as in the CT image 201, feature points 252 indicating the positions of predetermined parts of the subject 1.

[0044] <Step 106> The projection calculation unit 52 projects the three-dimensional CT image 201 from which the feature points 251 were extracted in step 104 in a predetermined direction to generate a two-dimensional CT projection image 211. The projection direction is set to the direction of the optical axis of the X-rays irradiated to the subject 1 by the X-ray imaging device 2. Hereinafter, the projected CT projection image 211 is also referred to as a "pseudo projection image (CT)."

[0045] <Step 107> The projection calculation unit 52 projects the three-dimensional MR image 202 from which the feature points 252 have been extracted in step 105 in a predetermined direction to generate a two-dimensional MR projection image 212. Hereinafter, the projected MR projection image 212 is also referred to as a "pseudo projection image (MR)."

[0046] <Step 108> The MR-CT image alignment unit 50 creates a superimposed image 220 of a pseudo projection image (CT) (CT projection image 211) and a pseudo projection image (MR) (MR projection image 212). This superimposed image 220 depicts the morphology 211a of the bones and the like of the subject 1 and tubular tissue 212a, and is an image (pseudo projection image) simulating an X-ray image taken while contrasting the tubular tissue with a contrast agent during surgery.

[0047] <Step 121> On the other hand, the intraoperative two-dimensional image acquisition unit 10 captures an X-ray image 203 of the subject 1 captured by the X-ray imaging device 2 during the operation. Hereinafter, the X-ray image 203 is also referred to as an "intraoperative image."

[0048] <Step 122> The intraoperative image feature point extraction unit 53 extracts feature points 253 indicating the position of a predetermined part of the subject 1 from the intraoperative image (X-ray image 203) (FIG. 3(d)). For example, when the tip of the treatment tool 301 is placed at a part (e.g., the nipple) predetermined by the user, the intraoperative image feature point extraction unit 53 extracts the tip of the treatment tool 301 from the X-ray image 203 by image processing, and sets the extracted position as the feature point 253.

[0049] <Step 109> The CT-X-ray image alignment unit 54 aligns the pseudo projection image (CT) (CT projection image 211) and the intraoperative image (X-ray image 203) in the parallel translation and rotation directions so that high-contrast parts of the CT projection image 211 (shapes 211a such as bones and body contours) match with high-contrast parts of the X-ray image 203 (shapes 203a such as bones and body contours), and determines parameters for matching coordinates in the parallel translation and rotation directions (FIGS. 3(c) and (d)).

[0050] <Step 110> The MR-X-ray image alignment unit 55 aligns the pseudo projection image (MR) (MR projection image 212) and the intraoperative image (X-ray image 203) in the translation direction and rotation direction using the parameters determined by the CT-X-ray image alignment unit 54 in step 109. Since the MR image 202 and the CT image 201 have been aligned in step 103, the MR projection image 212 can be aligned with the X-ray image 203 using the parameters determined for the CT projection image 211 in step 109.

[0051] <Step 111> Next, the MR-X-ray image alignment unit 55 displays the feature points 251 of the CT image 201 extracted in step 104 and the feature points 251 of the MR image 202 extracted in step 105 on the pseudo projection image (MR) (MR projection image 212).

[0052] <Step 112> The MR-X-ray image alignment unit 55 uses one of the feature points 251 of the CT image 201 and the feature points 252 of the MR image 202 displayed on the pseudo projection image (MR) (MR projection image 212) in step 111 and the feature points 253 of the intraoperative image (X-ray image 203) to further align the pseudo projection image (MR) (MR projection image 212) by moving it in a parallel direction so that it matches the intraoperative image (X-ray image 203).

[0053] For example, the MR-X-ray image alignment unit 55 aligns the pseudo projection image (MR) (MR projection image 212) with the intraoperative image (X-ray image 203) so that the feature point 251 of the CT image 201 matches the feature point 253 of the X-ray image 203. The MR-X-ray image alignment unit 55 may also align the pseudo projection image (MR) (MR projection image 212) with the intraoperative image (X-ray image 203) so that the feature point 252 of the MR image 202 matches the feature point 253 of the X-ray image 203.

[0054] <Step 113> In step 112 , the image superimposing unit 56 superimposes the intraoperative image (X-ray image 203 ) registered by the MR-X-ray image registration unit 55 on the pseudo projection image (MR) (MR projection image 212 ) to generate a superimposed image 230 .

[0055] <Step 114> The image superimposing unit 56 displays the superimposed image 230 as a guide image on the display unit 40 (FIG. 3(f)). At the same time, the superimposed image 220 (pseudo projection image) generated by the MR-CT image alignment unit 50 in step 108, which is a superimposed image of the pseudo projection image (CT) (CT projection image 211) and the pseudo projection image (MR) (MR projection image 212), is displayed on the display unit 40.

[0056] As described above, the medical image processing system 100 of the first embodiment can use the three-dimensional CT image 201 and MR image 202 acquired in advance to display a superimposed image 230 (guide image) in which a pseudo image of the tubular tissue 212a observed if contrast is applied is superimposed on the intraoperative image (X-ray image 203) before contrast is applied, which is taken by the surgeon during the procedure with the X-ray imaging device 2. This allows the surgeon to visually confirm the tubular tissue 212a, which is the target of the procedure, on the intraoperative image (X-ray image 203) before using a contrast agent.

[0057] In addition to the superimposed image 230 (guide image) of the intraoperative image (X-ray image 203), the display unit also displays a superimposed image 220 (pseudo projection image) of the pseudo projection image (CT) (CT projection image 211) and the pseudo projection image (MR) (MR projection image 212) generated in step 114, allowing the surgeon to compare the pseudo projection image with the intraoperative X-ray image 203 with high accuracy.

[0058] In the superimposed image 220 (pseudo projection image), the surgeon can grasp the morphology 211a of the body contour and bones of the subject 1 from the CT projection image 211 acquired in advance, and can also grasp the tubular tissue 212a from the MR projection image 212.

[0059] <<Embodiment 2>> The configuration of a medical image processing system 200 according to the second embodiment will be described.

[0060] In the first embodiment, in steps 111 and 112, the feature points 251 of the CT image 201 and the feature points 252 of the MR image 202 are displayed on a pseudo projection image (MR) (MR projection image 212), and the feature points 251 or 252 are aligned with the feature points 253 of the intraoperative image (X-ray image 203).

[0061] Ideally, the position coordinates of feature points that indicate the same part in multiple 3D medical images should match, but 3D medical images of a subject (human body) captured with different modalities do not necessarily match due to bodily movements such as breathing. Specifically, this is because intraoperative X-ray images are taken while the subject is breathing naturally, whereas X-ray CT images are taken while in a maximum inhalation state and MR images are taken while the subject is breathing naturally or while holding their breath.

[0062] In the second embodiment, a position or area where the feature point 253 of the X-ray image 203 is highly likely to exist is calculated using the position coordinates of the feature point 251 of the CT image 201 and the feature point 252 of the MR image 202, and imaging conditions such as the respiratory state when these images were captured, and the area where the presence probability is high is aligned with the feature point 253 of the X-ray image 203. A specific description will be given below.

[0063] Fig. 4 is a block diagram showing the configuration of a medical image processing system 200 of the second embodiment. Fig. 5 is a flowchart showing the operation of the medical image processing system 200. Figs. 6(a) and (c) are diagrams showing an example of feature points 251 and 252 and an existence region 250.

[0064] As is clear from FIG. 4, the medical image processing system 200 has a configuration in which a feature point existing region calculation unit 57 is added to the medical image processing system 100 of the first embodiment.

[0065] The feature point existence area calculation unit 57 calculates an existence area 250, which is an area where the feature point 253 of the X-ray image 203 may exist, using the coordinates of the feature point 252 of the MR projection image 212 and the feature point 251 of the CT projection image 211 (see Figures 6(a) and (c)).

[0066] The feature point existence region calculation section 57 further calculates the distribution of the existence probability of the feature point 253 in the existence region 250 based on the imaging conditions.

[0067] Specifically, the shape of the existence region 250 and the distribution of the existence probability of the feature point 253 can be determined in advance based on a combination of the respiratory state at the time when the X-ray image 203, the CT image 201, and the MR image 202 are captured, whether or not general body movement is taken into account, and whether or not pulsation is taken into account.

[0068] In general, the X-ray image 203 is often taken under natural breathing, the CT image 201 is taken under maximum inspiration, and the MR image 202 is often taken under natural breathing or breath-holding.

[0069] For example, as shown in the table of FIG. 7, when considering general body movements other than breathing, feature point 253 moves in all directions due to body movements, so the range of movement (existence area 250) becomes nearly circular, and the existence probability of feature point 253 becomes almost uniform within that shape.

[0070] Furthermore, when only natural breathing is taken into consideration, the feature point 253 moves back and forth in a specific direction, the range of movement (existence area 250) has a shape close to an ellipse with the direction of movement as its major axis, and the probability of the presence of the feature point 253 tends to be high at both ends of the major axis of the ellipse.

[0071] In addition, the coordinates of the feature point of the image taken with natural breathing (for example, feature point 252 in MR image 202) and the feature point of the image taken at the maximum inspiration position (for example, feature point 251 in CT image 201), and the corresponding existence region 250 of feature point 253 in X-ray image 203 are ellipses with the direction of movement as their major axis, since they move back and forth in a specific direction, and the existence probability is low near feature point 251 in the image at the maximum inspiration position. This is because the maximum inspiration position is not reached under natural breathing. On the other hand, the existence probability is high within the movement range of natural breathing.

[0072] In addition, the coordinates of the feature point of the image taken at the breath-holding position (e.g., feature point 252 in MR image 202 during breath-holding), the coordinates of the feature point of the image taken at the maximum inhalation position (e.g., feature point 251 in CT image 201), and the existence area 250 of feature point 253 in X-ray image 203 relative to them are in a shape close to an ellipse with the direction of movement as the long axis, since they are reciprocating movements in a specific direction, and the existence probability is high for feature point 252 in the image at the breath-holding position.

[0073] On the other hand, when only pulsation is taken into consideration, the movement is reciprocating in a specific direction, the range of movement (existence area 250) is elliptical, and there are areas at both ends of the reciprocating movement where the probability of the presence of feature point 253 is high.

[0074] Taking the above characteristics into consideration, the shape of the existence region 250 and the distribution of the existence probability of the feature point 253 can be determined in advance for each imaging condition.

[0075] The feature point existence region calculation unit 57 calculates the shape of the existence region 250 of the feature point 253 and the existence probability using the coordinates of the feature points 251 and 252, based on the shape of the existence region 250 for each predetermined imaging condition and the distribution of the existence probability of the feature point 253, according to the imaging conditions of the MR projection image 212, the CT projection image 211, and the X-ray image.

[0076] The MR-X-ray image alignment unit 55 aligns the feature point 253 of the X-ray image 203 to a position with a high probability of existence based on the existence region 250 calculated by the feature point existence region calculation unit 57 and the distribution of the existence probability within the existence region 250.

[0077] The other components (10, 20, 30, 40, 50-56) of the medical image processing system 200 are similar to those of the first embodiment, and therefore will not be described.

[0078] Next, the operation of each unit of the medical image processing system 200 will be described with reference to the flow in Fig. 5. Note that the description of operations similar to those in the flow in Fig. 2 of the first embodiment will be omitted, and different operations will be mainly described.

[0079] <Steps 101~110> The pseudo projection image (MR) (MR projection image 212) and the intraoperative image (X-ray image 203) are aligned in the translation direction and the rotation direction in the same manner as in the first embodiment.

[0080] <Step 131> The feature point existence region calculation unit 57 calculates the shape of the existence region 250 of the feature point 253 and the existence probability using the coordinates of the feature points 251, 252, based on the shape of the existence region 250 for each predetermined imaging condition and the distribution of the existence probability of the feature point 253, according to the imaging conditions of the MR projection image 212, the CT projection image 211, and the X-ray image (see Figures 6(a) and (d)).

[0081] <Step 132> The MR-X-ray image alignment unit 55 displays the feature points 251 of the CT image 201, the feature points 251 of the MR image 202, the existence area 250 calculated in step 131, and the probability distribution on a pseudo projection image (MR) (MR projection image 212) (see FIG. 8(a)).

[0082] moreover, <Step 133> The MR-X-ray image alignment unit 55 aligns the pseudo projection image (MR) (MR projection image 212) by translating the pseudo projection image (MR) (MR projection image 212) so that the feature point 251 of the CT image 201 and the feature point 252 of the MR image 202 displayed on the pseudo projection image (MR) (MR projection image 212) coincides with the feature point 253 of the intraoperative image (X-ray image 203) at a position with a high probability distribution, using the probability distribution (Figures 6(b), (c), (e), (f), Figures 8(b) and (c)).

[0083] <Steps 113-114> Similar to steps 113 and 114 of the flow in FIG. 2, the image superimposition unit 56 superimposes the intraoperative image (X-ray image 203) aligned by the MR-X-ray image alignment unit 55 in step 133 on the pseudo projection image (MR) (MR projection image 212) to generate a superimposed image 230 (FIG. 8(c)).

[0084] The image superimposing unit 56 displays the superimposed image 230 as a guide image on the display unit 40. At the same time, the image superimposing unit 56 displays the superimposed image 220 (pseudo projection image) on the display unit 40.

[0085] According to the medical image processing system 200 of the second embodiment, in addition to the effects achieved by the medical image processing system 100 of the first embodiment, the following effects can be further achieved. That is, a position where the presence probability of the feature point 253 of the X-ray image 203 is high can be calculated from the feature points 251 and 252, taking into consideration the body movement of the subject 1 according to imaging conditions such as the respiratory state when the CT image 201 and MR image 202 captured in advance and the respiratory state when the X-ray image 203 is captured during surgery. By positioning the MR projection image 212 on the X-ray image 203 so that the feature point 253 overlaps with the calculated position where the presence probability is high, the tubular tissue of the MR projection image 212 can be accurately superimposed and displayed on the X-ray image 203. [Explanation of symbols]

[0086] 1. Subject 2 X-ray equipment 3 Medical image server 10 Intraoperative 2D image acquisition unit 20 Pre-3D image acquisition section 30 Input section 40 Display section 50 MR-CT image alignment unit 51 Pre-image feature point extraction unit 52 Projection calculation section 53 Intraoperative image feature point extraction unit 54 CT-X-ray image alignment unit 55 MR-X-ray image alignment unit 56 Image Overlay Unit 57 Feature point existence area calculation unit 100 Medical image processing system 150 Image processing section 200 Medical Image Processing System 201 CT images 202 MR Imaging 203 X-ray image 203a Form 211 CT projection image 211a Form 212 MR Projection Images 212a Tubular tissue 220 Overlay Images 230 Overlay Images 250 Existence Area 251 feature points 252 feature points 253 feature points 301 Treatment tools

Claims

1. a pre-image acquisition unit that captures an MR image of a subject taken in advance by a magnetic resonance imaging device and a CT image of the subject taken in advance by an X-ray CT device; an intraoperative image acquisition unit that captures X-ray images of the subject during surgery; an MR-CT image registration unit that registers the MR image and the CT image and determines a correspondence relationship between the coordinates of the MR image and the CT image; a CT-X-ray image registration unit that registers the CT image and the X-ray image and determines a correspondence relationship between the coordinates of the CT image and the X-ray image; a preliminary image feature point extraction unit that extracts feature points indicating positions of predetermined parts of a subject in the MR image and the CT image, respectively; an intraoperative image feature point extraction unit that extracts feature points that indicate a position of the part of the subject in the X-ray image; an MR-X-ray image alignment unit that aligns the X-ray image and the MR image; an image superimposition unit that superimposes the X-ray image and the MR image that have been aligned by the MR-X-ray image alignment unit to generate a superimposed image and display the superimposed image as a guide image; a MR-X-ray image alignment unit aligns the X-ray image and the MR image using a correspondence between the coordinates of the MR image and the CT image and a correspondence between the coordinates of the CT image and the X-ray image, and then further aligns the X-ray image and the MR image at least in the translation direction using the coordinates of at least one of the feature points of the MR image and the feature points of the CT image and the coordinates of the feature points in the X-ray image.

2. 2. The medical image processing system according to claim 1, wherein the MR image includes an image of the tubular tissue of the subject, and the X-ray image includes an image of a treatment tool inserted into the subject.

3. 2. The medical image processing system according to claim 1, wherein the MR-X-ray image alignment unit, when performing alignment using the coordinates of the feature points, displays a point on the MR image indicating the coordinates of at least one of the feature points of the MR image and the feature point of the CT image, and displays a point on the X-ray image indicating the coordinates of the feature point, and performs alignment using a plurality of the points.

4. 2. The medical image processing system according to claim 1, further comprising a feature point existence region calculation unit, the feature point existence region calculation unit calculates an existence region, which is a region where the feature point may exist, by using coordinates of the feature point of the MR image and the feature point of the CT image; The medical image processing system is characterized in that, when performing alignment using the coordinates of the feature points, the MR-X-ray image alignment unit displays on the MR image the area of ​​existence and a point indicating the coordinates of at least one of the feature points of the MR image and the feature point of the CT image, and displays on the X-ray image a point indicating the coordinates of the feature point, and performs alignment using the multiple points and the area of ​​existence.

5. 5. The medical image processing system according to claim 4, wherein the feature point existence region calculation unit further calculates a distribution of the existence probability of the feature point in the existence region based on an imaging condition, The medical image processing system is characterized in that the MR-X-ray image alignment unit aligns the X-ray image and the MR image based on the existence probability.

6. 2. The medical image processing system according to claim 1, wherein the CT-X-ray image alignment unit aligns the images at least in the rotational direction so that the shape of a part of the CT image where the contrast is higher than a predetermined value coincides with the shape of a part of the X-ray image where the contrast is higher than a predetermined value.

7. 2. The medical image processing system according to claim 1, wherein the MR image and the CT image are both three-dimensional images; The medical image processing system according to claim 1, wherein the preliminary image feature point extraction unit extracts the feature points from the MR image and the CT image, which are three-dimensional images.

8. 2. The medical image processing system according to claim 1, wherein the MR image and the CT image are both three-dimensional images; The medical image processing system is characterized in that the MR-CT image alignment unit projects the MR image and the CT image of a three-dimensional image onto a two-dimensional image in a predetermined projection direction, and determines the correspondence between the coordinates of the MR image and the CT image of the projected two-dimensional image.

9. 2. The medical image processing system according to claim 1, wherein the MR-CT image alignment unit generates a superimposed image of the MR image and the CT image, and displays the superimposed image together with the guide image.

10. 2. The medical image processing system according to claim 1, further comprising an input unit for receiving designation of positions in the MR image and the CT image from a user, The medical image processing system is characterized in that the preliminary image feature point extraction unit extracts, as the feature points, positions on the MR image and the CT image that the input unit accepts as designation from a user.

11. 2. The medical image processing system according to claim 1, wherein the intraoperative image feature point extraction unit detects an insertion position of a treatment tool included in the X-ray image and extracts the detected insertion position of the treatment tool as the feature point.

12. A step of acquiring an MR image of a subject taken in advance by a magnetic resonance imaging apparatus and a CT image of the subject taken in advance by an X-ray CT apparatus; capturing an X-ray image of the subject during surgery; A step of registering the MR image and the CT image and determining a correspondence relationship between coordinates of the MR image and the CT image; A step of registering the CT image and the X-ray image and determining a correspondence relationship between coordinates of the CT image and the X-ray image; extracting feature points indicating positions of predetermined parts of a subject in the MR image and the CT image, respectively; extracting feature points indicative of a position of the region of the subject in the X-ray image; registering the X-ray image and the MR image; an image processing method including a step of superimposing the X-ray image and the MR image that have been registered by an MR-X-ray image registration unit, generating a superimposed image, and displaying the superimposed image as a guide image, a step of aligning the X-ray image and the MR image by using a correspondence between coordinates of the MR image and the CT image and a correspondence between coordinates of the CT image and the X-ray image, and then further aligning the X-ray image and the MR image at least in a translation direction by using coordinates of at least one of the feature points of the MR image and the feature points of the CT image and the coordinates of the feature points of the X-ray image.

Images