Information processing apparatus, information processing method, and program

Abstract

The present technology relates to an information processing apparatus, an information processing method, and a program, each of which is designed to provide an image that enables an operator to more safely and efficiently perform treatment. The information processing apparatus according to the present technology comprises a screen generation unit that generates a screen including a composite 3D model generated by combining a 3D vascular model representing a three-dimensional shape of a blood vessel of a patient with a 3D catheter model, which is a 3D model representing a three-dimensional shape of a catheter inserted into the blood vessel of the patient, generated on the basis of two fluoroscopic images captured simultaneously at different capture angles with an X-ray imaging apparatus. The present technology is applicable to, for example, an IVR treatment system.

Description

Information processing device, information processing method, and program

[0001] The present technology relates to an information processing device, an information processing method, and a program, and in particular to an information processing device, an information processing method, and a program that enable a surgeon to provide images that allow the surgeon to perform treatment more safely and efficiently.

[0002] Interventional radiology (IVR) is a common treatment method for performing vascular catheterization while viewing real-time fluoroscopic images captured by an X-ray imaging device. For example, Patent Document 1 describes a catheterization procedure support robot for supporting procedures using a catheter.

[0003] Conventionally, in interventional radiology, two roadmaps with different imaging angles are displayed, and the surgeon looks at the two roadmaps to understand the vascular structure and catheter position. The roadmap is an image in which a real-time fluoroscopic image is superimposed on a vascular map showing the shape of the blood vessels. The vascular map is generated based on angiographic fluoroscopic images taken using techniques such as DSA (Digital Subtraction Angiography).

[0004] JP 2023-39814 A

[0005] It is difficult for surgeons to intuitively grasp the three-dimensional structure of blood vessels by looking at the two roadmaps, which can result in slow IVR treatment or the surgeon inserting the catheter into the wrong blood vessel.It also takes a considerable amount of time to become able to intuitively grasp the three-dimensional structure of blood vessels by looking at the two roadmaps.

[0006] There is a method for displaying a 3D roadmap in which real-time fluoroscopic images are superimposed on a 3D vascular model that shows the three-dimensional shape of blood vessels. However, in the 3D roadmap, the blood vessels appear to overlap on the same plane, making it difficult for the surgeon to understand the depth relationships of the blood vessels.

[0007] The present technology has been developed in light of these circumstances, and is intended to provide images that enable surgeons to perform surgery and treatment more safely and efficiently.

[0008] An information processing device according to one aspect of the present technology includes a screen generation unit that generates a screen including a composite 3D model obtained by combining a 3D vascular model showing the three-dimensional shape of a patient's blood vessels with a 3D catheter model, which is a 3D model showing the three-dimensional shape of a catheter inserted into the blood vessels and is generated based on two fluoroscopic images simultaneously captured at different imaging angles by an X-ray imaging device.

[0009] An information processing device according to one aspect of the present technology includes generating a screen including a composite 3D model obtained by combining a 3D vascular model showing the three-dimensional shape of a patient's blood vessels with a 3D catheter model, which is a 3D model showing the three-dimensional shape of a catheter inserted into the blood vessels and is generated based on two fluoroscopic images captured simultaneously at different imaging angles by an X-ray imaging device.

[0010] A program according to one aspect of the present technology causes a computer to perform processing including generating a screen including a composite 3D model obtained by combining a 3D vascular model showing the three-dimensional shape of a patient's blood vessel with a 3D catheter model, which is a 3D model showing the three-dimensional shape of a catheter inserted into the blood vessel, and which is a 3D model generated based on two fluoroscopic images captured simultaneously at different imaging angles by an X-ray imaging device.

[0011] In one aspect of the present technology, a screen is generated that includes a composite 3D model that combines a 3D vascular model that shows the three-dimensional shape of a patient's blood vessels with a 3D catheter model that shows the three-dimensional shape of a catheter inserted into the blood vessels, the 3D model being generated based on two fluoroscopic images captured simultaneously at different imaging angles by an X-ray imaging device.

[0012] 15A and 15B are diagrams illustrating an example of the layout of an IVR treatment system according to an embodiment of the present technology; FIG. 16A is a diagram illustrating an example of the configuration of an X-ray imaging device; FIG. 16B is a diagram illustrating an example of a roadmap; FIG. 16C is a diagram illustrating an example of the configuration of a spatial layout roadmap; FIG. 16D is a first diagram illustrating an example of the spatial layout of roadmaps in a spatial layout roadmap; FIG. 16E is a second diagram illustrating an example of the spatial layout of roadmaps in a spatial layout roadmap; FIG. 16F is a diagram illustrating another example of the configuration of a spatial layout roadmap; FIG. 16G is a diagram illustrating an example of an image displayed together with the spatial layout roadmap; FIG. 16H is a diagram illustrating an example of a screen displayed on an examination room monitor; FIG. 16I is a diagram illustrating the display of an image of a synthetic 3D model that reproduces motion parallax; FIG. 16I is a diagram illustrating an example of a display of a spatial layout roadmap when the viewpoint of the spatial layout roadmap is specified using a direction based on the patient; FIG. 16I is a diagram illustrating a method for adjusting the thickness of a 3D blood vessel model based on an angiographic fluoroscopic image; FIG. 16I is a block diagram illustrating a detailed example of the configuration of a spatial layout roadmap generation unit; FIG. 16I is a flowchart illustrating processing performed by an information processing device; FIG. 16I is a flowchart illustrating 3D catheter model generation processing performed in step S3 of FIG. 15; FIG. 16I is a diagram illustrating size adjustment of a roadmap; FIG. 16I is a diagram illustrating preprocessing; FIG. 16I is a diagram illustrating a method for calculating an intersection point of a normal from a catheter pixel. 16 is a diagram illustrating a filter process using a 3D blood vessel model. FIG. 17 is a flowchart illustrating a superimposition process performed in step S4 of FIG. 15. FIG. 18 is a flowchart illustrating a spatial layout roadmap generation process performed in step S5 of FIG. 15. FIG. 19 is a diagram illustrating an example of a sub-monitor. FIG. 20 is a block diagram illustrating an example of the configuration of computer hardware.

[0013] Hereinafter, an embodiment of the present technology will be described. The description will be made in the following order: 1. Overview of IVR treatment system 2. Configuration and operation of information processing device 3. Other

[0014] 1. Overview of IVR Treatment System FIG. 1 is a diagram showing an example of the layout of an IVR treatment system according to an embodiment of the present technology.

[0015] The IVR treatment system in Fig. 1 is composed of an X-ray imaging device 1, an examination room monitor 2, an information processing device 11, and an operation room monitor 12. For example, the X-ray imaging device 1 and the examination room monitor 2 are placed in an examination room, and the information processing device 11 and the operation room monitor 12 are placed in an operation room. In the example in Fig. 1, the examination room and the operation room are adjacent to each other.

[0016] A bed is placed near the X-ray imaging device 1. The X-ray imaging device 1 uses X-rays to capture an image of the patient on the bed to obtain a fluoroscopic image. The X-ray imaging device 1 can also obtain an angiographic fluoroscopic image that depicts only the patient's blood vessels using a technique such as DSA. The fluoroscopic image or angiographic fluoroscopic image shows the patient's organs, such as blood vessels, as well as a catheter inserted into the blood vessels.

[0017] FIG. 2 is a diagram showing an example of the configuration of the X-ray imaging device 1.

[0018] 2, the X-ray imaging device 1 is configured with an arm unit 23-1 to which an X-ray generator 21-1 and a detector 22-1 are attached so as to face each other, and an arm unit 23-2 to which an X-ray generator 21-2 and a detector 22-2 are attached so as to face each other. The X-ray imaging device 1 is a biplane type device that can simultaneously perform imaging at different imaging angles using the two arms 23-1 and 23-2.

[0019] In the following, when there is no need to particularly distinguish between the X-ray generating devices 21-1 and 21-2, they will simply be referred to as the X-ray generating device 21, when there is no need to particularly distinguish between the detecting devices 22-1 and 22-2, they will simply be referred to as the detecting device 22, and when there is no need to particularly distinguish between the arm units 23-1 and 23-2, they will simply be referred to as the arm unit 23.

[0020] In the X-ray imaging device 1, a patient is placed between the X-ray generator 21 and the detector 22, and a fluoroscopic image is captured by projecting shadows of blood vessels and the like created by the irradiation of X-rays from the X-ray generator 21 onto the detector 22.

[0021] 1 is a large display device that is placed in a position that is easily visible to the operator during IVR treatment, for example. The examination room monitor 2 displays a roadmap and the like under the control of an information processing device 11.

[0022] FIG. 3 is a diagram showing an example of a roadmap.

[0023] The roadmap is an image in which a real-time fluoroscopic image captured by the X-ray imaging device 1 is superimposed on a vascular map showing the shape of blood vessels, and is an image showing the shape of blood vessels and the position and shape of the catheter. The vascular map is generated based on angiographic fluoroscopic images captured using a technique such as DSA.

[0024] The examination room monitor 2 displays, for example, a roadmap M1 shown on the left side of Fig. 3 and a roadmap M2 shown on the right side of Fig. 3. The roadmap M1 is an image in which a fluoroscopic image captured by, for example, the detection device 22-1 is superimposed on a vascular map corresponding to the imaging angle of the fluoroscopic image. The roadmap M2 is an image in which a fluoroscopic image captured by, for example, the detection device 22-2 is superimposed on a vascular map corresponding to the imaging angle of the fluoroscopic image.

[0025] In the following, the imaging angle of the roadmap refers to the imaging angle of the fluoroscopic image included in the roadmap, and the imaging angle of the fluoroscopic image includes both the angle of the X-ray generator 21 and the angle of the detection device 22 relative to the patient at the time of imaging the fluoroscopic image.

[0026] 1 is configured with a PC or the like, and generates a screen to be displayed on the examination room monitor 2 based on fluoroscopic images captured by the X-ray imaging device 1 and reference information acquired by other medical devices placed in the examination room. The reference information is information indicating the patient's electrocardiogram waveform, heart rate, pulse wave, blood oxygen saturation, arterial blood pressure, etc.

[0027] The operation room monitor 12 displays the screen displayed on the examination room monitor 2 and various images and information arranged on the screen.

[0028] A technician in the examination room can operate the screen of the examination room monitor 2 by operating the information processing device 11 while looking at the display on the operation room monitor 12 .

[0029] Conventionally, in IVR treatment, two roadmaps with different imaging angles (e.g., roadmaps M1 and M2 in Figure 3) are displayed on the examination room monitor 2, and the surgeon looks at the two roadmaps M1 and M2 to understand the three-dimensional structure of the blood vessels and the position of the catheter.

[0030] It is difficult for surgeons to intuitively grasp the three-dimensional structure of blood vessels by looking at the two roadmaps M1 and M2, which can result in IVR treatment taking a long time or the surgeon inserting the catheter into the wrong blood vessel.It also takes a considerable amount of time to be able to intuitively grasp the three-dimensional structure of blood vessels by looking at the two roadmaps M1 and M2.

[0031] There is a method of displaying a 3D roadmap by superimposing real-time fluoroscopic images onto a 3D vascular model, which is a 3D model that shows the three-dimensional shape of blood vessels. However, in the 3D roadmap, the blood vessels appear to overlap on the same plane, making it difficult for the surgeon to understand the depth relationships of the blood vessels.

[0032] Therefore, the information processing device 11 of the present technology generates a screen including a spatial arrangement roadmap, which is an image of a 3D vascular model arranged in a virtual space and a roadmap (perspective image) arranged so as to face the 3D vascular model at a position in the virtual space where the angle with respect to the 3D vascular model matches the imaging angle of the roadmap itself, viewed from a virtual viewpoint. In the present disclosure, the roadmap facing the 3D vascular model means that when the 3D vascular model is translated, the 3D vascular model and the blood vessels shown on the roadmap match to some extent.

[0033] FIG. 4 is a diagram showing an example of the configuration of a spatial layout roadmap.

[0034] In the spatial layout roadmap shown in FIG. 4, the sides of the 3D blood vessel model Mod1 are surrounded by semi-transparent roadmaps M1 and M2.

[0035] In the virtual space, the roadmaps M1 are placed at two symmetrical positions on either side of the 3D vascular model Mod1. Each roadmap M1 is placed so as to directly face the 3D vascular model Mod1 at a position where the angle relative to the 3D vascular model Mod1 in the virtual space matches the imaging angle of the roadmap M1 itself.

[0036] In other words, one roadmap M1 is placed at a position in virtual space corresponding to the position of the X-ray generator 21-1 at the imaging timing of the roadmap M1, and the other roadmap M1 is placed at a position in virtual space corresponding to the position of the detector 22-1 at the imaging timing of the roadmap M1.

[0037] In the virtual space, the two roadmaps M2 are placed at two symmetrical positions with respect to the 3D vascular model Mod1. Each roadmap M2 is placed so as to directly face the 3D vascular model Mod1 at a position where the angle relative to the 3D vascular model Mod1 in the virtual space matches the imaging angle of the roadmap M2 itself.

[0038] In other words, one roadmap M2 is placed at a position in virtual space corresponding to the position of the X-ray generator 21-2 at the imaging timing of the roadmap M2, and the other roadmap M2 is placed at a position in virtual space corresponding to the position of the detector 22-2 at the imaging timing of the roadmap M2.

[0039] 5 and 6 are diagrams showing examples of the spatial layout of the roadmap in the spatial layout roadmap.

[0040] In the spatial arrangement roadmap shown in Figure 5A, as described above with reference to Figure 4, four roadmaps (two roadmaps M1 and two roadmaps M2) are arranged around the 3D vascular model Mod1.

[0041] In the spatial arrangement roadmap shown in Fig. 5B, two roadmaps (one roadmap M1 and one roadmap M2) are arranged around the 3D vascular model Mod1. As shown in the upper part of Fig. 5B, the roadmaps M1 and M2 may be arranged to surround the front side of the 3D vascular model Mod1, or as shown in the lower part of Fig. 5B, the roadmaps M1 and M2 may be arranged to surround the back side of the 3D vascular model Mod1.

[0042] In the spatial arrangement roadmap shown in Fig. 5C, two roadmaps (one roadmap M1 and one roadmap M2) are arranged around the 3D vascular model Mod1. As shown in the upper part of Fig. 5C, the roadmaps M1 and M2 may be arranged to surround the left side of the 3D vascular model Mod1, or as shown in the lower part of Fig. 5B, the roadmaps M1 and M2 may be arranged to surround the right side of the 3D vascular model Mod1.

[0043] In the spatial arrangement roadmap shown in A of Fig. 6, two roadmaps (one roadmap M1 and one roadmap M2) are arranged around a 3D vascular model Mod1. As shown in A of Fig. 6, the roadmap M1 may be arranged at a position where it intersects with the 3D vascular model Mod1.

[0044] In the spatial arrangement roadmap shown in FIG. 6B, one roadmap M2 is arranged around the 3D blood vessel model.

[0045] In the spatial position roadmap shown in Fig. 6C, two roadmaps (one roadmap M1 and one roadmap M2) are arranged around the 3D vascular model Mod1. In Fig. 5A to Fig. 5C and Fig. 6A, the roadmap M1 and the roadmap M2 are arranged so that at least a portion of them are in contact with each other in the virtual space, but as shown in Fig. 6C, the roadmap M1 and the roadmap M2 may be arranged apart from each other.

[0046] FIG. 7 is a diagram showing another example of the configuration of the spatial layout roadmap.

[0047] As shown in FIG. 7, the spatial configuration roadmap may be an image including a 3D catheter model Mod2, which is a 3D model showing the three-dimensional shape of the catheter.

[0048] In the spatial arrangement roadmap including the 3D catheter model Mod2, the tip of the 3D catheter model Mod2 is connected to the tip of the catheter shown on the roadmap M1 and the roadmap M2 by a line (e.g., a straight line). In the example of Fig. 7, the tip of the 3D catheter model Mod2 is connected to the tip of the catheter shown on the roadmap M1 and the roadmap M2 by a dotted line, but the type of line is not particularly limited.

[0049] A straight line connects the tip of the 3D catheter model Mod2 with the tip of the catheter shown on the roadmap M1 and roadmap M2, making it easier for the surgeon to intuitively understand the correspondence between the 3D catheter model Mod2 and the catheter shown on the roadmap, as well as the correspondence between the 3D blood vessel model around the 3D catheter model Mod2 and the blood vessels around the catheter shown on the roadmap.

[0050] As shown in Figure 8, an image Pi1 showing the inside of the 3D vascular model Mod1 as seen from the tip of the 3D catheter model Mod2 may be displayed together with a spatial layout roadmap including the 3D vascular model Mod1, roadmap M1, roadmap M2, and catheter 3D model.

[0051] By displaying the image Pi1, the surgeon can easily intuitively grasp to which branch point in the blood vessel the catheter should be advanced.

[0052] In the spatial arrangement roadmaps of Figures 7 and 8, the roadmaps M1 and M2 are arranged to surround the back side of the 3D vascular model Mod1, but as described above with reference to Figures 5 and 6, the roadmaps M1 and M2 can be arranged in the virtual space in various forms.

[0053] FIG. 9 is a diagram showing an example of a screen displayed on the examination room monitor 2.

[0054] In the example of Figure 9, roadmaps M1 and M2 are arranged vertically on the left side of the screen, and perspective image Pi11 included in roadmap M1 and perspective image Pi12 included in roadmap M2 are arranged vertically in the center of the screen.

[0055] Reference information R1 is displayed in the upper right corner of the screen, and below the reference information is an image of the composite 3D model Mod3, which is a 3D model created by combining the 3D blood vessel model Mod1 and the 3D catheter model. By looking at the composite 3D model Mod3, the surgeon can intuitively grasp the three-dimensional positional relationship between the catheter and the blood vessel.

[0056] A spatial layout roadmap M11 is displayed on the lower right side of the screen.

[0057] The examination room monitor 2 is equipped with a tracking camera 31 used to detect the direction of the surgeon's line of sight. The information processing device 11 detects the direction of the surgeon's line of sight based on the image captured by the tracking camera 31, and displays an image of a synthetic 3D model Mod3 that reproduces motion parallax and a spatial layout roadmap based on the detected direction of the surgeon's line of sight.

[0058] 10 , when the operator U1 views the examination room monitor 2 from the left, an image of the composite 3D model Mod3 viewed from the virtual viewpoint on the left is displayed on the examination room monitor 2. When the operator U1 views the examination room monitor 2 from the front, an image of the composite 3D model Mod3 viewed from the virtual viewpoint on the front is displayed on the examination room monitor 2. When the operator U1 views the examination room monitor 2 from the right, an image of the composite 3D model Mod3 viewed from the virtual viewpoint on the right is displayed on the examination room monitor 2.

[0059] In this way, the information processing device 11 performs rendering based on the surgeon's line of sight, thereby generating an image of the composite 3D model Mod3 and a spatial layout roadmap as viewed from a virtual viewpoint corresponding to the surgeon's line of sight.

[0060] By simply moving their head, the surgeon can check from various angles the relative positions of the 3D vascular model Mod1 and the 3D catheter model Mod2 contained in the composite 3D model Mod3, as well as the correspondence between the composite 3D model Mod3 and the blood vessels and catheters shown on the roadmap.

[0061] The viewpoint (virtual viewpoint) of the image of the composite 3D model Mod3 and the spatial layout roadmap does not have to be linked to the viewpoint of the surgeon. For example, if the surgeon tilts his / her head 15°, the viewpoint of the image of the composite 3D model Mod3 and the spatial layout roadmap may be tilted 30°. The viewpoint of the image of the composite 3D model Mod3 and the spatial layout roadmap may be set by the information processing device 11 in accordance with gesture operation, voice operation, mouse operation, controller operation, etc. by the surgeon. In this case, the tracking camera 31 may or may not be provided on the examination room monitor 2.

[0062] The viewpoint of the image of the composite 3D model Mod3 and the spatial layout roadmap may be specified using a direction relative to the patient (left, right, anterior, posterior, cranial, caudal). For example, when "anterior" is specified, the spatial layout roadmap M11 viewed from a virtual viewpoint corresponding to the front of the patient is displayed on the examination room monitor 2, as shown in FIG. 11. It is also possible to specify the viewpoint of the image of the composite 3D model Mod3 and the spatial layout roadmap in finer increments, such as "LAO (left anterior oblique view) 45°."

[0063] In addition, in the composite 3D model and spatial layout roadmap, the transparency of the 3D model and roadmap can be adjusted as desired.

[0064] The synthetic 3D model image and the spatial layout roadmap may be displayed at life-size, which makes it easier for the surgeon to intuitively grasp the size of the blood vessel and the distance to move the catheter.

[0065] The thickness of the 3D vascular model may be automatically adjusted based on an angiographic fluoroscopic image. In other words, the 3D vascular model may be generated based on an angiographic fluoroscopic image. The thickness of the 3D vascular model varies depending on the threshold for blood vessel detection (noise reduction) performed when generating the 3D vascular model. Since the blood vessel detection threshold is usually set by a surgeon or technician, the thickness of the 3D vascular model does not necessarily match the thickness of the actual blood vessel.

[0066] FIG. 12 is a diagram illustrating a method for adjusting the thickness of a 3D blood vessel model based on an angiographic fluoroscopic image.

[0067] As shown in Figure 12, for example, the information processing device 11 adjusts the threshold value so that the thickness T1 of the blood vessel shown near the center of the angiographic fluoroscopic image Pi21 matches the thickness T1' of the blood vessel in the 3D vascular model Mod1, and so that the thickness T2 of the blood vessel shown in the lower right-center of the angiographic fluoroscopic image Pi21 matches the thickness T2' of the blood vessel in the 3D vascular model Mod1.

[0068] Since the thickness of blood vessels shown in the angiographic fluoroscopic image Pi21 is accurate to a certain extent, by dynamically setting the threshold value based on the thickness of blood vessels shown in the angiographic fluoroscopic image, it is possible to obtain a 3D vascular model that accurately represents the thickness of blood vessels.

[0069] 2. Configuration and Operation of Information Processing Apparatus Configuration of Information Processing Apparatus FIG. 13 is a block diagram showing an example of the configuration of the information processing apparatus 11. As shown in FIG.

[0070] As shown in FIG. 13, the information processing device 11 includes an image capture unit 51 , an image processing unit 52 , a spatial layout roadmap generation unit 53 , a distributor 54 , and a screen generation unit 55 .

[0071] The image capture unit 51 acquires the X-ray detection results (detection data) from the detection device 22 of the X-ray imaging device 1, and supplies the detection data to the image processing unit 52. The image capture unit 51 supplies the fluoroscopic images and angiographic fluoroscopic images supplied from the image processing unit 52 to the distributor 54.

[0072] The image processing unit 52 generates images by performing predetermined calculations and noise reduction on the detection data supplied from the image capturing unit 51. Images generated based on detection data acquired without contrast agent being injected into the blood vessels are fluoroscopic images, and images generated based on detection data acquired with contrast agent being injected into the blood vessels are angiographic fluoroscopic images. The image processing unit 52 supplies the generated fluoroscopic images and angiographic fluoroscopic images to the image capturing unit 51.

[0073] The image processing unit 52 generates a roadmap based on the fluoroscopic image and the angiographic fluoroscopic image, and supplies the roadmap to the distributor 54 .

[0074] The image processing unit 52 acquires imaging information, which is information relating to imaging by the X-ray imaging device 1, from the X-ray imaging device 1. The imaging information indicates, for example, the imaging angle, image size, and imaging magnification of a fluoroscopic image (angiographic fluoroscopic image).

[0075] The image processing unit 52 acquires DICOM (Digital Imaging and Communications in Medicine) data such as CT images and MRI images taken before or during treatment, and supplies the roadmap, DICOM data, and imaging information to the spatial arrangement roadmap generation unit 53. The image processing unit 52 can also generate a 3D vascular model based on the CT images and MRI images taken before or during treatment. The image processing unit 52 supplies the generated 3D vascular model to the spatial arrangement roadmap generation unit 53.

[0076] The spatial layout roadmap generator 53 generates a 3D vascular catheter model and a 3D vascular model based on the roadmap and DICOM data supplied from the image processor 52. The 3D vascular model may be acquired from another device.

[0077] The spatial layout roadmap generating unit 53 acquires, as tracking camera data, images captured by the tracking camera 31. The spatial layout roadmap generating unit 53 acquires, from the screen generating unit 55, layout information indicating the position where the spatial layout roadmap is to be placed on the screen of the examination room monitor 2.

[0078] The spatial layout roadmap generator 53 generates a spatial layout roadmap based on the 3D blood vessel model, the 3D catheter model, the roadmap, and the imaging information, and supplies the spatial layout roadmap to the distributor 54. The viewpoint of the spatial layout roadmap is set based on the tracking camera data and the layout information.

[0079] The distributor 54 distributes various images supplied from the image capture unit 51, image processing unit 52, and spatial layout roadmap generation unit 53 to the screen generation unit 55 (examination room monitor 2) and the operation room monitor 12B.

[0080] The distributor 54 controls the images to be distributed to the screen generator 55 and the operation room monitor 12B based on operation information indicating the operation content by the engineer, which is supplied from the input unit 13 connected to the information processing device 11, for example.

[0081] The control room monitor 12B displays the image supplied from the distributor 54.

[0082] The input unit 13 is composed of a mouse, a keyboard, and the like.

[0083] The screen generator 55 acquires reference information from, for example, medical equipment arranged in an examination room. The screen generator 55 generates a screen for the examination room monitor 2 by arranging the images and reference information supplied from the distributor 54. The screen generator 55 supplies the generated screen to the examination room monitor 2 and the operation room monitor 12A.

[0084] The operation room monitor 12A displays the same screen as that displayed on the examination room monitor 2.

[0085] FIG. 14 is a block diagram showing a detailed configuration example of the spatial layout roadmap generating unit 53.

[0086] As shown in FIG. 14 , the spatial layout roadmap generator 53 includes a 3D blood vessel model generator 71 , a 3D catheter model generator 72 , a superimposing unit 73 , a line-of-sight detector 74 , and an image generator 75 .

[0087] The 3D vascular model generation unit 71 generates a 3D vascular model based on the DICOM data and supplies the 3D vascular model to the 3D catheter model generation unit 72 and the superimposition unit 73. When a 3D vascular model is supplied from the image processing unit 52 or the like, the 3D vascular model generation unit 71 supplies the 3D vascular model to the 3D catheter model generation unit 72 and the superimposition unit 73.

[0088] The 3D catheter model generation unit 72 adjusts the size of the road map based on the imaging information, and generates a 3D catheter model based on the road map, the imaging information, and the 3D blood vessel model supplied from the 3D blood vessel model generation unit 71. The 3D catheter model generation unit 72 supplies the generated 3D catheter model to the superimposition unit 73.

[0089] The superimposing unit 73 matches the orientation and size of the 3D vascular model supplied from the 3D vascular model generating unit 71 with the 3D catheter model supplied from the 3D catheter model generating unit 72 based on the imaging information. The superimposing unit 73 superimposes (combines) the 3D vascular model and the 3D catheter model whose orientations and sizes have been matched to generate a composite 3D model that is a 3D model including the 3D vascular model and the 3D catheter model. The superimposing unit 73 supplies the generated composite 3D model to the image generating unit 75.

[0090] The line-of-sight detection unit 74 detects the surgeon's viewpoint and line-of-sight direction based on the tracking camera data, and supplies line-of-sight information indicating the surgeon's viewpoint and line-of-sight direction to the image generation unit 75 .

[0091] The image generation unit 75 places the composite 3D model supplied from the superimposition unit 73 at, for example, the origin in the virtual space, and places a roadmap in the virtual space based on the imaging information. The image generation unit 75 determines the positional relationship between the surgeon's viewpoint and the spatial layout roadmap on the screen of the examination room monitor 2 based on the layout information and the line-of-sight information supplied from the line-of-sight detection unit 74. The image generation unit 75 renders the virtual space in which the 3D model and the roadmap are placed, and generates an image of the virtual space as a spatial layout roadmap as seen from a viewpoint corresponding to the positional relationship. Note that the image generation unit 75 can also render a virtual space in which only the composite 3D model is placed, to generate an image of the composite 3D model.

[0092] Operation of Information Processing Device Next, with reference to the flowchart of FIG. 15, a process performed by the information processing device 11 having the above-described configuration will be described.

[0093] In step S1, the information processing device 11 acquires various data such as detection data and imaging information of the X-ray imaging device 1, DICOM data, reference information, tracking camera data, and layout information.

[0094] In step S2, the 3D vascular model generating unit 71 generates a 3D vascular model based on the DICOM data.

[0095] In step S3, the 3D catheter model generation unit 72 performs a 3D catheter model generation process. In the 3D catheter model generation process, a 3D catheter model is generated based on a roadmap, etc. Details of the 3D catheter model generation process will be described later with reference to FIG. 16 .

[0096] In step S4, the superimposing unit 73 performs a superimposing process. In the superimposing process, the 3D blood vessel model and the 3D catheter model are superimposed to generate a composite 3D model. The superimposing process will be described in detail later with reference to FIG. 21.

[0097] In step S5, the image generation unit 75 performs a spatial layout roadmap generation process. In the spatial layout roadmap generation process, a spatial layout roadmap is generated based on the composite 3D model and the roadmap. The spatial layout roadmap generation process will be described in detail later with reference to FIG. 22 .

[0098] In step S6, the screen generator 55 arranges the reference information, the spatial layout roadmap, and the like to generate a screen for the examination room monitor 2.

[0099] In step S7, the screen generator 55 displays the generated screen on the examination room monitor 2.

[0100] The 3D catheter model generation process performed in step S3 of FIG. 15 will be described with reference to the flowchart of FIG.

[0101] In step S21, the 3D catheter model generating unit 72 aligns the positions and sizes (angle of view) of two road maps (fluoroscopic images) captured by the X-ray imaging device 1 based on the imaging information.

[0102] FIG. 17 is a diagram for explaining the size adjustment of the roadmap.

[0103] As shown on the left side of Fig. 17, the two roadmaps M31 and M32 captured by the X-ray imaging device 1 may have different image sizes and imaging magnifications. As indicated by the white arrows in Fig. 17, the 3D catheter model generation unit 72 crops the roadmaps M31 and M32 or changes the sizes of the roadmaps M31 and M32 so that the image sizes and imaging magnifications of the roadmaps M31 and M32 match.

[0104] In step S22, the 3D catheter model generation unit 72 performs preprocessing on the two roadmaps.

[0105] FIG. 18 is a diagram illustrating the pre-processing.

[0106] As shown on the left side of Fig. 18, the two roadmaps M31 and M32 show not only catheters but also blood vessels and other organs. The 3D catheter model generation unit 72 performs preprocessing on the roadmaps M31 and M32, including line segment detection, threshold processing, line enhancement, noise reduction, and the like.

[0107] The roadmaps M31' and M32' after preprocessing are images in which, for example, the pixel values ​​of pixels estimated to contain a catheter (catheter pixels) are greater than 0, and the pixel values ​​of all other pixels are 0, as shown at the tips of the white arrows in Figure 18.

[0108] In step S23, the 3D catheter model generation unit 72 calculates normals from the catheter pixels in each of the two road maps after preprocessing, and calculates the intersection of the normals of the two road maps.

[0109] FIG. 19 is a diagram illustrating a method for calculating the intersection of normals from a catheter pixel.

[0110] As shown on the left side of Figure 19, each of the two pre-processed roadmaps M31' and M32' is aligned (step S1 in Figure 17) and placed at a position in virtual space corresponding to the position of the detection device 22 at the time of imaging the roadmap.

[0111] First, the 3D catheter model generation unit 72 calculates normals from the catheter pixel in the row to be processed for each of the roadmaps M31' and M32'. In the example of Figure 19, the normal from the catheter pixel P1 in the row to be processed is calculated for the roadmap M31', and the normal from the catheter pixel P2 in the row to be processed is calculated for the roadmap M32'.

[0112] Next, the 3D catheter model generation unit 72 calculates the intersection P11 of the normal from the catheter pixel P1 and the normal from the catheter pixel P2, and generates an intersection image Pi31 that shows the position of the intersection P11 on a plane including the catheter pixels P1, P2, and the intersection P11.

[0113] The 3D catheter model generating unit 72 generates an intersection image for each row of the roadmaps M31' and M32', as indicated by the white arrows in FIG.

[0114] During or before generating the intersection image, processing may be performed to emphasize pixels that are likely to show a catheter, such as by calculating only normals for catheter pixels whose pixel values ​​are equal to or greater than a threshold, or by emphasizing (increasing pixel values) catheter pixels that are adjacent to other catheter pixels on the preprocessed roadmap.

[0115] The 3D catheter model generating unit 72 constructs a 3D catheter model using a point group of the intersection points included in each intersection image.

[0116] In step S24, the 3D catheter model generation unit 72 corrects the 3D catheter model by performing a filtering process using a 3D blood vessel model on the 3D catheter model formed from the point group of intersections.

[0117] FIG. 20 is a diagram illustrating the filtering process using a 3D blood vessel model.

[0118] The shape of the 3D catheter model Mod31 shown on the left side of FIG. 20 and configured as a point cloud of intersections of normals from catheter pixels may include parts that are not actually catheters.

[0119] Therefore, the 3D catheter model generation unit 72 superimposes the 3D catheter model Mod31 and the 3D blood vessel model Mod32, and constructs the final 3D catheter model Mod31' using only the intersection points between the 3D catheter model Mod31 and the 3D blood vessel model Mod32. Note that the 3D catheter model may be interpolated based on the 3D blood vessel model.

[0120] For example, if a catheter overlaps on the roadmap, it is difficult to accurately reflect the 3D shape of the actual catheter in the 3D catheter model simply by calculating the intersection point of normals from the catheter pixel.Even in such cases, filtering using a 3D vascular model makes it possible to accurately reflect the 3D shape of the actual catheter in the 3D catheter model.

[0121] After the filtering process is performed in step S24, the process returns to step S3 in FIG. 15, and the subsequent processes are performed.

[0122] The superimposition process performed in step S4 of FIG. 15 will be described with reference to the flowchart of FIG.

[0123] In step S41, the superimposing unit 73 adjusts the size and orientation of the 3D catheter model and the 3D blood vessel model based on the imaging information.

[0124] In step S42, the superimposing unit 73 superimposes the 3D catheter model and the 3D blood vessel model to generate a composite 3D model.

[0125] Thereafter, the process returns to step S4 in FIG. 14, and the subsequent processes are carried out.

[0126] The spatial layout roadmap generation process performed in step S5 of FIG. 15 will be described with reference to the flowchart of FIG.

[0127] In step S61, the image generation unit 75 adjusts the sizes of the roadmap and the composite 3D model based on the imaging information.

[0128] In step S62, the image generation unit 75 places the roadmap and the composite 3D model in the virtual space based on the imaging information.

[0129] In step S63, the image generator 75 generates a spatial layout roadmap by rendering the virtual space based on the surgeon's viewpoint (and line of sight direction).

[0130] As described above, in the IVR treatment system of the present technology, a screen including a composite 3D model is displayed on the examination room monitor 2, which is a combination of a 3D vascular model, which is a 3D model showing the three-dimensional shape of a patient's blood vessels, and a 3D vascular model, which is a 3D model showing the three-dimensional shape of a catheter inserted into the blood vessels and is a 3D model generated based on two roadmaps (fluoroscopic images) captured simultaneously at different imaging angles by the X-ray imaging device 1.

[0131] The composite 3D model is included on the screen of the examination room monitor 2 in the form of, for example, a spatial layout roadmap. As described above, the spatial layout roadmap is an image viewed from a virtual viewpoint of the composite 3D model placed in virtual space and the roadmap placed directly facing the composite 3D model at a position in virtual space where the angle relative to the composite 3D model matches the imaging angle of the roadmap itself. The surgeon can grasp the three-dimensional structure of the blood vessels using the composite 3D model included in the spatial layout roadmap, while checking the position and movement of the catheter in detail using the roadmap.

[0132] For surgeons, it is easier to intuitively grasp the three-dimensional positional relationship between blood vessels and catheters, which is expected to shorten the time it takes to deliver the catheter to the target site. It is also expected to improve the safety of IVR treatment and shorten the time it takes for surgeons to acquire spatial awareness. Furthermore, it will reduce the mental stress experienced by surgeons due to difficulty in understanding the three-dimensional structure of blood vessels.

[0133] Reducing the time it takes to deliver the catheter to the target site reduces the time required for interventional radiology treatment, thereby reducing the amount of X-ray radiation the patient receives.

[0134] <3. Others> The composite 3D model and the spatial layout roadmap may not be displayed on the examination room monitor 2, but may be displayed on a sub-monitor that is separate from the examination room monitor 2 and placed in the examination room.

[0135] FIG. 23 is a diagram showing an example of a sub-monitor.

[0136] As shown in A of Fig. 23, the sub-monitor may be configured as a naked-eye 3D display 101 (spatial reproduction display) that can present a 3D image consisting of a left-eye image and a right-eye image having parallax to the surgeon without using dedicated eyewear. Alternatively, as shown in B of Fig. 23, the sub-monitor may be configured as a 3D monitor 102 that can present a 3D image using dedicated eyewear (polarized glasses). Furthermore, as shown in C of Fig. 23, the sub-monitor may be configured as a 2D monitor 103.

[0137] Regarding the computer, the above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware or a general-purpose personal computer.

[0138] FIG. 24 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0139] A CPU (Central Processing Unit) 501 , a ROM (Read Only Memory) 502 , and a RAM (Random Access Memory) 503 are interconnected by a bus 504 .

[0140] An input / output interface 505 is also connected to the bus 504. An input unit 506 including a keyboard, a mouse, etc., and an output unit 507 including a display, a speaker, etc. are connected to the input / output interface 505. Also connected to the input / output interface 505 are a storage unit 508 including a hard disk, a nonvolatile memory, etc., a communication unit 509 including a network interface, etc., and a drive 510 that drives removable media 511.

[0141] In a computer configured as described above, the CPU 501 performs the above-described series of processes by, for example, loading a program stored in the storage unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executing it.

[0142] The program executed by the CPU 501 is installed in the storage unit 508 by being recorded on, for example, a removable medium 511 or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting.

[0143] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0144] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0145] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0146] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0147] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0148] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0149] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0150] Example of configuration combinations The present technology can also be configured as follows.

[0151] (1) An information processing device comprising: a screen generation unit that generates a screen including a composite 3D model obtained by combining a 3D blood vessel model showing the three-dimensional shape of a patient's blood vessel with a 3D catheter model that shows the three-dimensional shape of the catheter inserted into the blood vessel, the 3D model being a 3D model generated based on two fluoroscopic images simultaneously captured by an X-ray imaging device at different imaging angles. (2) The screen generation unit generates the screen including a spatial layout roadmap that is an image viewed from a virtual viewpoint of the composite 3D model placed in a virtual space and at least one fluoroscopic image placed so as to directly face the composite 3D model at a position in the virtual space where an angle with respect to the composite 3D model matches the imaging angle of the fluoroscopic image itself. (3) The information processing device according to (2), wherein the spatial layout roadmap is an image including two fluoroscopic images with different imaging angles. (4) The information processing device according to (3), wherein the two fluoroscopic images with different imaging angles are positioned so as to be at least partially in contact in the virtual space. (5) The information processing device according to (3), wherein the two perspective images having different imaging angles are arranged apart from each other in the virtual space. (6) The information processing device according to any of (3) to (5), wherein the two perspective images having different imaging angles are arranged at two symmetrical positions in the virtual space with the composite 3D model in between. (7) The information processing device according to any of (3) to (5), wherein at least one of the two perspective images having different imaging angles is arranged at a position intersecting the composite 3D model in the virtual space. (8) The information processing device according to any of (2) to (7), wherein the spatial layout roadmap is an image including a line connecting the tip of the 3D catheter model and the tip of the catheter shown in the perspective image. (9) The information processing device according to any of (2) to (8), wherein the screen generation unit generates the screen further including an image showing the interior of the 3D blood vessel model as viewed from the position of the tip of the 3D catheter model.(10) The information processing device according to any one of (2) to (9), wherein the spatial layout roadmap is an image of the composite 3D model and the fluoroscopic images viewed from the virtual viewpoint corresponding to a line of sight direction of a surgeon. (11) The information processing device according to any one of (2) to (9), wherein the spatial layout roadmap is an image of the composite 3D model and the fluoroscopic images viewed from the virtual viewpoint set in accordance with an operation by a surgeon. (12) The information processing device according to any one of (1) to (11), further comprising a 3D catheter model generation unit that calculates normals from pixels where the catheter is estimated to appear in each of two fluoroscopic images having different imaging angles, and constructs the 3D catheter model with a point cloud of intersections of the normals for each of the two fluoroscopic images. (13) The information processing device according to (12), wherein the 3D catheter model generation unit corrects the 3D catheter model constructed from the point clouds based on the 3D vascular model. (14) The information processing device according to any one of (1) to (13), further comprising a 3D vascular model generation unit that generates the 3D vascular model based on an angiographic fluoroscopic image captured by the X-ray imaging device with a contrast agent injected into the blood vessel. (15) An information processing method comprising: generating a screen including a composite 3D model obtained by combining a 3D vascular model showing the three-dimensional shape of a patient's blood vessel with a 3D catheter model showing the three-dimensional shape of a catheter inserted into the blood vessel, the 3D catheter model being a 3D model generated based on two fluoroscopic images captured simultaneously at different imaging angles by an X-ray imaging device. (16) A program for causing a computer to execute processing including: generating a screen including a composite 3D model obtained by combining a 3D vascular model showing the three-dimensional shape of a patient's blood vessel with a 3D catheter model showing the three-dimensional shape of a catheter inserted into the blood vessel, the 3D catheter model being a 3D model generated based on two fluoroscopic images captured simultaneously at different imaging angles by an X-ray imaging device.

[0152] REFERENCE SIGNS LIST 1 X-ray imaging device, 2 Examination room monitor, 11 Information processing device, 12 Operation room monitor, 13 Input unit, 21 X-ray generator, 22 Detection device, 23 Arm unit, 31 Tracking camera, 51 Image capture unit, 52 Image processing unit, 53 Spatial layout roadmap generation unit, 54 Distributor, 55 Screen generation unit, 71 3D blood vessel model generation unit, 72 3D catheter model generation unit, 73 Superposition unit, 74 Gaze detection unit, 75 Image generation unit, 101 Autostereoscopic display, 102 3D monitor, 103 2D monitor

Claims

1. An information processing device equipped with a screen generation unit that generates a screen including a composite 3D model obtained by combining a 3D vascular model showing the three-dimensional shape of a patient's blood vessels with a 3D catheter model that shows the three-dimensional shape of a catheter inserted into the blood vessels, the 3D model being a 3D model generated based on two fluoroscopic images taken simultaneously at different imaging angles by an X-ray imaging device.

2. The information processing device according to claim 1, wherein the screen generation unit generates the screen including a spatial layout roadmap, which is an image of the composite 3D model placed in a virtual space and at least one perspective image placed so as to directly face the composite 3D model at a position in the virtual space where an angle relative to the composite 3D model matches an imaging angle of the perspective image itself, viewed from a virtual viewpoint.

3. The information processing device according to claim 2, wherein the spatial layout roadmap is an image including two perspective images taken at different imaging angles.

4. The information processing device according to claim 3, wherein the two perspective images with different imaging angles are arranged in the virtual space so that at least a portion of the images are in contact with each other.

5. The information processing device according to claim 3, wherein the two perspective images with different imaging angles are positioned apart from each other in the virtual space.

6. The information processing device according to claim 3, wherein the two perspective images with different imaging angles are positioned at two symmetrical positions in the virtual space with the composite 3D model in between.

7. The information processing device according to claim 3, wherein at least one of the two perspective images with different imaging angles is positioned at a position that intersects with the composite 3D model in the virtual space.

8. The information processing device according to claim 2, wherein the spatial layout roadmap is an image including a line connecting the tip of the 3D catheter model and the tip of the catheter shown in the fluoroscopic image.

9. The information processing device according to claim 2, wherein the screen generator generates the screen further including an image showing the inside of the 3D blood vessel model as viewed from the position of the tip of the 3D catheter model.

10. The information processing device according to claim 2, wherein the spatial layout roadmap is an image of the composite 3D model and the perspective image viewed from the virtual viewpoint corresponding to the line of sight of the surgeon.

11. The information processing device according to claim 2, wherein the spatial layout roadmap is an image of the composite 3D model and the perspective image viewed from the virtual viewpoint set in response to an operation by the surgeon.

12. The information processing device according to claim 1, further comprising a 3D catheter model generation unit that calculates normals from pixels where the catheter is estimated to appear in each of the two perspective images captured at different angles, and constructs the 3D catheter model using a point cloud of intersections of the normals in each of the two perspective images.

13. The information processing device according to claim 12, wherein the 3D catheter model generation unit corrects the 3D catheter model configured from the point cloud based on the 3D blood vessel model.

14. The information processing device according to claim 1, further comprising a 3D vascular model generation unit that generates the 3D vascular model based on an angiographic fluoroscopic image captured by the X-ray imaging device with a contrast agent injected into the blood vessels.

15. An information processing method including generating a screen including a composite 3D model obtained by combining a 3D vascular model showing the three-dimensional shape of a patient's blood vessels with a 3D catheter model showing the three-dimensional shape of a catheter inserted into the blood vessels, the 3D model being generated based on two fluoroscopic images taken simultaneously at different imaging angles by an X-ray imaging device.

16. A program for causing a computer to execute a process including generating a screen including a composite 3D model obtained by combining a 3D blood vessel model showing the three-dimensional shape of a patient's blood vessel with a 3D catheter model showing the three-dimensional shape of a catheter inserted into the blood vessel, the 3D model being a 3D model generated based on two fluoroscopic images taken simultaneously at different imaging angles by an X-ray imaging device.

Images