Information processing device, information processing method, program, and information processing system

Abstract

The present technology pertains to an information processing device, an information processing method, a program, and an information processing system that make it possible to suitably provide an image to both an operator and a person other than the operator. The information processing device of the present technology is provided with an image generation unit that generates a first display image on the basis of the viewing position of the user of a spatial reproduction display having a display surface. The first display image is an image displayed on the spatial reproduction display, wherein a 2D image is displayed in 2D on the display surface and a 3D model is displayed in 3D. The present technology can be applied, for example, to a surgical system.

Description

Information processing device, information processing method, program, and information processing system

[0001] The present technology relates to an information processing device, an information processing method, a program, and an information processing system, and in particular to an information processing device, an information processing method, a program, and an information processing system that are capable of providing images appropriately to both a surgeon and a person other than the surgeon.

[0002] Interventional radiology (IVR) is a common treatment method that involves performing vascular catheter treatment while viewing real-time angiographic fluoroscopic images taken using techniques such as DSA (Digital Subtraction Angiography).

[0003] In IVR treatment, CG images generated based on pre-taken CT (Computed Tomography) images or MRI (Magnetic Resonance Imaging) images are displayed next to angiographic fluoroscopic images or superimposed on angiographic projection images. This allows the surgeon to know the direction and angle of the blood vessels beyond the current catheter position, and is expected to enable safer and more efficient treatment.

[0004] The displayed CG image is a 2D rendering of a 3D model showing the three-dimensional shape of blood vessels, etc. For example, as described in Patent Document 1, if a 3D model of blood vessels, etc. and an angiographic fluoroscopic image are displayed on a spatial reproduction display capable of 3D display in which the 3D model is visually recognized as a solid, the surgeon can recognize the shape of the blood vessels in a more three-dimensional manner.

[0005] International Publication No. 2023 / 140120

[0006] When 3D models of blood vessels and other objects and angiographic fluoroscopic images are displayed on a spatial reproduction display, the spatial reproduction display normally tracks only the visual position of the surgeon to display the 3D image, so the angiographic fluoroscopic images and other images appear blurry to assistants and nurses around the surgeon.

[0007] The present technology has been made in view of such circumstances, and makes it possible to provide images in a suitable manner to both the surgeon and persons other than the surgeon.

[0008] An information processing device according to a first aspect of the present technology includes an image generation unit that generates a first display image, which is an image in which a 2D image is displayed in 2D on a display surface and a 3D model is displayed in 3D on a spatial reproduction display having the display surface, based on a user's viewing position of the spatial reproduction display.

[0009] An information processing method according to a first aspect of the present technology is an information processing method for generating a first display image, which is an image in which a 2D image is displayed in 2D on a display surface and a 3D model is displayed in 3D on a spatial reproduction display having the display surface, based on a user's viewing position of the spatial reproduction display.

[0010] A program according to a first aspect of the present technology is a program for causing a computer to execute a process of generating a first display image, which is an image in which a 2D image is displayed in 2D on a display surface and a 3D model is displayed in 3D, and which is displayed on a spatial reproduction display having the display surface, based on a user's viewing position of the spatial reproduction display.

[0011] An information processing system according to a second aspect of the present technology includes an information processing device having an image generation unit that generates a first display image, which is an image in which a 2D image is displayed in 2D on a display surface and a 3D model is displayed in 3D, based on a user's viewing position, and a spatial reproduction display that has the display surface and displays the first display image.

[0012] In a first aspect of the present technology, a first display image is generated based on a user's viewing position of the spatial reproduction display, the first display image being an image in which a 2D image is displayed in 2D on a display surface and a 3D model is displayed in 3D, the first display image being an image displayed on a spatial reproduction display having the display surface.

[0013] In a second aspect of the present technology, an information processing device generates a first display image, which is an image in which a 2D image is displayed in 2D on a display surface and a 3D model is displayed in 3D, based on a user's viewing position, and the first display image is displayed on a spatial reproduction display having the display surface.

[0014] 1 is a diagram illustrating an example of the configuration of a surgery system according to an embodiment of the present technology. FIG. 1 is a diagram illustrating an example of the display surface of a spatial reproduction display. FIG. 2 is a first diagram illustrating an example of the display of a surgical field image. FIG. 3 is a second diagram illustrating an example of the display of a surgical field image. FIG. 4 is a diagram illustrating an example of a display method of an angiographic fluoroscopic image. FIG. 5 is a diagram illustrating another example of the display method of an angiographic fluoroscopic image. FIG. 6 is a diagram illustrating an example of the arrangement of a 3D model and an angiographic fluoroscopic image. FIG. 7 is a diagram illustrating an example of the display of a 3D model and an angiographic fluoroscopic image. FIG. 8 is a flowchart illustrating processing performed by an information processing device. FIG. 9 is a diagram illustrating an example of the resolution of an image displayed on a spatial reproduction display. FIG. 10 is a flowchart illustrating processing performed by an information processing device when displaying a high-resolution 2D image. FIG. 11 is a diagram illustrating 2D images that can be seen by a surgeon and a person other than the surgeon in each display method. FIG. 12 is a diagram illustrating an example of the configuration of a C-arm. FIG. 13 is a diagram illustrating an example of the arrangement of a 3D object in a virtual space. FIG. 14 is a diagram illustrating an example of changing the orientation of a 3D object. FIG. 15 is a diagram illustrating another example of the arrangement of a 3D object in a virtual space. FIG. 16 is a diagram illustrating an example of the actual display of a 3D object. FIG. 17 is a flowchart illustrating processing performed by an information processing device when arranging a 3D object in a positional relationship according to the positional relationship between the C-arm and a patient. It is a block diagram showing an example of the hardware configuration of a computer, and a screen display method for displaying a screen image on a display device, the screen image being displayed on a display device, and a screen displaying a screen image being displayed on a display device.

[0015] Hereinafter, embodiments for carrying out the present technology will be described. The description will be made in the following order: 1. Overview of Surgical System 2. First Embodiment 3. Second Embodiment 4. Third Embodiment

[0016] 1. Overview of Surgery System FIG. 1 is a diagram illustrating an example of the configuration of a surgery system according to an embodiment of the present technology.

[0017] The surgical system in FIG. 1 is composed of a C-arm 1, a display device 2, and an information processing device 3.

[0018] The C-arm 1 is an X-ray imaging device that takes an angiographic fluoroscopic image (X-ray fluoroscopic image) that depicts only the patient's blood vessels (at least a part of the living body) using an X-ray technique such as DSA.

[0019] Display device 2 is a display device, such as a spatial reproduction display, that displays a 3D image consisting of a left-eye image and a right-eye image with parallax between them, allowing the user to view virtual 3D objects placed in a specified space as three-dimensional objects.

[0020] The display device 2 is installed, for example, so that the display surface is perpendicular to a horizontal plane in real space, or so that the display surface faces diagonally upward with respect to the horizontal plane in real space. The display device 2 is provided with, for example, an imaging device 11 as a sensor used to detect the user's viewing position.

[0021] The information processing device 3 is a device that controls the entire surgical system of the present technology, and is, for example, a computer having an information processing circuit including a CPU, memory, etc. For example, the information processing device 3 controls the display by the display device 2, so that an angiographic fluoroscopic image is presented to the user (operator).

[0022] The information processing device 3 includes a capture unit 21 , a UI processing unit 22 , a visual position detection unit 23 , a 3D image generation unit 24 , and an output control unit 25 .

[0023] The capture unit 21 captures the images captured by the C-arm 1 to obtain angiographic fluoroscopic images, and supplies the images to the 3D image generation unit 24. Angiographic fluoroscopic images are 2D images. The capture unit 21 can also obtain operative field images in which at least a portion of a patient's living body is photographed as a subject using an imaging device such as a general camera, stereo camera, surgical microscope, or endoscope. Angiographic fluoroscopic images and operative field images can be considered real-life images, which are images of a subject actually photographed using an imaging device.

[0024] The UI processing unit 22 acquires operation details input by the user using an input device (not shown) or gestures, and supplies the operation details to the 3D image generating unit 24 .

[0025] The viewing position detection unit 23 detects the user's viewing position (position and posture of the viewpoint) based on the captured image captured by the imaging device 11 , and supplies the detection result to the 3D image generation unit 24 .

[0026] The 3D image generation unit 24 acquires DICOM (Digital Imaging and Communications in Medicine) data such as CT images and MRI images taken before surgery or treatment, and generates a 3D model based on the DICOM data that shows the three-dimensional shape of at least a part of the patient's body (e.g., blood vessels) as seen in an angiography fluoroscopic image, for example.

[0027] The 3D image generation unit 24 places the angiographic fluoroscopic image supplied from the capture unit 21 and the generated 3D model in a virtual space. The 3D image generation unit 24 generates a 3D image by rendering the virtual space based on the user's viewing position detected by the viewing position detection unit 23. For example, the 3D image generation unit 24 generates a 3D image so that the angiographic fluoroscopic image and the 3D model can be seen appropriately from the user's viewing position. Here, the placement of the angiographic fluoroscopic image and the 3D model is controlled according to the operation content supplied from the UI processing unit 22.

[0028] The slice data acquired as DICOM data may be arranged in a virtual space as a 3D model.

[0029] The 3D image generating unit 24 supplies the generated 3D image to the output control unit 25 .

[0030] The output control unit 25 controls the display by the display device 2. Specifically, the output control unit 25 supplies the 3D image supplied from the 3D image generation unit 24 to the display device 2, and causes the display device 2 to display the left-eye image and the right-eye image that constitute the 3D image. By delivering the left-eye image to the left eye of the user and the right-eye image to the right eye of the user, for example, an angiographic fluoroscopic image and a 3D model of blood vessels arranged side by side are presented to the user in real time.

[0031] While viewing angiographic fluoroscopic images, users insert catheters and perform endovascular treatments such as aneurysm embolization, stent graft insertion, and angioplasty. By viewing a 3D model of the blood vessels, users can proceed with treatment while understanding the three-dimensional state of the blood vessels beyond the current catheter position.

[0032] Next, a spatial reproduction display applied as the display device 2 of the surgical system will be described with reference to FIGS.

[0033] In recent years, a new type of spatial reproduction display, called a naked-eye stereoscopic image display device, has been proposed, which can display content in three dimensions without the need for special glasses. Such a spatial reproduction display can display images that are horizontally shifted for each viewing angle, allowing the user to perceive depth due to the difference (parallax) between the image for the left eye and the image for the right eye.

[0034] It should be noted that a spatial reproduction display using dedicated glasses may be applied as the display device 2.

[0035] FIG. 2 is a diagram showing an example of a display surface of a spatial reproduction display.

[0036] As described above, the spatial reproduction display is disposed, for example, so that the display surface 12 faces diagonally upward with respect to the horizontal plane in real space. In this case, the spatial reproduction display reproduces a virtual space VW that intersects with the display surface 12, in other words, has areas both in front of and behind the display surface 12 when viewed from the user's viewing position E.

[0037] 3D objects placed in the virtual space VW are displayed so that, for example, some of them appear to protrude beyond the display surface 12. The spatial reproduction display can express virtual depth by displaying 3D images in which different images are delivered to the user's left and right eyes, giving the user the sensation that the 3D objects are present in real space. By positioning the display surface 12 so that it faces diagonally upward relative to the horizontal plane in real space, the spatial reproduction display can easily reproduce the virtual space VW.

[0038] FIG. 3 is a diagram showing an example of a display of an operative field image.

[0039] 3, the surgical field image P1 is projected and displayed on a virtual plate placed in the virtual space VW, for example. The virtual plate onto which the surgical field image P1 is projected is placed so as to always face the user's viewing position.

[0040] Therefore, as shown in the upper part of Fig. 4, when the user views the virtual space VW from the right side while facing the spatial reproduction display (virtual space VW), the virtual plate onto which the surgical field image P1 is projected is positioned so as to face the right side. Also, as shown in the lower part of Fig. 4, when the user views the virtual space VW from the left side while facing the spatial reproduction display, the virtual plate onto which the surgical field image P1 is projected is positioned so as to face the left side.

[0041] The angiographic fluoroscopic image is also projected and displayed on the virtual plate in the same manner as the surgical field image.

[0042] FIG. 5 is a diagram showing an example of a method for displaying an angiographic fluoroscopic image.

[0043] As shown in the speech bubble in Figure 5, an angiographic fluoroscopic image P11L for the left eye and an angiographic fluoroscopic image P11R for the right eye, which have a parallax between them, are displayed on the spatial reproduction display, so that the user sees the angiographic fluoroscopic image P11 projected onto a virtual plate placed in virtual space. A spatial reproduction display typically tracks the viewing position of only one person (e.g., the surgeon) to display a 3D image. Therefore, to assistants and nurses around the surgeon, the angiographic fluoroscopic image P11 appears blurry, as shown in the upper part of Figure 5. Hereinafter, the display of an image for the left eye and an image for the right eye, which have a parallax between them, will also be referred to as 3D display.

[0044] FIG. 6 is a diagram showing another example of a method for displaying an angiographic fluoroscopic image.

[0045] As shown in the balloon in Fig. 6, the same image is displayed on the spatial reproduction display as an angiographic fluoroscopic image P12L for the left eye and an angiographic fluoroscopic image P12R for the right eye, allowing the surgeon, assistant, nurse, etc. to clearly see the angiographic fluoroscopic image P12, as shown in the upper part of Fig. 6. In this case, the surgeon, assistant, nurse, etc. sees the angiographic fluoroscopic image P12 as being displayed on the display surface 12 of the spatial reproduction display. Hereinafter, displaying the same image as the image for the left eye and the image for the right eye will also be referred to as 2D display or display surface display.

[0046] 2. First Embodiment FIG. 7 is a diagram showing an example of the arrangement of a 3D model and an angiographic fluoroscopic image.

[0047] As shown in Figure 7, when the 3D model Mo1 of the blood vessel and the angiographic fluoroscopic image P21 are displayed side by side, in the virtual space VW, the angiographic fluoroscopic image P21 projected onto the virtual plate is placed next to the 3D model Mo1 of the blood vessel as seen from, for example, the viewing position of the angiographic fluoroscopic image P21 (the position of the X-ray generator of the C-arm 1).

[0048] In this technology, the virtual plate onto which the angiographic fluoroscopic image P21 is projected is placed on the display surface 12 of the display device 2. In other words, the angiographic fluoroscopic image P21 is displayed on the display surface 12. In yet other words, the areas in which the angiographic fluoroscopic image P21 is displayed on the left-eye image and the right-eye image displayed on the display device 2 are the same image.

[0049] FIG. 8 is a diagram showing a display example of a 3D model and an angiographic fluoroscopic image.

[0050] On the display device 2, the 3D model Mo1 placed in the virtual space is displayed in 3D based on the viewing position of the surgeon U1, which is tracked using the imaging device 11, as shown by the dashed-dotted arrow in Figure 8. Therefore, as shown by the solid arrow connecting the surgeon U1 and the 3D model Mo1, the surgeon U1 can clearly and appropriately view the 3D model Mo1 as a three-dimensional object. On the other hand, as shown by the dotted arrow connecting the assistant U2 and the 3D model Mo1, the 3D model Mo1 appears blurry or distorted to the assistant U2.

[0051] Furthermore, in the display device 2, an angiographic fluoroscopic image P21 is displayed on the display surface 12. Therefore, as shown by the solid arrow connecting the surgeon U1 and the angiographic fluoroscopic image P21, the surgeon U1 can clearly view the angiographic fluoroscopic image P21. On the other hand, as shown by the solid arrow connecting the assistant U2 and the angiographic fluoroscopic image P21, the assistant U2 can also view the angiographic fluoroscopic image P21.

[0052] In the display device 2, the area (image) displayed on the display surface 12 is highlighted, as shown by the dashed line surrounding the angiographic fluoroscopic image P21 in Fig. 8, making it clear that the 2D image is clearly visible even from a viewing position that is not being tracked. This allows people other than the surgeon to easily find an image that is clearly visible to them and focus on the clearly visible image while ignoring the blurred 3D model Mo1.

[0053] Next, the processing performed by the information processing device 3 having the above configuration will be described with reference to the flowchart in Fig. 9. Here, acquisition of angiographic fluoroscopic images and 3D models of blood vessels, acquisition of operation details, and detection of the user's viewing position are performed as appropriate.

[0054] In step S1, the information processing device 3 determines whether to display information that is easy to view for persons other than the surgeon. Whether to display information that is easy to view for persons other than the surgeon is determined based on, for example, the operation content by the surgeon. In addition, it is also possible to determine whether or not there are people around the surgeon based on the captured image captured by the imaging device 11, and if there are people around the surgeon, to determine that information should be displayed that is easy to view for persons other than the surgeon.

[0055] If it is determined in step S1 that the display is to be easy to view for persons other than the surgeon, the 3D image generator 24 arranges the angiographic fluoroscopic image on the display surface 12 of the display device 2 in step S2.

[0056] In step S3, the information processing device 3 displays the angiographic fluoroscopic image on the display surface. Specifically, the 3D image generation unit 24 generates a left-eye image and a right-eye image for displaying the angiographic fluoroscopic image on the display surface 12, and the output control unit 25 displays the left-eye image and the right-eye image on the display device 2.

[0057] In step S4, the 3D image generating unit 24 highlights the angiographic fluoroscopic image (the area where the display surface display is performed).

[0058] On the other hand, if it is determined in step S1 that the display should not be easy to view for persons other than the surgeon, the 3D image generator 24 displays the angiographic fluoroscopic image on a virtual plate in step S5. Specifically, the angiographic fluoroscopic image is projected onto a virtual plate placed in virtual space so as to directly face the viewing position of the surgeon, and a left-eye image and a right-eye image are generated for displaying this virtual plate on the display device 2. The output controller 25 then causes the display device 2 to display the left-eye image and the right-eye image.

[0059] After the display surface is displayed in step S3 or the virtual plate is displayed in step S5, in step S6, the 3D image generation unit 24 places a 3D model of the blood vessel next to the angiographic projection image (virtual plate) in the virtual space.

[0060] In step S7, the information processing device 3 displays the 3D model in 3D on the display device 2. Specifically, the 3D image generation unit 24 generates a left-eye image and a right-eye image that allow the 3D model to be seen appropriately from the user's viewing position, and the output control unit 25 displays the left-eye image and the right-eye image on the display device 2.

[0061] As described above, in the information processing device of the present technology, a display image (first display image) is generated based on the user's viewing position, in which a 2D image such as an angiographic fluoroscopic image is displayed in 2D on the display surface 12 and a 3D model of blood vessels or the like is displayed in 3D, and the display image is an image displayed on the display device 2. By viewing the angiographic fluoroscopic image displayed on the display device 2 and also the 3D model of the blood vessels, the surgeon can intuitively understand the direction and angle of the blood vessels beyond the current catheter position, enabling them to perform precise treatment efficiently.

[0062] Because precise treatment can be performed efficiently, the time required for surgery and treatment is shortened, reducing the burden on patients. It is also expected to have effects such as efficient use of operating rooms and reduced time and labor required by surgeons. When performing interventional radiology treatment, the patient's exposure time to X-rays can be shortened, reducing the burden on patients.

[0063] Furthermore, persons other than the surgeon can view the angiographic fluoroscopic image that the surgeon is viewing, and can assist the surgeon while viewing the angiographic fluoroscopic image.

[0064] 3. Second Embodiment FIG. 10 is a diagram showing an example of the resolution of an image displayed on a spatial reproduction display.

[0065] When attempting to display a 2D image with a resolution of, for example, 3840 × 2160 on the display surface of a spatial reproduction display, the original 2D image is typically down-converted to a resolution of 1920 × 2160, for example, by averaging pixel values ​​every two pixels in the horizontal direction, as shown by arrow #1 in Figure 10. Therefore, the horizontal resolution of the displayed 2D image is half or less of the resolution of the original 2D image.

[0066] On the other hand, in this technology, when a 2D image with a resolution of, for example, 3840 × 2160 is displayed on the display surface of the spatial reproduction display, the original 2D image is cut out column by column and distributed into left-eye and right-eye images, thereby generating left-eye and right-eye images, as shown by arrow #2 in Fig. 10. By displaying the left-eye and right-eye images with a resolution of 2160 × 1920 generated in this way, a 2D image with the same resolution as the original 2D image, 3840 × 2160, can be viewed from the viewing position of the tracked surgeon.

[0067] As described above, by generating images for the left eye and the right eye by cutting out each column of the original 2D image and distributing them into images for the left eye and the right eye, it is possible to increase the resolution of the displayed 2D image.

[0068] Next, the processing performed by the information processing device 3 when displaying a high-resolution 2D image will be described with reference to the flowchart in Fig. 11. Here, acquisition of an angiographic fluoroscopic image and a 3D model of blood vessels, acquisition of operation details, and detection of the user's viewing position are performed as appropriate. Note that, for simplicity of explanation, Fig. 11 shows only the processing flow related to displaying an angiographic fluoroscopic image.

[0069] In step S21, the information processing device 3 determines whether or not to perform high-resolution display of an angiographic fluoroscopic image. Whether or not to perform high-resolution display of a 2D image is determined based on, for example, the operation content by the operator.

[0070] If it is determined in step S21 that a high-resolution display of a 2D image is to be performed, in step S22, the 3D image generation unit 24 generates an image for the left eye and an image for the right eye by cutting out each column of the angiographic fluoroscopic image and distributing it between an image for the left eye and an image for the right eye.

[0071] In step S23, the information processing device 3 performs high-resolution display of the angiographic fluoroscopic image. Specifically, the output control unit 25 causes the display device 2 to display the left-eye image and the right-eye image generated in step S22.

[0072] In step S24, the information processing device 3 determines whether or not to provide a display that is easy to view for people other than the surgeon.

[0073] If it is determined in step S24 that the display should not be easy to view for persons other than the surgeon, in step S25 the information processing device 3 causes the display device 2 to display the angiographic fluoroscopic image at high resolution on a virtual plate. Specifically, the 3D image generation unit 24 generates a left-eye image and a right-eye image for displaying the angiographic fluoroscopic image at high resolution on a virtual plate on the display device 2, and the output control unit 25 causes the display device 2 to display the left-eye image and the right-eye image.

[0074] On the other hand, if it is determined in step S24 that a display that is easy to view for persons other than the surgeon is to be performed, in step S26 the information processing device 3 causes the display device 2 to display the angiographic fluoroscopic image at high resolution on the display surface. Specifically, the 3D image generation unit 24 generates a left-eye image and a right-eye image for displaying the angiographic fluoroscopic image at high resolution on the display surface of the display device 2, and the output control unit 25 causes the display device 2 to display the left-eye image and the right-eye image.

[0075] If it is determined in step S21 that high resolution display is not to be performed, in step S27, the 3D image generation unit 24 averages the pixel values ​​of the angiographic fluoroscopic image every two pixels in the horizontal direction to generate an image for the left eye and an image for the right eye.

[0076] In step S28, the information processing device 3 determines whether or not to provide a display that is easy to view for people other than the surgeon.

[0077] If it is determined in step S28 that the display should not be made in a manner that is easy for people other than the operator to view, then in step S29, the information processing device 3 causes the display device 2 to display the angiographic fluoroscopic image as a virtual plate.

[0078] On the other hand, if it is determined in step S28 that the display should be easy to view for those other than the operator, then in step S30, the information processing device 3 causes the display device 2 to display the angiographic fluoroscopic image on the display screen at high resolution.

[0079] FIG. 12 is a diagram illustrating 2D images that can be viewed by the surgeon and a person other than the surgeon in each display method.

[0080] When a high-resolution virtual plate is displayed, the surgeon sees a high-resolution 2D image positioned directly opposite the surgeon's viewing position, as shown in Figure 12. In this case, when viewed from a viewing position other than the surgeon's, crosstalk occurs in which the left-eye image and the right-eye image are mixed together in one eye, and therefore, people other than the surgeon, such as assistants and nurses, cannot see the 2D image.

[0081] When a high-resolution display is used, the surgeon sees a high-resolution 2D image displayed on the display surface, while anyone other than the surgeon sees a 2D image with crosstalk occurring on each column.

[0082] When a normal virtual plate is displayed, the surgeon sees a 2D image with normal resolution positioned directly opposite his or her viewing position. In this case, crosstalk occurs when viewed from a viewing position other than the surgeon, so no one other than the surgeon can see the 2D image.

[0083] When normal display screen display is performed, the surgeon sees a 2D image with normal resolution displayed on the display screen, and people other than the surgeon can clearly see the 2D image.

[0084] 4. Third Embodiment FIG. 13 is a diagram showing an example of the configuration of a C-arm 1. As shown in FIG.

[0085] 13, the C-arm 1 is configured, for example, by attaching an X-ray generator 51 and a detector 52 to an arm section 53 so that they face each other. With a patient positioned between the X-ray generator 51 and the detector 52, X-rays are irradiated from the X-ray generator 51, and shadows of the patient's blood vessels and the like are projected onto the detector 52, thereby capturing an angiographic fluoroscopic image.

[0086] FIG. 14 is a diagram showing an example of the arrangement of 3D objects in a virtual space.

[0087] In the example of Figure 14, a 3D model Mo11 representing the X-ray generator 51, a 3D model Mo12 of blood vessels, and an angiographic fluoroscopic image P51 (virtual plate) are arranged in a virtual space so as to reproduce the positional relationship between the X-ray generator 51, the patient, and the detection device 52 shown in Figure 13.

[0088] By displaying a 3D image showing a virtual space in which a 3D model Mo11 of an X-ray generator, a 3D model Mo12 of blood vessels, and an angiographic fluoroscopic image P51 are arranged, the surgeon can intuitively refer to the part of the 3D model Mo12 of blood vessels that corresponds to the part of interest in the angiographic fluoroscopic image P51.

[0089] The operator can change the orientation of the 3D object placed in the virtual space by operating an input device such as a mouse or keyboard. For example, as shown in the upper, middle, and lower rows of Figure 15, the entire virtual space is displayed rotating around the 3D model Mo11 of the X-ray generator according to the user's operation.

[0090] FIG. 16 is a diagram showing another example of the arrangement of 3D objects in the virtual space.

[0091] 16, only the 3D model Mo12 of the blood vessel and the angiographic fluoroscopic image P51 may be arranged in the virtual space. In the virtual space, the 3D model Mo12 of the blood vessel and the angiographic fluoroscopic image P51 are arranged so as to reproduce the positional relationship between the patient and the detection device 52.

[0092] In this case, as shown in the upper, middle, and lower rows of FIG. 16, the entire virtual space is displayed rotating around the 3D model Mo12 of the blood vessels in accordance with the user's operation.

[0093] FIG. 17 is a diagram showing an example of an actual display of a 3D object.

[0094] 17, a 3D model Mo12 of blood vessels and an angiographic fluoroscopic image P61 are displayed on the display device 2, positioned in virtual space in a positional relationship that corresponds to the positional relationship between the patient and the detection device 52. Note that the position of the 3D objects is fixed, and the surgeon can view the 3D model Mo22 of blood vessels from various directions by changing the viewing position.

[0095] Next, the processing performed by the information processing device 3 when arranging a 3D object in a positional relationship corresponding to the positional relationship between the C-arm 1 and the patient will be described with reference to the flowchart in Fig. 18. Here, acquisition of an angiographic fluoroscopic image and a 3D model of blood vessels, acquisition of operation details, and detection of the user's viewing position are performed as appropriate.

[0096] In step S41, the 3D image generating unit 24 projects an angiographic fluoroscopic image onto a virtual plate placed in a virtual space.

[0097] In step S42, the 3D image generator 24 places the 3D model of the X-ray generator and the 3D model of the blood vessels, for example, on the front side of the virtual plate. Here, the 3D model of the X-ray generator, the 3D model of the blood vessels, and the angiographic fluoroscopic image are placed so as to reproduce the positional relationship between the X-ray generator 51, the patient, and the detection device 52.

[0098] In step S43, the information processing device 3 determines whether or not to fix the arrangement of the 3D object. Whether or not to fix the arrangement of the 3D object is set in advance by, for example, the surgeon.

[0099] If it is determined in step S43 that the arrangement of the 3D objects is to be fixed, in step S44 the information processing device 3 causes the display device 2 to reproduce the virtual space as seen from the viewing position of the surgeon with the arrangement of the 3D objects fixed. Specifically, the 3D image generation unit 24 generates a left-eye image and a right-eye image that show the virtual space as seen from the viewing position of the tracking surgeon with the arrangement of the 3D objects fixed, and the output control unit 25 causes the display device 2 to display the left-eye image and the right-eye image.

[0100] If it is determined in step S43 that the position of the 3D object is not to be fixed, in step S45, the information processing device 3 rotates the angiographic fluoroscopic image and the 3D model of the blood vessels around the 3D model of the X-ray generator in accordance with the user's operation. The information processing device 3 also causes the display device 2 to reproduce the virtual space as seen from the operator's viewing position. Specifically, the 3D image generation unit 24 generates left-eye and right-eye images that show the virtual space as seen from the tracking operator's viewing position while rotating the 3D object or virtual space in accordance with the user's operation, and the output control unit 25 causes the display device 2 to display the left-eye and right-eye images.

[0101] As described above, in the information processing device of the present technology, a display image (second display image) is an image displayed on the display device 2, which is a 3D display of a virtual space in which a virtual plate onto which a 2D image such as an angiographic fluoroscopic image is projected and a 3D model such as a blood vessel are arranged in a predetermined positional relationship, and is generated based on the user's viewing position.

[0102] It should be noted that each of the first, second, and third embodiments can be implemented as an embodiment on its own, and a plurality of embodiments may also be combined.

[0103] FIG. 19 is a diagram showing an example of a screen displayed on the display device 2.

[0104] 19A shows a side-by-side display screen (first display image) in which a 3D model Mo101 of a blood vessel is displayed alongside a reference image P102 and an angiographic fluoroscopic image P103, which are 2D images displayed on the display screen. In the side-by-side display screen, for example, the 3D model Mo101 is displayed on the left side, the reference image P102 is displayed on the upper right side, and the angiographic fluoroscopic image P103 is displayed on the lower right side.

[0105] On the parallel display screen, the reference image P102 and the angiographic fluoroscopic image P103 (areas including them) are highlighted as shown enclosed by dashed lines.

[0106] Furthermore, in the parallel display screen, a monitoring information part Pa1 is displayed in the upper part, which presents information related to time such as an electrocardiogram waveform, heart rate, pulse wave, blood oxygen saturation, arterial pressure, and a timer. The monitoring information part Pa1 is arranged on the display surface and displayed in a highlighted manner, as shown by the dashed line surrounding it.

[0107] 19B shows a superimposed display screen (second display image) in which a 3D model Mo101 of blood vessels and an angiographic fluoroscopic image P103 are arranged and displayed according to the positional relationship between the patient and the detection device 52. In the superimposed display screen as well, a highlighted monitoring information part Pa1 is displayed in the upper portion, arranged on the display surface.

[0108] For example, the output control unit 25 of the information processing device 3 can cause the screen displayed on the display device 2 to transition between a parallel display screen and a superimposed display screen. For example, the screen displayed on the display device 2 can seamlessly transition from the parallel display screen to the superimposed display screen.

[0109] 20 and 21 are diagrams for explaining the flow of transition from a parallel display screen to an overlaid display screen.

[0110] When the parallel display screen shown in the upper part of Figure 20 is displayed, for example, if the surgeon instructs a screen transition, first, as shown in the lower part of Figure 20, the reference image P102 disappears and the angiography fluoroscopic image P103 moves to the upper right side of the parallel display screen.

[0111] Next, as shown in the upper part of Fig. 21, the highlighting of the angiographic fluoroscopic image P103 is ended, and the 3D model Mo101 and the angiographic fluoroscopic image P103 move within the virtual space. Finally, as shown in the lower part of Fig. 21, a superimposed display screen is displayed.

[0112] As described above, by transitioning the screen displayed on the display device 2, the surgeon can, in some situations, show an angiographic fluoroscopic image to a person other than the surgeon and receive assistance from that person, and in other situations, proceed with treatment while viewing 3D objects placed in the virtual space and understanding the direction and angle of the blood vessels.

[0113] <Regarding the Computer> The above-described series of processes can be executed by hardware or software. When the series of processes are executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware, or into a general-purpose personal computer, etc.

[0114] FIG. 22 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0115] A CPU (Central Processing Unit) 501 , a ROM (Read Only Memory) 502 , and a RAM (Random Access Memory) 503 are interconnected by a bus 504 .

[0116] An input / output interface 505 is also connected to the bus 504. An input unit 506 including a keyboard, a mouse, etc., and an output unit 507 including a display, a speaker, etc. are connected to the input / output interface 505. Also connected to the input / output interface 505 are a storage unit 508 including a hard disk, a nonvolatile memory, etc., a communication unit 509 including a network interface, etc., and a drive 510 that drives removable media 511.

[0117] In a computer configured as described above, the CPU 501 performs the above-described series of processes by, for example, loading a program stored in the storage unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executing it.

[0118] The program executed by the CPU 501 is installed in the storage unit 508 by being recorded on, for example, a removable medium 511 or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting.

[0119] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0120] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0121] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0122] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0123] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0124] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0125] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0126] <Examples of Combinations of Configurations> The present technology can also have the following configurations.

[0127] (1) An information processing device comprising: an image generation unit that generates a first display image, which is an image in which a 2D image is displayed in 2D on a display surface and a 3D model is displayed in 3D on a spatial reproduction display having the display surface, based on a user's viewing position of the spatial reproduction display. (2) The information processing device described in (1), wherein an area including the 2D image in the first display image is highlighted. (3) The information processing device described in (1) or (2), wherein the image generation unit generates the first display image by cutting out pixels of the 2D image column by column and distributing them into an image for the left eye and an image for the right eye that constitute the first display image. (4) The information processing device described in (1) or (2), wherein the image generation unit generates the image for the left eye and the image for the right eye that constitute the first display image by averaging pixel values ​​of the 2D image every two pixels in the horizontal direction. (5) The information processing device according to any one of (1) to (4), wherein the image generation unit generates a second display image, which is an image displayed on the spatial reproduction display, based on the user's viewing position, and is an image displaying a 3D representation of a virtual space in which a virtual plate onto which the 2D image is projected and the 3D model are arranged in a predetermined positional relationship. (6) The information processing device according to (5), further comprising a control unit that transitions the image displayed on the spatial reproduction display between the first display image and the second display image. (7) The information processing device according to (5) or (6), wherein the 2D image includes an X-ray fluoroscopic image of at least a part of a patient's living body. (8) The information processing device according to (7), wherein the 3D model represents a shape of at least a part of the living body. (9) The information processing device according to (8), wherein the virtual plate and the 3D model are arranged in the virtual space in a positional relationship corresponding to a positional relationship between a detection device that captures the X-ray fluoroscopic image and the patient. (10) An information processing method that generates a first display image, which is an image in which a 2D image is displayed in 2D on a display surface and a 3D model is displayed in 3D on a spatial reproduction display having the display surface, based on a user's viewing position of the spatial reproduction display.(11) A program for causing a computer to execute a process of generating, based on a user's viewing position of the spatial reproduction display, a first display image, which is an image in which a 2D image is displayed in 2D on a display surface and a 3D model is displayed in 3D, and which is to be displayed on a spatial reproduction display having the display surface. (12) An information processing system comprising: an information processing device including an image generation unit that generates, based on a user's viewing position, a first display image, which is an image in which a 2D image is displayed in 2D on a display surface and a 3D model is displayed in 3D on the display surface, and a spatial reproduction display that has the display surface and displays the first display image. (13) The information processing system according to (12), further comprising a sensor used to detect the user's viewing position.

[0128] DESCRIPTION OF SYMBOLS 1 C-arm, 2 Display device, 3 Information processing device, 11 Imaging device, 12 Display surface, 21 Capture unit, 22 UI processing unit, 23 Viewing position detection unit, 24 3D image generation unit, 25 Output control unit, 51 X-ray generator, 52 Detection device, 53 Arm unit

Claims

1. An information processing device having an image generation unit that generates a first display image, which is an image in which a 2D image is displayed in 2D on a display surface and a 3D model is displayed in 3D on a spatial reproduction display having the display surface, based on a user's viewing position of the spatial reproduction display.

2. The information processing device according to claim 1, wherein an area including the 2D image in the first display image is highlighted.

3. The information processing device according to claim 1, wherein the image generation unit generates the first display image by cutting out pixels of the 2D image column by column and distributing the pixels into a left eye image and a right eye image that constitute the first display image.

4. The information processing device according to claim 1, wherein the image generation unit generates a left-eye image and a right-eye image that constitute the first display image by averaging pixel values ​​of the 2D image every two pixels in the horizontal direction.

5. The information processing device according to claim 1, wherein the image generation unit generates a second display image, which is an image displayed on the spatial reproduction display in 3D, in which a virtual space in which the virtual plate onto which the 2D image is projected and the 3D model are arranged in a predetermined positional relationship, based on the user's viewing position.

6. The information processing device according to claim 5, further comprising a control unit that transitions the image displayed on the spatial reproduction display between the first display image and the second display image.

7. The information processing device according to claim 5, wherein the 2D image includes an X-ray fluoroscopic image of at least a portion of a patient's living body.

8. The information processing device according to claim 7, wherein the 3D model represents a shape of at least a part of the living body.

9. The information processing device according to claim 8, wherein in the virtual space, the virtual plate and the 3D model are arranged in a positional relationship corresponding to a positional relationship between a detection device that captures the X-ray fluoroscopic image and the patient.

10. An information processing method, comprising: generating a first display image, which is an image in which a 2D image is displayed in 2D on a display surface and a 3D model is displayed in 3D on a spatial reproduction display having the display surface, based on a user's viewing position of the spatial reproduction display.

11. A program for causing a computer to execute a process of generating a first display image, which is an image in which a 2D image is displayed in 2D on a display surface and a 3D model is displayed in 3D on a spatial reproduction display having the display surface, based on the user's viewing position of the spatial reproduction display.

12. An information processing system comprising: an information processing device having an image generation unit that generates a first display image, which is an image in which a 2D image is displayed in 2D on a display surface and a 3D model is displayed in 3D, based on a user's viewing position; and a spatial reproduction display that has the display surface and displays the first display image.

13. The information processing system according to claim 12, further comprising a sensor used to detect the user's viewing position.

Images