Information processing device, information processing method, and program

Abstract

The present technology pertains to an information processing device, an information processing method, and a program, which make it possible to provide an image which enables an operator to more safely and efficiently perform surgery and treatment. An information processing device according to the present technology comprises an image generation unit that adjusts a CG image obtained by imaging a 3D model with a virtual camera, on the basis of a first guide indicating a scale in the depth direction in an actually captured image obtained by actually imaging a subject with an imaging device and a second guide indicating a scale in the depth direction in the CG image, and generates a display image for displaying the actually captured image and the CG image on a display device. The present technology can be applied, for example, to a surgical system.

Description

Information processing device, information processing method, and program

[0001] The present technology relates to an information processing device, an information processing method, and a program, and in particular to an information processing device, an information processing method, and a program that enable a surgeon to provide images that allow the surgeon to perform surgery or treatment more safely and efficiently.

[0002] In surgeries using surgical robots, surgical microscopes, endoscopes, etc., and in IVR (Interventional Radiology) treatments using X-rays, etc., 3D (stereoscopic) display of surgical field images makes it easier for surgeons to perceive the structure of the body, allowing for more efficient surgery (see, for example, Patent Document 1).

[0003] Furthermore, it is expected that these surgeries and treatments will be able to be performed more safely and efficiently by displaying CG images generated based on pre-taken CT (Computed Tomography) images or MRI (Magnetic Resonance Imaging) images next to or superimposed on the surgical field image.

[0004] JP 2019-97890 A

[0005] However, if the focal length and angle of view between the surgical field image and the CG image do not match, the sense of depth and the shape of the living body shown in each image may not match, making it difficult to use the CG image as a precise comparison (reference) image for the surgical field image.

[0006] The present technology has been developed in light of these circumstances, and is intended to provide images that enable surgeons to perform surgery and treatment more safely and efficiently.

[0007] An information processing device according to one aspect of the present technology includes an image generation unit that adjusts a CG image based on a first guide indicating the depth scale in a live-action image of a subject actually photographed with a photographing device and a second guide indicating the depth scale in a CG image of a 3D model photographed with a virtual camera, and generates a display image for displaying the live-action image and the CG image on a display device.

[0008] An information processing method according to one aspect of the present technology adjusts a CG image based on a first guide indicating the depth scale in a live-action image of a subject actually photographed with a photographing device, and a second guide indicating the depth scale in a CG image of a 3D model photographed with a virtual camera, and generates a display image for displaying the live-action image and the CG image on a display device.

[0009] A program according to one aspect of the present technology causes a computer to execute a process of adjusting a CG image based on a first guide indicating the scale of the depth direction in a live-action image of a subject actually photographed with a photographing device and a second guide indicating the scale of the depth direction in a CG image of a 3D model photographed with a virtual camera, and generating a display image for displaying the live-action image and the CG image on a display device.

[0010] In one aspect of the present technology, the CG image is adjusted based on a first guide indicating the depth scale in a live-action image of a subject actually photographed with a photographing device, and a second guide indicating the depth scale in a CG image of a 3D model photographed with a virtual camera, and a display image is generated for displaying the live-action image and the CG image on a display device.

[0011] 1 is a diagram illustrating a configuration of a surgery system according to an embodiment of the present technology. FIG. 2 is a block diagram illustrating an example of the functional configuration of an information processing device. FIG. 3 is a diagram illustrating an example of a display surface of a spatial reproduction display. FIG. 4 is a first diagram illustrating an example of a display of a preoperative image. FIG. 5 is a second diagram illustrating an example of a display of a preoperative image. FIG. 6 is a first diagram illustrating an example of a depth guide. FIG. 7 is a second diagram illustrating an example of a depth guide. FIG. 8 is a diagram illustrating an example of an arrangement of a surgical field image and a preoperative image on which a depth guide is superimposed. FIG. 9 is a diagram illustrating an example of display of a preoperative image and a surgical field image. FIG. 10 is a diagram illustrating a modified example of the display surface of a display device. FIG. 11 is a diagram illustrating another example of a depth guide. FIG. 12 is a diagram illustrating an example of how a depth guide appears. FIG. 13 is a flowchart illustrating processing performed by an information processing device. FIG. 14 is a first diagram illustrating the flow of processing performed by an information processing device. FIG. 15 is a second diagram illustrating the flow of processing performed by an information processing device. FIG. 16 is a diagram illustrating an example of how a 3D model and a depth guide appear in preoperative images with different angles of view. FIG. 17 is a diagram illustrating a method of adjusting the size of a surgical field image. FIG. 18 is a diagram illustrating an example of superimposed display of a surgical field image and a preoperative image. FIG. 19 is a diagram illustrating an example of display of a surgical field image and a preoperative image when the angle of view is 30 degrees. 1 is a diagram explaining how the size of each frame constituting the depth guide appears when viewed from a certain viewpoint. FIG. 1 is a first diagram explaining how the size of each frame constituting the depth guide appears when photographed at a certain angle of view. FIG. 2 is a second diagram explaining how the size of each frame constituting the depth guide appears when photographed at a certain angle of view. FIG. 3 is a flowchart explaining processing performed by an information processing device when displaying a 3D surgical field image and a preoperative 3D image. FIG. 4 is a first diagram explaining the flow of processing for displaying a 3D surgical field image and a preoperative 3D image. FIG. 5 is a second diagram explaining the flow of processing for displaying a 3D surgical field image and a preoperative 3D image. FIG. 6 is a diagram explaining an example of superimposed display of a 3D surgical field image and a preoperative 3D image. FIG. 7 is a diagram explaining another example of superimposed display of a 3D surgical field image and a preoperative 3D image. FIG. 8 is a first diagram explaining how the appearance of the depth guide differs depending on the convergence angle. FIG. 9 is a second diagram explaining how the appearance of the depth guide differs depending on the convergence angle. FIG. 10 is a fourth diagram explaining how the appearance of the depth guide differs depending on the convergence angle. FIG. 2 is a block diagram illustrating an example of the hardware configuration of a computer.

[0012] Hereinafter, embodiments of the present technology will be described. The description will be made in the following order: 1. Overview of the surgical system 2. Embodiments of the present technology 3. Modifications

[0013] 1. Overview of Surgery System FIG. 1 is a diagram showing a configuration of a surgery system according to an embodiment of the present technology.

[0014] 1 is composed of a master device 100 and a slave device 500. In the surgery system, a user (surgeon) controls the master device 100 to drive the slave device 500 and perform surgery.

[0015] The master device 100 is an information processing device that has an input function for operating the slave device 500, a control function for the slave device 500, and a presentation function for presenting signals output from the camera and sensor of the slave device 500 to the user.

[0016] The master device 100 is composed of an information processing device 101, an input device 200 (an input device 200R for the right hand and an input device 200L for the left hand) that is held and operated by the user, a support base 300, and a display device 400 that displays images based on signals from the slave device 500.

[0017] In the master unit 100, the information processing device 101 outputs a control signal to the slave unit 500 based on an input from the input device 200, thereby controlling the slave unit 500. In addition, in the master unit 100, the information processing device 101 controls vibration (tactile presentation) by the input device 200 or displays on the display device 400 based on a signal from the slave unit 500, thereby presenting an image of the surgical field and feedback from the slave unit 500 to the user.

[0018] The information processing device 101 is a device that processes information in the master device 100, and is, for example, a computer having an information processing circuit equipped with a CPU, memory, and the like.

[0019] The input device 200 is a device that accepts input from a user, and is a device that can input, for example, a hand movement or a gripping movement of the user.

[0020] The support base 300 is a base on which the user can place both arms or both elbows, and is, for example, an aluminum frame. Note that the support base 300 may be provided with a touch display or the like that allows input to the master device 100 and the slave device 500.

[0021] The display device 400 is a display device, such as a spatial reproduction display, that displays a 3D image consisting of a left-eye image and a right-eye image having a parallax therebetween, allowing a user to visually recognize a virtual 3D object placed in a predetermined space as a three-dimensional object. Note that the display device 400 may also be a display that displays a 2D image.

[0022] The display device 400 is installed, for example, so that the display surface faces diagonally upward with respect to the horizontal plane in real space. The inclination of the display surface may be changeable based on the placement angle of the input device 200. In the example of Fig. 1, the display device 400 is placed behind the input device 200 as seen from the user, but it may also be placed in front of the input device 200.

[0023] The slave device 500 is composed of an information processing device 501, a medical device 600, and an arm 700 that supports the medical device.

[0024] In the slave device 500, the information processing device 501 controls the medical device 600 or the arm 700 based on the signal transmitted from the master device 100, thereby performing control based on the user's input.

[0025] The information processing device 501 is a device that processes information in the slave device 500, and is, for example, a computer having an information processing circuit equipped with a CPU, memory, and the like.

[0026] The medical device 600 is a device that operates based on input from the master device 100, and is, for example, an imaging device such as a stereo camera or an endoscope, or an end effector equipped with an electric scalpel or forceps.

[0027] The slave device 500 preferably includes a plurality of sets of the medical device 600 and the arm 700. For example, the slave device 500 may include a first arm equipped with an imaging device and a second arm equipped with an end effector.

[0028] The end effector preferably includes a detection device that detects contact with an affected area, etc. This allows appropriate feedback to be provided to the user by vibrating the input device or restricting its movement based on the pressure applied to the end effector.

[0029] FIG. 2 is a block diagram showing an example of the functional configuration of the information processing device 101.

[0030] As shown in FIG. 2, the information processing device 101 includes a capture unit 111 , a UI processing unit 112 , a visual position detection unit 113 , a 3D image generation unit 114 , and an output control unit 115 .

[0031] The capture unit 111 captures images captured by the imaging device 611, generates a surgical field image, and supplies the generated image to the 3D image generation unit 114. The surgical field image may be a 3D image or a 2D image. The capture unit 111 can also acquire an angiographic fluoroscopic image captured by an X-ray imaging device as the surgical field image. The imaging device 611 is configured with a general camera, a stereo camera, a surgical microscope, an endoscope, or the like, and is provided in, for example, the slave device 500. The imaging device 611 captures an image of the surgical field of surgery or treatment, using at least a portion of the patient's living body as the subject. The surgical field image can be considered a real-life image, which is an image of the subject actually captured by the imaging device 611.

[0032] The UI processing unit 112 acquires operation contents input by the user using the input device 200 or gestures, and supplies the operation contents to the 3D image generating unit 114 .

[0033] The viewing position detection unit 113 detects the user's viewing position (position and posture of the viewpoint) based on the captured image captured by the imaging device 411 provided on the display device 400, and supplies the detection result to the 3D image generation unit 114.

[0034] The 3D image generation unit 114 acquires information indicating the imaging conditions of the imaging device 611 from the imaging device 611. The imaging conditions of the imaging device 611 include the focal length, angle of view, convergence point, convergence angle, etc. The 3D image generation unit 114 also acquires DICOM (Digital Imaging and Communications in Medicine) data such as CT images and MRI images captured before surgery or treatment, and generates a 3D model indicating, for example, the shape of the patient's living body shown in the surgical field image based on the DICOM data.

[0035] The 3D image generation unit 114 generates a CG image by capturing an image of a 3D model with a virtual camera (volume rendering). The 3D model is captured by the virtual camera under the same conditions as those of the imaging device 611. Hereinafter, the CG image is also referred to as a preoperative image generated based on a CT image or an MRI image captured before surgery or treatment.

[0036] The 3D image generation unit 141 arranges the operative field image provided by the capture unit 111 and the generated preoperative image in a virtual space. The 3D image generation unit 141 generates a 3D image by rendering the virtual space based on the user's viewing position detected by the viewing position detection unit 113. For example, the 3D image generation unit 141 generates a 3D image so that the angiographic fluoroscopic image and the 3D model are appropriately viewed from the user's viewing position. The 3D images (left-eye image and right-eye image) generated by the 3D image generation unit 141 can also be considered display images for displaying the operative field image and the preoperative image arranged in the virtual space on the display device 400. Here, the arrangement of the operative field image and the preoperative image is controlled according to the operation content provided by the UI processing unit 112.

[0037] The 3D image generation unit 114 supplies the generated 3D image to the output control unit 115 .

[0038] The output control unit 115 controls the display by the display device 400. Specifically, the output control unit 115 supplies the 3D image supplied from the 3D image generation unit 114 to the display device 400, and causes the display device 400 to display the left-eye image and the right-eye image that constitute the 3D image. By delivering the left-eye image to the left eye of the user and the right-eye image to the right eye of the user, for example, a preoperative image P1 and a surgical field image P2, which show a 3D model of a living body viewed from the same viewing position as the surgical field image (imaging device 611), are arranged side by side and presented to the user in real time.

[0039] Next, a spatial reproduction display applied as the display device 400 of the surgical system will be described with reference to FIGS.

[0040] In recent years, a new type of spatial reproduction display, called a naked-eye stereoscopic image display device, has been proposed, which can display content in three dimensions without the need for special glasses. Such a spatial reproduction display can display images that are horizontally shifted for each viewing angle, allowing the user to perceive depth due to the difference (parallax) between the image for the left eye and the image for the right eye.

[0041] It should be noted that a spatial reproduction display using dedicated glasses may be applied as the display device 400.

[0042] FIG. 3 is a diagram showing an example of a display surface of a spatial reproduction display.

[0043] As described above, the spatial reproduction display is disposed, for example, so that the display surface 412 faces diagonally upward with respect to the horizontal plane in real space. In this case, the spatial reproduction display reproduces a virtual space VW that intersects with the display surface 412, in other words, has areas both in front of and behind the display surface 412 when viewed from the user's viewing position E.

[0044] A 3D object placed in the virtual space VW is displayed so that, for example, a portion of it appears to protrude from the display surface 412. The spatial reproduction display expresses virtual depth by displaying a 3D image in which different images are delivered to the user's left eye and right eye, for example, and can give the user the sensation that the 3D object is present in real space.

[0045] FIG. 4 is a diagram showing an example of a display of a preoperative image.

[0046] 4, the surgical field image P11 is projected and displayed on a virtual plate placed in the virtual space VW, for example. The virtual plate onto which the surgical field image P11 is projected is placed so as to always face the user's viewing position.

[0047] Therefore, as shown in the upper part of Fig. 5, when the user views the virtual space VW from the right side while facing the spatial reproduction display (virtual space VW), the virtual plate onto which the surgical field image P11 is projected is positioned so as to face the right side. Also, as shown in the lower part of Fig. 5, when the user views the virtual space VW from the left side while facing the spatial reproduction display, the virtual plate onto which the surgical field image P11 is projected is positioned so as to face the left side.

[0048] 2. Embodiment of the Present Technology When a surgical field image and a preoperative image are simultaneously displayed on the display device 400, if the focal length, angle of view, convergence angle, etc. of the surgical field image and the preoperative image do not match, the sense of depth and the shape of the living body depicted in the surgical field image and the preoperative image may not match. In this case, it is difficult to use the preoperative image as a precise reference image for the surgical field image.

[0049] Therefore, the information processing device 101 of the present technology superimposes (adds) a depth guide on each of the preoperative image and the operative field image, and matches the focal length, angle of view, convergence angle, etc. of the preoperative image and the operative field image based on the depth guide. The depth guide indicates the scale in the depth direction of each of the preoperative image and the operative field image, and serves as a clue for the user's depth perception.

[0050] 6 and 7 are diagrams showing examples of depth guides.

[0051] As shown in FIG. 6A, the depth guide has, for example, a shape in which multiple rectangles are arranged concentrically.

[0052] As shown in B of Figure 6, the depth guide is superimposed on the preoperative image P21 so that at least the outermost rectangle surrounds the preoperative image P21. In other words, the depth guide is superimposed on the preoperative image P21 as the background of the preoperative image P21. Note that the preoperative image P21 and the depth guide may be superimposed by being combined using alpha blending or the like.

[0053] By superimposing the depth guide on the preoperative image P21, the user can easily understand the sense of depth of the 3D model shown in the preoperative image P21.

[0054] 7A, the depth guide is superimposed on the operative field image P22 so that at least the outermost rectangle surrounds the operative field image P22. In other words, the depth guide is superimposed on the operative field image P22 as the background of the operative field image P22. Note that the operative field image P22 and the depth guide may be superimposed by being combined using alpha blending or the like.

[0055] Compared with the operative field image P22 shown in FIG. 7B where no depth guide is superimposed, the operative field image P22 with the depth guide superimposed is an image in which the sense of depth of the subject is easier to understand.

[0056] In the example of Figure 7, a depth guide is superimposed on the surgical field image P22, which is a cut-out area showing the subject, but the depth guide may also be superimposed on the entire captured image (e.g., a rectangular image) captured by the imaging device 611.

[0057] FIG. 8 is a diagram showing an example of the arrangement of a surgical field image and a preoperative image on which a depth guide is superimposed.

[0058] 8, the preoperative image P21 with the superimposed depth guide is projected and displayed, for example, on a virtual plate placed in the virtual space VW. The 3D model and depth guide shown in the preoperative image P21 are placed three-dimensionally in another virtual space, for example, and the 3D model and depth guide DG1 are photographed with a virtual camera to generate the preoperative image P21 with the superimposed depth guide DG1.

[0059] In other words, the depth guide is three-dimensionally arranged in the virtual space, and in the virtual space, the depth guide is configured, for example, by a plurality of rectangles of the same shape arranged at regular intervals in the depth direction as viewed from the viewing position of the virtual camera. The position of the depth guide is determined, for example, based on the angle of view and the convergence angle of the surgical field image.

[0060] 8, the operative field image P22 is projected and displayed on, for example, a virtual plate placed in the virtual space VW, and the depth guide superimposed on the operative field image P22 is three-dimensionally arranged in the virtual space VW. That is, in the virtual space VW, the depth guide is configured, for example, by multiple rectangles of the same shape arranged at regular intervals in the depth direction as viewed from the user's viewing position.

[0061] FIG. 9 is a diagram showing an example of displaying a preoperative image and an operative field image.

[0062] As shown in Figure 9, when the preoperative image P21 and the operative field image P22 are displayed side by side, in the virtual space VW, the preoperative image P21 and the operative field image P22, each with a depth guide superimposed thereon, are projected and displayed on a single virtual plate.

[0063] FIG. 10 is a diagram showing a modified example of the display surface 412 of the display device 400. In FIG.

[0064] As shown in FIG. 10, the display device 400 may be disposed so that the display surface 412 is perpendicular to a horizontal plane in real space, for example.

[0065] FIG. 11 is a diagram showing another example of the depth guide.

[0066] For simplicity, the depth guide G1 is assumed to be composed of two rectangles. In the example of Fig. 11, the depth guide G1 is composed of two rectangles, one on the front side (frame F1) and one on the back side (frame F2), and is composed of a cube with spheres at each vertex. In the virtual space, the surgical field image P31 is placed within the depth guide G1 (cube).

[0067] FIG. 12 is a diagram illustrating an example of how the depth guide appears.

[0068] As shown in A of FIG. 12, it is assumed that the subject Obj1 of the surgical field image P31 is virtually placed within the depth guide G1, and the surgical field image P31 is captured so that the depth guide G1 is captured in the full angle of view.

[0069] For example, an image captured at a viewing position E1 with a field of view of 60 degrees will look like the image shown in FIG. 12B, and an image captured at a viewing position E2 with a field of view of 30 degrees will look like the image shown in FIG. 12C.

[0070] Compared to the image captured with a 60-degree angle of view, in the image captured with a 30-degree angle of view, the frame F2 on the far side appears to be closer to the viewer. In other words, the distance between frames F1 and F2 appears to be narrower. In other words, the size of frame F2 appears to be larger.

[0071] In this way, the appearance of the depth guide G1 differs depending on the angle of view, and the appearance of the subject Obj1 also differs depending on the angle of view.

[0072] Next, the processing performed by the information processing device 101 having the above configuration will be described with reference to the flowchart in Fig. 13. Here, acquisition of an operative field image and a preoperative image, acquisition of operation details, and detection of the user's viewing position are performed as appropriate.

[0073] For ease of explanation, it is assumed that the user faces display surface 412 of display device 400, and the virtual plate is placed on display surface 412. In other words, the left-eye image and the right-eye image are substantially the same image except for the area where the depth guide is displayed.

[0074] In step S1, the 3D image generator 114 arranges an operative field image of a predetermined size in a virtual space. Specifically, as shown in A of Fig. 14, an operative field image P41 with a field angle of 60 degrees, for example, is projected onto a virtual plate P71 arranged in the virtual space.

[0075] In step S2, the 3D image generator 114 superimposes on the surgical field image a depth guide adjusted based on the focal length and angle of view of the imaging device 611. For example, as shown in B of Fig. 14, a depth guide G11 corresponding to a 60-degree angle of view is superimposed on the surgical field image P41.

[0076] In step S3, the 3D image generator 114 places the preoperative image in the virtual space. For example, as shown in Fig. 15C, the preoperative image P42 is placed to the left of the surgical field image P41 as viewed from the user's viewing position. Note that a depth guide G12 is superimposed on the preoperative image P42 in advance.

[0077] In step S4, the 3D image generator 114 adjusts the viewing position, angle of view, and size of the preoperative image so that the depth guide superimposed on the operative field image matches the depth guide superimposed on the preoperative image. For example, as shown in Fig. 15D, the preoperative image P42 and the depth guide G12 are adjusted so that they appear as if a 3D model of a living body were captured by a virtual camera with a 60-degree angle of view from the same viewing position as the imaging device 611.

[0078] FIG. 16 shows examples of how the 3D model and depth guide appear in preoperative images with different angles of view.

[0079] The example in Figure 16 shows how the 3D model of the living body and the depth guide appear in preoperative images when the angle of view is 30 degrees, 40 degrees, 50 degrees, and 60 degrees. The larger the angle of view, the further back the frame on the depth guide appears to be. The appearance of the 3D model of the living body also differs depending on the angle of view.

[0080] FIG. 17 is a diagram for explaining a method for adjusting the size of an operative field image.

[0081] As shown in A of FIG. 17, for example, when an operative field image P41 is captured, a scale Sc1, which is a scale indicating the size of the subject, is also captured together with the subject.

[0082] 17B, ​​when the preoperative image P42 is captured, a scale Sc2, which is a scale indicating the size of the 3D model, is also captured together with the 3D model of the living body. By adjusting the size of the preoperative image P42 so that the scale Sc1 and the scale Sc2 are the same, the size of the subject captured in the operative field image P41 and the size of the 3D model captured in the preoperative image P42 can be made to match.

[0083] 13 , in step S5, the information processing device 101 determines whether to display the operative field image and the preoperative image side by side (horizontally). Whether to display the operative field image and the preoperative image side by side can be switched in response to, for example, a user operation.

[0084] If it is determined in step S5 that the operative field image and the preoperative image are to be displayed side by side, the process proceeds to step S6. In step S6, the 3D image generator 114 generates an image for the left eye and an image for the right eye by rendering the actual image and the preoperative image arranged side by side on the virtual plate, and the output controller 115 displays the actual image and the preoperative image side by side on the display device 400.

[0085] On the other hand, if it is determined in step S5 that the operative field image and the preoperative image should not be displayed side by side, the process proceeds to step S7. In step S7, the 3D image generator 114 generates an image for the left eye and an image for the right eye by rendering an image obtained by combining the actual image and the preoperative image using alpha blending or the like and projecting the image onto the virtual plate, and the output controller 115 displays the actual image and the preoperative image superimposed on the display device 400.

[0086] FIG. 18 is a diagram showing an example of superimposed display of an operative field image and a preoperative image.

[0087] Because the preoperative image is adjusted so that the depth guide superimposed on the operative field image matches the depth guide superimposed on the preoperative image, the size and orientation of the subject and the 3D model match, as shown in A of Fig. 18. In the example of A of Fig. 18, the operative field image and preoperative image are superimposed when the angle of view is 60 degrees.

[0088] By displaying the surgical field image and the preoperative image superimposed on each other, the user can directly compare the subject shown in the surgical field image with the 3D model shown in the preoperative image.

[0089] On the other hand, if the surgical field image and the preoperative image are superimposed without adjusting the preoperative image, as shown in B of Figure 18, the depth guide superimposed on the surgical field image and the depth guide superimposed on the preoperative image will be misaligned, and the size and orientation of the subject and the 3D model will not match.

[0090] FIG. 19 is a diagram showing an example of display of an operative field image and a preoperative image when the angle of view is 30 degrees.

[0091] In the example of Fig. 19A, an operative field image P41 and a preoperative image P42 with a 30-degree angle of view are displayed side by side. In the example of Fig. 19B, an operative field image and a preoperative image with a 30-degree angle of view are displayed superimposed on each other.

[0092] As described above, in the information processing device of the present technology, the preoperative image is adjusted based on the depth guide (first guide) added to the operative field image and the depth guide (second guide) added to the preoperative image, and a display image is generated for displaying the operative field image and the preoperative image on the display device 400. By viewing the display on the display device 400 and comparing the preoperative image, the appearance of which has been optimized based on the angle of view, etc., with the operative field image, the user can correctly recognize the shape and position of the living body included in the operative field.

[0093] Therefore, users can efficiently perform sophisticated surgery and treatment while viewing preoperative images and surgical field images. Because sophisticated surgery and treatment can be performed efficiently, the time required for surgery and treatment is shortened, reducing the burden on patients. In addition, it is expected to have effects such as efficient use of the operating room and reduction in the time and labor required by surgeons.

[0094] When performing interventional radiology surgery using catheters, the user can intuitively perform surgery by comparing preoperative images with angiographic fluoroscopic images, which serve as surgical field images. This reduces the patient's exposure time to X-rays and reduces the burden on the patient.

[0095] FIG. 20 is a diagram illustrating how the size of each frame constituting the depth guide appears when viewed from a certain viewpoint.

[0096] 20, the depth guide is configured by arranging four frames F11 to F14 in a line in the depth direction at a fixed interval D. When shooting is performed from a viewing position E21 such that the frame F11, which is the foremost of the four frames F11 to F14, is photographed to fill the angle of view, frames farther from the viewing position E21 appear smaller in the photographed image.

[0097] If the reduction ratio of frame F11 in the captured image is 1 (based on the size of frame F11 in the captured image), the reduction ratio of frame F12 is expressed as L / (L + 2 D tan(θ / 2)). θ represents the angle of view, and L represents the left-right width. Furthermore, the reduction ratio of frame F13 in the captured image is expressed as L / (L + 4 D tan(θ / 2)), and the reduction ratio of frame F14 is expressed as L / (L + 6 D tan(θ / 2)). The distance x between viewing position E21 and frame F11 does not contribute to the reduction ratio of each frame.

[0098] If θ=30 degrees and D=L, the reduction ratio of the frame F12 in the captured image is 0.65, the reduction ratio of the frame F13 is 0.48, and the reduction ratio of the frame F14 is 0.38.

[0099] 21 and 22 are diagrams for explaining how the size of each frame constituting the depth guide appears when photographing at a certain angle of view.

[0100] As shown in Figure 21, when shooting is performed from viewing position E21 at a field of view α such that frame F12 is captured to the full extent of the field of view, if the reduction ratio of frame F12 appearing in the captured image is 1 (based on the size of frame F12 appearing in the captured image), the reduction ratio of frame F13 is expressed as L / (L + 2·D·tan(α / 2)), and the reduction ratio of frame F14 is expressed as L / (L + 4·D·tan(α / 2)).

[0101] The angle of view α is expressed as 2·arctan(tan(θ / 2) / (1+2·tan(θ / 2))).

[0102] If α=19.5 degrees, then as shown on the left side of FIG. 22, the reduction ratio of the frame F13 shown in the image captured at the angle of view α is 0.74, and the reduction ratio of the frame F14 is 0.59.

[0103] The reduction ratio of frame F13 relative to frame F11 is converted to a reduction ratio relative to frame F12 (0.48 divided by 0.65) to obtain 0.74. The reduction ratio of frame F14 relative to frame F11 is converted to a reduction ratio relative to frame F12 (0.38 divided by 0.65) to obtain 0.59.

[0104] Therefore, the image captured at the angle of view α and the image cropped from the image captured at the angle of view θ so that the frame F12 fills the entire angle of view are the same image.

[0105] Generally, humans see a field of view with a field of view of about 120 degrees, cropping an area with a field of view of about 47 degrees, or concentrating on an even narrower area. The human field of view (field of view) can be likened to an ultra-wide-angle lens with a focal length of about 10-12mm (35mm equivalent), but because only a portion of the field of view is focused on, the field of view that humans actually see is equivalent to a lens with a focal length of 50mm (35mm equivalent).

[0106] 3. Modifications A 3D surgical field image in which a pair of surgical field images having a parallax between them are delivered to the left and right eyes of the user, or a preoperative 3D image in which a pair of preoperative images having a parallax between them are delivered to the left and right eyes of the user, may be displayed. The 3D surgical field image is an image obtained by photographing the surgical field with, for example, a stereo camera as the imaging device 611. Note that a preoperative 3D image and a 2D surgical field image may also be displayed.

[0107] A stereo camera is composed of, for example, a left camera placed on the left side of the subject and a right camera placed on the right side of the subject. For example, an image captured by the left camera is displayed as an image for the left eye, and an image captured by the right camera is displayed as an image for the right eye, so that the surgical field appears three-dimensional to the user.

[0108] The process performed by the information processing device 101 when displaying a 3D operative field image and a preoperative 3D image will be described with reference to the flowchart in Fig. 23. Here, the acquisition of the operative field image and the preoperative image, the acquisition of the operation content, and the detection of the user's viewing position are performed as appropriate.

[0109] For ease of explanation, it is assumed that the user faces the display surface 412 of the display device 400, and the virtual plate is placed on the display surface 412. In other words, the left-eye image and right-eye image are substantially the same except for the area where the surgical field 3D image, the preoperative 3D image, and the depth guide are displayed.

[0110] In step S21, the 3D image generator 114 arranges a 3D image of the surgical field at a predetermined size in the virtual space. Specifically, as shown in A of Fig. 24, the 3D image of the surgical field P101 is projected onto a virtual plate P151 arranged in the virtual space.

[0111] As shown in the balloon in Figure 24A, in the image for the left eye presented to the user, a 2D surgical field image P101L captured by the left camera of the stereo camera is projected onto the virtual plate. In the image for the right eye presented to the user, a 2D surgical field image P101R captured by the right camera of the stereo camera is projected onto the virtual plate. The surgical field images P101L and P101R have a parallax based on the convergence angle of the stereo camera.

[0112] In step S22, the 3D image generator 114 superimposes a depth guide, adjusted based on the focal length and angle of view of the stereo camera, on the 3D surgical field image. For example, as shown in B of Fig. 24, a depth guide G101 that enables stereoscopic viewing is superimposed on the 3D surgical field image P101. The superimposition of the depth guide G101 makes it easier for the user to correctly recognize the sense of depth of the subject displayed in the 3D surgical field image P101.

[0113] As shown in the speech bubble in B of Figure 24, in the left eye image and right eye image presented to the user, a depth guide G101L or depth guide G101R with parallax based on the convergence angle of the stereo camera is superimposed on the surgical field image P101L or surgical field image P101R, respectively.

[0114] In step S23, the 3D image generator 114 places a preoperative 3D image in the virtual space. For example, as shown in A of Fig. 25, a preoperative 3D image P102 is placed to the left of the surgical field 3D image P101 as viewed from the user's viewing position. Note that a depth guide G102 that enables stereoscopic viewing is superimposed on the preoperative 3D image P102 in advance.

[0115] 25A, for example, a 2D preoperative image P102L or a 2D preoperative image P102R having parallax based on the convergence angle of the stereo camera is disposed in each of the left-eye and right-eye images presented to the user. Also, in each of the left-eye and right-eye images presented to the user, a depth guide G102L or a depth guide G102R having parallax based on the convergence angle of the stereo camera is superimposed on the preoperative image P102L or P102R.

[0116] In step S24, the 3D image generation unit 114 adjusts the viewing position, angle of view, and size of the preoperative 3D image so that the depth guide superimposed on the surgical field 3D image matches the depth guide superimposed on the preoperative 3D image.

[0117] Thus, in steps S23 and S24, the parallax between the preoperative image P102L and the preoperative image P102R is adjusted so that the parallax (convergence angle) between the operative field image P101L and the operative field image P101R and the parallax (convergence angle) between the preoperative image P102L and the preoperative image P102R coincide. Furthermore, the preoperative image P102L is adjusted so that the depth guide G101L superimposed on the operative field image P101L coincides with the depth guide G102L superimposed on the preoperative image P102L. Furthermore, the preoperative image P102R is adjusted so that the depth guide G101R superimposed on the operative field image P101R coincides with the depth guide G102R superimposed on the preoperative image P102R.

[0118] Therefore, as shown in A of Figure 25, the perceived depth of the operative field 3D image P101 and the preoperative 3D image P102 match. If the parallax between the operative field image P101L' and the operative field image P101R' and the parallax between the preoperative image P102L' and the preoperative image P102R' do not match, the depth of the preoperative image P102' will appear distorted to the user, as shown by the hatching in B of Figure 25. For example, the depth may appear stretched, compressed, or distorted.

[0119] In step S25, the information processing device 101 determines whether to display the 3D surgical field image and the preoperative 3D image side by side. Whether to display the 3D surgical field image and the preoperative 3D image side by side can be switched in response to, for example, a user operation.

[0120] If it is determined in step S25 that the operative field 3D image and the preoperative 3D image are to be displayed side by side, the process proceeds to step S26. In step S26, the 3D image generator 114 performs rendering with the actual 3D image and the preoperative 3D image arranged side by side to generate an image for the left eye and an image for the right eye, and the output controller 115 causes the display device 400 to display the actual 3D image and the preoperative 3D image side by side.

[0121] On the other hand, if it is determined in step S25 that the operative field 3D image and the preoperative 3D image should not be displayed side by side, the process proceeds to step S27. In step S27, the 3D image generator 114 generates left-eye and right-eye images by rendering the actual 3D image and the preoperative 3D image after combining them using alpha blending or the like, and the output controller 115 superimposes the actual 3D image and the preoperative 3D image and displays them on the display device 400.

[0122] FIG. 26 is a diagram showing an example of a superimposed display of a 3D surgical field image and a preoperative 3D image.

[0123] Because the preoperative images are adjusted so that the parallax between the two surgical field images constituting the surgical field 3D image matches the parallax between the two preoperative images constituting the preoperative 3D image, the depth guides match in both the left-eye image P111L and the right-eye image P111R in which the surgical field image and the preoperative image are superimposed, as shown in the speech bubble in Figure 26A. When such left-eye image P111L and right-eye image P111R are displayed, the size and orientation of the subject and the 3D model appear to match to the user in the 3D image P111.

[0124] By displaying the 3D surgical field image and the preoperative 3D image superimposed on each other, the user can directly compare the subject shown in the 3D surgical field image with the 3D model shown in the preoperative 3D image.

[0125] On the other hand, if the surgical field 3D image and the preoperative 3D image are superimposed without the parallax between the two surgical field images and the parallax between the two preoperative images matching, the depth guides of the surgical field image and the preoperative image will be misaligned in both the left-eye image P111L' and the right-eye image P111R', in which the surgical field image and the preoperative image are superimposed, as shown in the speech bubble in Figure 26B. When such a left-eye image P111L' and right-eye image P111R' are displayed, the depth of the 3D image P111' appears distorted to the user, as indicated by the hatching. For example, the depth may appear stretched, compressed, or distorted.

[0126] FIG. 27 is a diagram showing another example of superimposed display of the operative field 3D image and the preoperative 3D image.

[0127] As shown in Fig. 27, a surgical field 3D image without a superimposed depth guide and a preoperative 3D image may be superimposed and displayed. In this case, as shown in the balloon in Fig. 27, a left-eye image and a right-eye image with the surgical field image without a superimposed depth guide and the preoperative image superimposed thereon are displayed, thereby displaying a 3D image P121 without a superimposed depth guide.

[0128] When a 3D image P121 without a depth guide superimposed thereon is displayed, the parallax of the preoperative image is adjusted so that the parallax between the two surgical field images matches the parallax between the two preoperative images, and the preoperative image is adjusted so that the depth guide superimposed on the surgical field image matches the depth guide superimposed on the preoperative image. After the preoperative image is adjusted based on the depth guide, a left-eye image and a right-eye image with the depth guide removed are generated.

[0129] In this way, when adjusting the preoperative 3D image, a depth guide may be temporarily added to each of the surgical field 3D image and the preoperative 3D image.

[0130] As described above, by comparing the preoperative 3D image, which has been optimized for viewing angle, with the surgical field 3D image, the user can easily and correctly recognize the shape and position of the living body in the surgical field. In addition, the surgical system of this technology displays the surgical field 3D image and the preoperative 3D image with matching convergence angles, allowing the user to more accurately recognize the sense of depth of the subject shown in the surgical field 3D image and the 3D model shown in the preoperative 3D image.

[0131] 28 to 31 are diagrams illustrating differences in how the depth guide appears depending on the convergence angle.

[0132] 28 to 31 , the depth guide is configured by four frames F21 to F24 arranged side by side at regular intervals in the depth direction. Here, frame F22 of the four frames F21 to F24 is virtually arranged on the display surface 412 of the display device 400. It is assumed that a convergence point exists on the display surface 412.

[0133] When viewed from viewing position E51, for example, frame F11 is located in front of display surface 412, so when the image for the left eye and the image for the right eye are compared, the position of frame F11 in the image for the right eye appears to be to the left of the position of frame F11 in the image for the left eye, as shown at the tip of outline arrow #11 in Fig. 28. On the other hand, for example, frame F13 is located behind display surface 412, so when the image for the left eye and the image for the right eye are compared, the position of frame F11 in the image for the left eye appears to be to the left of the position of frame F11 in the image for the right eye, as shown at the tip of outline arrow #12 in Fig. 28.

[0134] 29, in the left-eye image P201L, the center position of frame F11 is shifted to the right with respect to the center position of frame F12, and the center positions of frames F13 and F14 are shifted to the left with respect to the center position of frame F12. Also, as shown in the lower right of Fig. 29, in the right-eye image P201R, the center position of frame F11 is shifted to the left with respect to the center position of frame F12, and the center positions of frames F13 and F14 are shifted to the right with respect to the center position of frame F12.

[0135] By displaying such a left-eye image P201L and a right-eye image P201R, the user can see a 3D image P201 showing a depth guide as seen from a viewing position E51, as shown in the upper part of Figure 29.

[0136] Next, referring to FIG. 30, we will explain how the depth guide appears when viewed from the same viewing position E51 as in the example described with reference to FIG. 28, but with a larger convergence angle than in this example.

[0137] When viewed from viewing position E51, comparing the left-eye image and the right-eye image, the position of frame F11 in the right-eye image appears to be to the left of the position of frame F11 in the left-eye image, as indicated by the tip of outline arrow #21 in Fig. 30. Furthermore, the amount of parallax of frame F11 is greater than the amount of parallax of frame F11 in the example of Fig. 28.

[0138] On the other hand, when comparing the left-eye image and the right-eye image, the position of frame F13 in the left-eye image appears to the left of the position of frame F13 in the right-eye image, as indicated by the tip of the white arrow #22 in Fig. 30. The amount of parallax of frame F13 is also greater than the amount of parallax of frame F13 in the example of Fig. 28.

[0139] Therefore, as shown in the lower left of Fig. 31, in left-eye image P201L, the center position of frame F11 is shifted further to the right with respect to the center position of frame F12 as compared to the example in Fig. 30, and the center positions of frames F13 and F14 are shifted further to the left with respect to the center position of frame F12 as compared to the example in Fig. 30. Also, as shown in the lower right of Fig. 31, in right-eye image P201R, the center position of frame F11 is shifted further to the left with respect to the center position of frame F12 as compared to the example in Fig. 30, and the center positions of frames F13 and F14 are shifted further to the right with respect to the center position of frame F12 as compared to the example in Fig. 30.

[0140] <Regarding the Computer> The above-described series of processes can be executed by hardware or software. When the series of processes are executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware, or into a general-purpose personal computer, etc.

[0141] FIG. 32 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0142] A CPU (Central Processing Unit) 1001 , a ROM (Read Only Memory) 1002 , and a RAM (Random Access Memory) 1003 are interconnected by a bus 1004 .

[0143] An input / output interface 1005 is further connected to the bus 1004. An input unit 1006 including a keyboard, a mouse, etc., and an output unit 1007 including a display, a speaker, etc. are connected to the input / output interface 1005. Also connected to the input / output interface 1005 are a storage unit 1008 including a hard disk, a nonvolatile memory, etc., a communication unit 1009 including a network interface, etc., and a drive 1010 that drives removable media 1011.

[0144] In a computer configured as described above, the CPU 1001 performs the above-described series of processes by, for example, loading a program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it.

[0145] The program executed by the CPU 1001 is installed in the storage unit 1008 by being recorded on, for example, a removable medium 1011 or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting.

[0146] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0147] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0148] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0149] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0150] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0151] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0152] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0153] <Examples of Combinations of Configurations> The present technology can also have the following configurations.

[0154] (1) An information processing device comprising: an image generation unit that adjusts a CG image based on a first guide indicating the scale in the depth direction in a live-action image of a subject actually photographed with an imaging device and a second guide indicating the scale in the depth direction in a CG image of a 3D model photographed with a virtual camera, and generates a display image for displaying the live-action image and the CG image on a display device. (2) The image generation unit adjusts the CG image so that the first guide and the second guide match. (3) The information processing device described in (1) or (2), wherein the live-action image is a surgical field image in which at least a portion of a patient's living body is photographed as the subject. (4) The information processing device described in (3), wherein the CG image is an image of the 3D model showing the shape of at least a portion of the living body photographed with the virtual camera. (5) The information processing device described in any of (1) to (4), wherein the display device is a spatial reproduction display. (6) The information processing device according to (5), wherein the image generation unit projects the real-life image and the CG image onto a virtual plate placed in a virtual space so as to directly face the viewing position of a user of the display device, and generates the display image by rendering the virtual space. (7) The information processing device according to (6), wherein the image generation unit projects the real-life image and the CG image side by side onto the virtual plate. (8) The information processing device according to (6), wherein the image generation unit superimposes the real-life image and the CG image and projects them onto the virtual plate. (9) The information processing device according to any of (1) to (8), wherein the image generation unit adjusts at least one of the focal length, angle of view, and size of the CG image based on the first guide and the second guide. (10) The information processing device according to any of (1) to (8), wherein the CG image is a 3D image in which a pair of images having parallax from each other are delivered to the left eye and right eye, respectively, of the user of the display device. (11) The information processing device according to (10), wherein the real-life images are 3D images in which a pair of images having parallax with each other are delivered to the left eye and the right eye of the user, respectively.(12) The information processing device according to (11), wherein the image generation unit adjusts the convergence angle of the CG image based on the convergence angle of the real-life image. (13) The information processing device according to any of (1) to (12), wherein the image generation unit generates the display image for displaying on the display device the real-life image on which the first guide is superimposed and the CG image on which the second guide is superimposed. (14) The information processing device according to (13), wherein the first guide and the second guide shown in the display image have a shape in which multiple rectangles are arranged concentrically. (15) The information processing device according to any of (1) to (12), wherein the image generation unit generates the display image for displaying on the display device the real-life image and the CG image on which the first guide and the second guide are not superimposed. (16) An information processing method, comprising: adjusting a CG image based on a first guide indicating the scale in the depth direction in a real-life image of a subject actually photographed with a photographing device and a second guide indicating the scale in the depth direction in a CG image of a 3D model photographed with a virtual camera; and generating a display image for displaying the real-life image and the CG image on a display device. (17) A program for causing a computer to execute the process of: adjusting the CG image based on a first guide indicating the scale in the depth direction in a real-life image of a subject actually photographed with a photographing device and a second guide indicating the scale in the depth direction in a CG image of a 3D model photographed with a virtual camera; and generating a display image for displaying the real-life image and the CG image on a display device.

[0155] REFERENCE SIGNS LIST 100 Master device, 101 Information processing device, 111 Capture unit, 112 UI processing unit, 113 Viewing position detection unit, 114 3D image generation unit, 115 Output control unit, 400 Display device, 411 Image capture device, 412 Display surface, 500 Slave device, 611 Image capture device

Claims

1. An information processing device having an image generation unit that adjusts a CG image based on a first guide indicating the scale of the depth direction in a real-life image of a subject actually photographed by a photographing device and a second guide indicating the scale of the depth direction in a CG image of a 3D model photographed by a virtual camera, and generates a display image for displaying the real-life image and the CG image on a display device.

2. The information processing device according to claim 1, wherein the image generating unit adjusts the CG image so that the first guide and the second guide coincide with each other.

3. The information processing device according to claim 1, wherein the actual image is a surgical field image in which at least a portion of a patient's living body is photographed as the subject.

4. The information processing device according to claim 3, wherein the CG image is an image of the 3D model showing the shape of at least a portion of the living body photographed by the virtual camera.

5. The information processing device according to claim 1, wherein the display device is a spatial reproduction display.

6. The information processing device according to claim 5, wherein the image generation unit projects the real-life image and the CG image onto a virtual plate arranged in a virtual space so as to face directly toward the viewing position of a user of the display device, and generates the display image by rendering the virtual space.

7. The information processing device according to claim 6, wherein the image generating section projects the real image and the CG image side by side onto the virtual plate.

8. The information processing device according to claim 6, wherein the image generating section superimposes the real image and the CG image and projects them onto the virtual plate.

9. The information processing device according to claim 1, wherein the image generation unit adjusts at least one of the focal length, the angle of view, and the size of the CG image based on the first guide and the second guide.

10. The information processing device according to claim 1, wherein the CG image is a 3D image in which a pair of images having a parallax with respect to each other are delivered to the left eye and right eye, respectively, of a user of the display device.

11. The information processing device according to claim 10, wherein the real-life images are 3D images in which a pair of images having a mutual parallax are delivered to the left eye and right eye of the user, respectively.

12. The information processing device according to claim 11, wherein the image generating section adjusts the convergence angle of the CG image based on the convergence angle of the real image.

13. The information processing device according to claim 1, wherein the image generation unit generates the display image for displaying on the display device the real-life image with the first guide superimposed thereon and the CG image with the second guide superimposed thereon.

14. The information processing device according to claim 13, wherein the first guide and the second guide shown in the displayed image have a shape in which multiple rectangles are arranged concentrically.

15. The information processing device according to claim 1, wherein the image generation unit generates the display image for displaying the real-life image and the CG image on the display device without the first guide and the second guide being superimposed.

16. An information processing method comprising: adjusting a CG image based on a first guide indicating the scale of the depth direction in a real-life image of a subject actually photographed by a photographing device, and a second guide indicating the scale of the depth direction in a CG image of a 3D model photographed by a virtual camera; and generating a display image for displaying the real-life image and the CG image on a display device.

17. A program for causing a computer to execute a process of adjusting a CG image based on a first guide indicating the scale of the depth direction in a real-life image of a subject actually photographed by a photographing device and a second guide indicating the scale of the depth direction in a CG image of a 3D model photographed by a virtual camera, and generating a display image for displaying the real-life image and the CG image on a display device.

Images