3D annotation system for surgical support

Abstract

To achieve recording and drawing of an annotation corresponding to a three-dimensional video.SOLUTION: A computer includes a reception part for an imaging signal by cameras for the right eye and the left eye, a background video generation part for the right eye and the left eye based on the imaging signal, a depth position information generation part for a pointer, a pointer upper, lower, left and right position information generation part based on an operation signal transmitted from an input device, an annotation start and end information generation part based on the operation signal, an annotation related information storage part for storing depth position information and upper, lower, left and right position information of the pointer between record start information generation and record end information generation, a pointer video generation part for the right eye and the left eye and an annotation related video generation part, and a background annotation video synthesis part for synthesizing a background video for the right eye and the left eye, a pointer video and an annotation related video to generate a final video for the right eye and the left eye.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a system for drawing annotations by handwriting or the like on an image displayed on a monitor. [Background technology]

[0002] Currently, in procedures using surgical support robots in the medical field, a robotic arm equipped with surgical instruments and an endoscope are inserted, and the doctor operates the robotic arm while watching the endoscopic images from inside a control box called a surgical console.

[0003] Doctors need to move surgical instruments such as forceps attached to the tip of a robot arm back and forth, up and down, and left and right in the three-dimensional space inside the patient's body. For this reason, surgical support robots use endoscopes to capture images of the inside of the body as three-dimensional images, and display the images on a three-dimensional monitor inside the operation box, allowing doctors to grasp the internal space in three dimensions (see, for example, Patent Document 1). Generally, three-dimensional monitors in medical settings use a polarized system, and doctors can view the images in three dimensions by wearing three-dimensional polarized glasses. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special table 2009-512514 Summary of the Invention [Problem to be solved by the invention]

[0005] Even if the doctor performing the procedure or those around him or her tried to add annotations (such as area indications of the affected area or lines to indicate the surgical site) to the three-dimensional image using an external input device, the annotations were displayed on a two-dimensional surface (the surface of the display), making it impossible to correctly indicate areas deep inside the body (blood vessels, organs, etc.).

[0006] In view of the above circumstances, the present invention aims to provide a system etc. capable of depicting annotations taking into account depth. [Means for solving the problem]

[0007] The present invention for achieving the above object is a three-dimensional annotation depiction system realized by a computer, the computer including a camera image receiving unit that receives a right-eye image signal and a left-eye image signal of a subject imaged by a right-eye camera and a left-eye camera, a background image generating unit that generates a right-eye background image and a left-eye background image based on the right-eye image signal and the left-eye image signal, a pointer up-down, left-right position information generating unit that generates up-down, left-right position information of the pointer based on an operation signal transmitted from an annotation input device that operates a pointer, a pointer depth position information generating unit that generates depth position information of the pointer, an annotation start / end information generating unit that generates annotation recording start information and recording end information based on the operation signal, and stores the depth position information of the pointer between the generation timing of the recording start information and the generation timing of the recording end information as annotation depth position information, and a pointer image generation unit that generates a pointer image for the right eye and a pointer image for the left eye by referring to at least the up, down, left, and right position information of the pointer between the generation timing of the recording start information and the generation timing of the recording end information, an annotation-related image generation unit that generates an annotation-related image for the right eye and an annotation-related image for the left eye by referring to the depth position information of the annotation and the up, down, left, and right position information of the annotation, and a background annotation image synthesis unit that synthesizes the background image for the right eye and the background image for the left eye, the pointer image for the right eye and the pointer image for the left eye, and the annotation-related image for the right eye and the annotation-related image for the left eye to generate a final image for the right eye and a final image for the left eye.

[0008] In relation to the above three-dimensional annotation drawing system, the computer is further characterized by having a left and right image synthesis unit that generates a three-dimensional final image by superimposing the final image for the right eye and the final image for the left eye.

[0009] In relation to the above three-dimensional annotation depiction system, the pointer depth position information generation unit is characterized in that it generates depth position information of the pointer based on an operation signal indicating depth movement transmitted from the annotation input device.

[0010] In relation to the above-mentioned three-dimensional annotation depiction system, the computer is characterized in that it includes a subject depth position information calculation unit that calculates depth position information of the subject based on the background image for the right eye and the background image for the left eye, and the pointer depth position information generation unit generates depth position information of the pointer based on depth position information of the subject corresponding to up, down, left, and right position information of the pointer.

[0011] In relation to the above three-dimensional annotation depiction system, the pointer image generation unit generates the right eye pointer image and the left eye pointer image including parallax based on depth position information of the pointer. Effect of the Invention

[0012] According to the present invention, it is possible to obtain the excellent effect of realizing recording and depiction of annotations corresponding to three-dimensional images. [Brief description of the drawings]

[0013] [Figure 1] 1 is a block diagram showing the overall configuration of a three-dimensional annotation drawing system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing a general-purpose configuration of a computer used in the three-dimensional annotation drawing system. [Diagram 3]1A is a block diagram showing the functional configuration of an annotation input device used in the three-dimensional annotation drawing system, and FIG. 1B is a front view showing the same functional configuration realized by a mouse-type input device. [Figure 4] 2 is a block diagram showing the functional configuration of an annotation processing device realized by a computer in the three-dimensional annotation drawing system. FIG. [Diagram 5] 2 is a block diagram showing the functional configuration of a 3D image generating device realized by a computer in the 3D annotation drawing system. FIG. [Figure 6] 1A is a schematic diagram showing the subject (surgical field) in the three-dimensional annotation depiction system, as viewed from the imaging axis direction of the camera, and (B) is a schematic diagram showing the subject viewed from above and perpendicular to the imaging axis direction. [Figure 7] Schematic diagrams showing (A) background image for the right eye and background image for the left eye, (B) pointer image for the right eye and pointer image for the left eye, (C) annotation-related image for the right eye and annotation-related image for the left eye, and (D) final image for the right eye and final image for the left eye generated by the 3D annotation depiction system. [Figure 8] 13 is a diagram showing a state in which a 3D final video is generated by superimposing the right-eye final video and the left-eye final video. FIG. [Figure 9] In the same three-dimensional annotation depiction system, (A) is a schematic diagram showing a three-dimensional subject recognized by a user as viewed from the imaging axis direction of the camera, and (B) is a schematic diagram showing the three-dimensional subject as viewed from above and perpendicular to the imaging axis direction. [Figure 10] In the same three-dimensional annotation depiction system, (A) is a schematic diagram showing a three-dimensional subject recognized by a user as viewed from the imaging axis direction of the camera, and (B) is a schematic diagram showing the three-dimensional subject as viewed from above and perpendicular to the imaging axis direction. [Figure 11] A block diagram showing the functional configuration of a 3D video generation device according to a modified example of the 3D annotation drawing system. [Figure 12]FIG. 13 is a block diagram showing the functional configuration of an annotation processing device according to a modified example of the three-dimensional annotation drawing system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, a three-dimensional annotation drawing system according to an embodiment of the present invention will be described with reference to the accompanying drawings. In this embodiment, a case where the three-dimensional annotation drawing system is used in combination with a surgical support robot in a medical field is illustrated, but the present invention is not limited to this, and the system can also be used in combination with a production line in a factory, etc.

[0015] (Overall composition)

[0016] As shown in FIG. 1, the three-dimensional annotation drawing system 1 includes a right eye camera 10R and a left eye camera 10L mounted on an endoscope 5 inserted into the body of a patient K, a surgical robot arm (robotic forceps) 20 inserted into the body, a three-dimensional image generating device 100 to which images from the right eye camera 10R and the left eye camera 10L are input, a first three-dimensional display device 60 provided on the surgical console 50, a robot operation device 22 provided on the surgical console 50, a first annotation input device 270 provided on the surgical console 50, an annotation processing device 200 to which operation information of the first annotation input device 270 is input, a second three-dimensional display device 80 provided outside the surgical console 50, and a second annotation input device 280 provided outside the surgical console 50.

[0017] The surgery console 50 is viewed and operated by the doctor I performing the surgery. On the other hand, the second three-dimensional display device 80 and the second annotation input device 280 are viewed and operated by a supporter D, such as another doctor, who supports the doctor I.

[0018] The robot operation device 22 is a so-called master control, and is operated by the doctor I to control the operation of the robot arm 20 and the medical instrument at the tip of the robot arm 20.

[0019] The right-eye camera 10R captures images of the inside of the patient K's body from the viewpoint of the right eye of the doctor I. The left-eye camera 10L captures images of the inside of the patient K's body from the viewpoint of the left eye of the doctor I. Therefore, when comparing the right-eye image captured by the right-eye camera 10R and the left-eye image captured by the left-eye camera 10L, parallax occurs.

[0020] The 3D image generating device 100 is a so-called computer that uses right-eye imaging signals and left-eye imaging signals captured by the right-eye camera 10R and left-eye camera 10L to generate final 3D images (right-eye images and left-eye images), and transmits the 3D images to the first 3D display device 60 and the second 3D display device 80.

[0021] The first three-dimensional display device 60 and the second three-dimensional display device 80 are so-called 3D monitors, and display three-dimensional images. There are various three-dimensional display methods for 3D monitors. For example, in the case of a polarization method, a right-eye image and a left-eye image having different polarization directions (polarization rotation directions) are displayed in a superimposed manner (this includes both a case where the images themselves are displayed superimposed, and a case where the images are perceived superimposed by being alternately arranged in a striped or lattice-like area). The doctor I or the supporter D uses the polarized glasses 90 to make the right eye perceive only the right-eye image, and the left eye perceive only the left-eye image. In addition to the polarization method, the 3D monitor may have independent right-eye and left-eye monitors such as a head-mounted display (HMD), so that the right eye perceives the right-eye image on the right-eye monitor, and the left eye perceives the left-eye image on the left-eye monitor. In addition, a projector method may be adopted as the 3D monitor.

[0022] The first annotation input device 270 and the second annotation input device 280 are so-called mouse-type input devices, and the doctor I and the supporter D operate the mouse-type input device to draw annotations on the images while viewing the three-dimensional images on the first three-dimensional display device 60 and the second three-dimensional display device 80. Note that, although a mouse input device is illustrated here, the present invention is not limited to this, and various input devices such as a touch pad type input device, a touch pen type input device, and a stick type input device can be selected.

[0023] The annotation processing device 200 is a so-called computer, which receives operation information transmitted from the first annotation input device 270 and the second annotation input device 280, generates and stores annotation-related information, and transmits the annotation-related information to the 3D image generation device 100. The 3D image generation device 100, which has received the annotation-related information, generates 3D images for annotation (annotation-related images for the right eye and annotation-related images for the left eye). The final 3D images are generated by synthesizing the 3D images for annotation with 3D images for background (right-eye background images and left-eye background images) generated from the right-eye imaging signal and the left-eye imaging signal.

[0024] 2 shows a general-purpose internal configuration of a computer 40 employed in the 3D image generation device 100 and the annotation processing device 200. The computer 40 includes a CPU 41, a RAM 42, a ROM 43, an input device 44, a display device 45, an input / output interface 46, a bus 47, and a storage device 48. Note that the input device 44 (input keys, a keyboard, a mouse, etc.) and the display device 45 (display) are used when operating the computer 40 itself, and may be omitted.

[0025] The CPU 41 is a so-called central processing unit, and executes various programs to realize various functions of the 3D image generation device 100 and the annotation processing device 200. The RAM 42 is a so-called RAM (random access memory), and is used as a working area for the CPU 41. The ROM 43 is a so-called ROM (read only memory), and stores the basic OS and various programs executed by the CPU 41 (for example, the image generation program of the 3D image generation device 100 and the annotation processing program of the annotation processing device 200).

[0026] The storage device 48 is a hard disk, SSD memory, DAT, or the like, and is used to store large amounts of information.

[0027] Power and control signals are input and output through the input / output interface 46. The bus 47 is wiring that integrally connects the CPU 41, RAM 42, ROM 43, input device 44, display device 45, input / output interface 46, storage device 48, etc. for communication.

[0028] When the basic OS and various programs stored in the ROM 43 are executed by the CPU 41, the computer 40 functions as the 3D image generation device 100 and the annotation processing device 200.

[0029] (Details of annotation input device and annotation processing device)

[0030] 3A shows the functional configuration of the first annotation input device 270. Note that the second annotation input device 280 has the same configuration, so a description thereof will be omitted.

[0031] The first annotation input device 270 has a pointer depth movement instruction section 272 , a pointer up / down / left / right movement instruction section 274 , an annotation start / end instruction section 276 , an annotation deletion instruction section 278 , and an annotation type instruction section 279 .

[0032] FIG. 3B shows an example in which these functions are reflected in the mouse-type input device 270A. In the case of right-handed operation, the left click area corresponds to the annotation start / end instruction section 276, the right click area corresponds to the annotation deletion instruction section 278, the scroll wheel corresponds to the pointer depth movement instruction section 272, and the up / down / left / right movement detection section of the device itself corresponds to the pointer up / down / left / right movement instruction section 274. When the scroll wheel is rotated forward, the pointer is moved to the rear side, and when the scroll wheel is rotated forward, the pointer is moved to the front side. In this embodiment, the annotation type instruction section 279 is realized by a combination of the left click area and the up / down / left / right movement detection section. For example, the pointer is moved to the annotation type (character, square frame, circular frame, straight line, curve, free-form line, etc.) to be recorded from the type list 340 (see FIG. 8) displayed on the screen of the first or second three-dimensional display device 60, 80, and the left click is performed.

[0033] 4 shows the functional configuration (program configuration) of the annotation processing device 200. The annotation processing device 200 receives operation signals (right-click signal, left-click signal, scroll wheel signal, up / down / left / right movement signal) transmitted from the annotation input device 270, and generates and transmits annotation-related information. The annotation processing device 200 has a pointer depth position information generation unit 202, a pointer up / down / left / right position information generation unit 204, an annotation start / end information generation unit 206, an annotation deletion information generation unit 208, an annotation type information generation unit 209, an annotation-related information storage unit 210, and an annotation-related information transmission unit 220.

[0034] The relationship between the above-mentioned functions of the first annotation input device 270 and the annotation processing device 200 and the actual surgical field of the patient K will be described with reference to Fig. 6. Fig. 6(A) shows the surgical field of the patient K as viewed from the optical axis direction (defined here as the Z-axis direction / depth direction) of the endoscope 5, and Fig. 6(B) shows the surgical field as viewed from a direction perpendicular to the optical axis and in a vertical direction (defined here as the Y-axis direction). The direction perpendicular to the optical axis and in a horizontal direction is defined here as the X-axis direction, and the plane formed by the X-axis and Y-axis is defined as the XY plane.

[0035] <Move pointer>

[0036] In FIG. 6, it is assumed that the pointer P is virtually moved in the order of position A, position B, and position C. The doctor I rotates the scroll wheel serving as the pointer depth movement instruction unit 272 forward. The pointer depth position information generating unit 202, which has received this signal, generates information (pointer depth position information) for moving in the order of coordinates Paz, Pbz, and Pcz in the depth direction (Z-axis direction). The doctor I also moves the entire mouse-type input device 270A serving as the pointer up / down / left / right movement instruction unit 274. The pointer up / down / left / right position information generating unit 204, which has received this signal, generates information (pointer up / down / left / right position information) for moving in the order of coordinates (Pax, Pay), (Pbx, Pby), and (Pcx, Pcy) in the XY plane direction. As a result of combining the pointer depth position information and the pointer up / down / left / right position information, the pointer P moves in the order of position A, position B, and position C.

[0037] <Annotation record>

[0038] When Doctor I operates the annotation type indication unit 279, the annotation type information generation unit 209 receives the signal and generates an annotation type signal. For example, when a selection area for the annotation type to be recorded (character, square frame, circular frame, straight line, curve, free-form line, etc.) is displayed in the type list 340 on the image (see FIG. 8), and the doctor I moves the pointer to "straight line" and left-clicks, an annotation type signal meaning straight line depiction is generated.

[0039] When the doctor I operates the annotation start / end instruction unit 276, the annotation start / end information generating unit 206 that receives the signal generates an annotation recording start signal and a recording end signal. For example, when the left click of the mouse type input device 270A is pressed, a recording start signal is generated, and when the left click is released, a recording end signal is generated. As a result, while the left click is being pressed, an annotation is recorded along the movement trajectory of the pointer P. For example, in FIG. 7, when the recording start signal S and the recording end signal E are generated while the pointer P is moving from position A to position B, the annotation M is displayed on the first and second three-dimensional display devices 60, 80 by combining the movement trajectory of the pointer P from the start signal S to the end signal E and the selected annotation type signal.

[0040] Specifically, a series of pointer P movement trajectories (depth and up / down / left / right position information) starting from the pointer depth position information and pointer up / down / left / right position information at the time when the start signal S is generated to the pointer depth position information and pointer up / down / left / right position information at the time when the end signal E is generated are stored in the annotation related information storage unit 210 together with the annotation type signal.

[0041] In this embodiment, the depth and top / bottom / left / right position information of the pointer P from the start signal S to the end signal E is defined as annotation depth position information and annotation top / bottom / left / right position information. In addition, the annotation depth position information, annotation top / bottom / left / right position information, and annotation type information are collectively defined as annotation related information.

[0042] The annotation-related information stored in the annotation-related information storage unit 210 is transmitted to the 3D image generation device 100 by the annotation-related information transmission unit 220. When annotations are recorded multiple times, each piece of annotation-related information is sequentially stored as a database in the annotation-related information storage unit 210, and all of the annotation-related information being stored is transmitted collectively from the annotation-related information transmission unit 220 to the 3D image generation device 100. Note that the generation of an image of the annotation M is realized by the 3D image generation device 100 that receives the annotation-related information.

[0043] The annotation-related information transmission unit 220 transmits pointer depth position information and up / down / left / right position information of the pointer P (hereinafter, pointer-related information) to the 3D image generation device 100 simultaneously with the annotation-related information stored in the annotation-related information storage unit 210. As a result, a video of the pointer P is generated by the 3D image generation device 100 separately from the annotation M.

[0044] When Doctor I operates the annotation erasure instruction unit 278, the annotation erasure information generation unit 208 receives the signal and generates an annotation erasure signal to erase the previously generated annotation M. That is, the annotation-related information stored in the annotation-related information storage unit 210 is erased, and transmission of the erased information to the 3D image generation device 100 is stopped.

[0045] For example, when the pointer P is at an arbitrary position and the doctor I right-clicks on the mouse-type input device 270A, an erase signal may be generated to erase all annotations M included in the annotation-related information. On the other hand, an annotation erase signal may be generated to erase only a specific annotation M included in the annotation-related information by aligning the pointer P on a specific annotation M and then right-clicking.

[0046] (3D imaging device)

[0047] 5 shows the functional configuration of the 3D image generation device 100. The 3D image generation device 100 has a camera image receiving unit 102 (right eye camera image receiving unit 102R, left eye camera image receiving unit 102L), a background image generation unit 104 (right eye background image generation unit 104R, left eye background image generation unit 104L), a pointer image generation unit 106 (right eye pointer image generation unit 106R, left eye pointer image generation unit 106L), an annotation related image generation unit 108 (right eye annotation related image generation unit 108R, left eye annotation related image generation unit 108L), a background annotation image synthesis unit 110 (right eye background annotation image synthesis unit 110R, left eye background annotation image synthesis unit 110L), a left / right image synthesis unit 112, and an image output unit 114.

[0048] The right-eye camera video receiving section 102R of the camera video receiving section 102 receives a right-eye imaging signal captured by the right-eye camera 10R. Similarly, the left-eye camera video receiving section 102L receives a left-eye imaging signal captured by the left-eye camera 10L.

[0049] As shown in FIG. 7(A), the right-eye background image generating unit 104R of the background image generating unit 104 generates a right-eye background image 304R using a right-eye imaging signal. Similarly, the left-eye background image generating unit 104L generates a left-eye background image 304L using a left-eye imaging signal. The right-eye background image 304R and the left-eye background image 304L may be the right-eye imaging signal and the left-eye imaging signal themselves, but they can be still images or videos that have been subjected to various image processing to improve visibility. Naturally, parallax occurs in the right-eye background image 304R and the left-eye background image 304L. In this embodiment, an example is shown in which an organ U1, an organ U2, an organ U3, a blood vessel U4, and an organ U5 are arranged in the surgical field from the front side to the back side in the Z-axis direction.

[0050] As shown in Fig. 7(B), the right-eye pointer image generation unit 106R of the pointer image generation unit 106 generates a right-eye pointer image 306R by referring to the pointer-related information received from the annotation processing device 200. Similarly, the left-eye pointer image generation unit 106L generates a left-eye pointer image 306L by referring to the pointer-related information. Specifically, as shown in Fig. 6(B), when the pointer-related information (Pax, Pay, Paz) of the pointer P existing at the position A is received, the right-eye pointer image generation unit 106R calculates the right-eye pointer coordinates Ar by projecting the pointer P along the visual axis of the right eye onto the image reference plane α (for example, the XY plane when the Z-axis coordinate is 0 or the reference position) based on the bilateral parallax and convergence angle specific to humans, and inserts the image of the pointer P into the right-eye pointer coordinates Ar in the right-eye pointer image 306R. Similarly, the left eye pointer image generation unit 106L calculates left eye pointer coordinates Al by projecting the pointer P onto the image reference plane α along the visual axis of the left eye based on the bilateral parallax and convergence angle specific to humans, and inserts the image of the pointer P into the left eye pointer coordinates Al in the left eye pointer image 306L. As a result, the right eye pointer image 306R and the left eye pointer image 306L become pointer images for 3D stereoscopic vision that reflect the effects of parallax and convergence angle.

[0051] When the pointer P is at position B in Fig. 6, the image of the pointer P is inserted at the right eye pointer coordinate Br in the right eye pointer image 306R, and the image of the pointer P is inserted at the left eye pointer coordinate Bl in the left eye pointer image 306L. When the pointer P is at position C in Fig. 6, the image of the pointer P is inserted at the right eye pointer coordinate Cr in the right eye pointer image 306R, and the image of the pointer P is inserted at the left eye pointer coordinate Cl in the left eye pointer image 306L.

[0052] 7C, the right-eye annotation-related video generating unit 108R of the annotation-related video generating unit 108 uses the annotation-related information to generate the right-eye annotation-related video 308R. Similarly, the left-eye annotation-related video generating unit 108L uses the annotation-related information to generate the left-eye annotation-related video 308L.

[0053] Specifically, as shown in Fig. 6B, when annotation-related information corresponding to annotation M is received, the right-eye annotation-related video generating unit 108R calculates right-eye annotation coordinates Mr by projecting the annotation M onto the video reference plane α along the visual axis of the right eye based on the bilateral parallax and convergence angle specific to humans, and inserts the video of the annotation M into the right-eye annotation coordinates Mr in the right-eye annotation-related video 308R. Similarly, the left-eye annotation-related video generating unit 108L calculates left-eye annotation coordinates Ml by projecting the annotation M onto the video reference plane α along the visual axis of the left eye based on the bilateral parallax and convergence angle specific to humans, and inserts the video of the annotation M into the left-eye annotation coordinates Ml in the left-eye annotation-related video 308L. As a result, the annotation-related image for the right eye 308R and the annotation-related image for the left eye 308L become annotation-related images for 3D stereoscopic vision that reflect the effects of parallax and convergence angle.

[0054] As shown in FIG. 7(D), the right-eye background annotation video composition unit 110R of the background annotation video composition unit 110 superimposes the right-eye background video 304R, the right-eye pointer video 306R, and the right-eye annotation-related video 308R to generate the right-eye final video 310R. Similarly, the left-eye background annotation video composition unit 110L superimposes the left-eye background video 304L, the left-eye pointer video 306L, and the left-eye annotation-related video 308L to generate the left-eye final video 310L. Note that the right-eye final video 310R and the left-eye final video 310L in this state are in a so-called side-by-side format. For example, in the case of a two-eye display such as a head-mounted display, the images may be output as is from the video output unit 114 and displayed on the right-eye display and the left-eye display.

[0055] 8, the left-right image synthesis unit 112 synthesizes a right-eye final image 310R and a left-eye final image 310L to generate a single 3D final image 312. There are various synthesis methods, but for example, when the first 3D display device 60 and the second 3D display device 80 are polarization type monitors, the right-eye final image 310R is separated into comb-shaped regions V1, V2...Vn, the left-eye final image 310L is separated into comb-shaped regions W1, W2...Wn, and the right-eye comb-shaped regions V1, V2...Vn and the left-eye comb-shaped regions W1, W2...Wn are alternately arranged to generate a single 3D final image 312. A doctor I or a supporter D uses polarized glasses 90 to recognize only the right-eye pectinate regions V1, V2,..., Vn with the right eye and only the left-eye pectinate regions W1, W2,..., Wn with the left eye, thereby generating a 3D stereoscopic image in his / her head. It is preferable to add a type list 340 to the 3D final image 312.

[0056] The video output unit 114 transmits this 3D final video 312 to the first 3D display device 60 and the second 3D display device 80.

[0057] (Example of use)

[0058] Next, a method of using the present three-dimensional annotation depiction system 1 will be described with reference to Figures 9 and 10. Figures 9(A) and 10(A) show a schematic diagram of a three-dimensional space (operative field) recognized by the three-dimensional final image 312 displayed on the first three-dimensional display device 60 and the second three-dimensional display device 80 as viewed from the Z-axis direction by a doctor I or a supporter D, and Figures 9(B) and 10(B) show a schematic diagram of the three-dimensional space as viewed from the Y-axis direction.

[0059] FIG. 9 illustrates an example in which a linear annotation is recorded on the surface of a blood vessel U4. First, the doctor I or the supporter D uses the mouse-type input device (the first annotation input device 270 and the second annotation input device 280) to select an annotation type (here, a straight line) from the type list 340 in FIG. 9(A). Then, the doctor I or the supporter D aligns the pointer P with the target position Sxy at which the annotation recording is to be started on the XY plane, that is, the blood vessel U4. Then, the mouse-type input device's scroll wheel is moved forward and backward to move the pointer P along the depth direction (arrow Sz) in FIG. 9(B) (S1, S2, S3). Since the doctor I or the supporter D recognizes the image of the pointer P stereoscopically in a three-dimensional space, the operation of the scroll wheel and the movement of the pointer P in the depth direction are linked. The doctor I or the supporter D stops the scroll wheel when it is visually confirmed that the depth direction position S2 of the pointer P matches the depth direction position of the blood vessel U4.

[0060] Thereafter, the doctor I or the assistant D again aligns the pointer P with the exact start position Sxy (S2) where he / she wants to start recording the annotation, and inputs an instruction to start recording the annotation by depressing the left click of the mouse input device, and then moves the mouse input device to the end position Exy (E2), and releases the left click to input an instruction to end recording the annotation. As a result, a linear annotation M is recorded on the surface of the blood vessel U4 in the three-dimensional space. This annotation M will always be displayed as long as the corresponding annotation-related information is held in the annotation-related information storage unit 210.

[0061] A case where a square frame annotation M is recorded on the surface of an organ U5 following the recording of the annotation M in FIG. 9 will be illustrated with reference to FIG. 10. First, the doctor I or the supporter D uses a mouse-type input device to select an annotation type (square frame in this case) from the type list 340 in FIG. 9(A). Then, the doctor I or the supporter D aligns the pointer P with the target position Sxy at which the annotation recording is to be started on the XY plane, that is, the organ U5. Then, the pointer P is moved along the depth direction (arrow Sz) in FIG. 10(B) by moving the scroll wheel of the mouse-type input device forward and backward (S1, S2, S3). Since the image of the pointer P is recognized stereoscopically in a three-dimensional space by the doctor I or the supporter D, the operation of the scroll wheel and the movement of the pointer P in the depth direction are linked. The doctor I or the supporter D stops the scroll wheel when it is visually confirmed that the depth direction position S2 of the pointer P matches the depth direction position of the organ U5.

[0062] After that, the doctor I or the assistant D again aligns the pointer P with the exact start position Sxy (S2) where he / she wants to start recording the annotation, presses down the left click of the mouse input device to input an instruction to start recording the annotation, moves the mouse input device to the end position Exy (E2), and releases the left click to input an instruction to end recording the annotation. As a result, a square frame annotation M with the start position Sxy (S2) and the end position Exy (E2) on the diagonal is recorded on the surface of the organ U5 in three-dimensional space.

[0063] Next, a modified example of the three-dimensional image generating device 100 in the three-dimensional annotation depicting system 1 of the present embodiment will be introduced with reference to Figs. 11 and 12. As shown in Fig. 11, the modified three-dimensional image generating device 100 has a subject depth information calculation unit 120. This subject depth information calculation unit 120 calculates parallax and the like by performing a comparison process (e.g., pattern matching) of a right-eye image signal and a left-eye image signal captured by the right-eye camera 10R and the left-eye camera 10L, and calculates depth position information in the Z-axis direction for all dots on the XY plane of the subjects (organ U1, organ U2, organ U3, blood vessel U4, and organ U5). This subject depth position information is referred to when determining pointer depth position information in the annotation processing device 200 of Fig. 12.

[0064] This allows the doctor I or the supporter D to simply align the pointer P on the XY plane using the pointer up / down / left / right movement instruction unit 274 of the first annotation input device 270 and the second annotation input device 280, and the annotation processing device 200 can refer to the depth position information of the pointer P on the XY coordinates from the subject depth position information. In other words, the depth position of the pointer P is automatically adjusted so as to always trace the front surface of the subject.

[0065] Furthermore, if the device has an automatic depth adjustment function for the pointer P, the pointer P displayed in the final image does not need to be a three-dimensional image, and may be a two-dimensional image. Only the annotation display can be a three-dimensional image.

[0066] Incidentally, the present invention is not limited to the above-described embodiment, and it goes without saying that various modifications can be made without departing from the spirit and scope of the present invention. [Explanation of symbols]

[0067] 1 3D annotation depiction system 5 Endoscopy 10L Left eye camera 10R Right eye camera 20 Robot Arm 22 Robot operation device 40 calculator 50 Surgical Console 90 polarized glasses 100 Three-dimensional image generating device 102 Camera image receiving unit 104 Background image generation unit 106 Pointer image generation unit 108 Annotation-related image generation unit 110 Background annotation video synthesis unit 112 Left and right image synthesis unit 114 Video output section 200 Annotation Processing Device 202 Pointer depth position information generation unit 204 Pointer up / down / left / right position information generation unit 206 Annotation start / end information generation unit 208 Annotation deletion information generation unit 209 Annotation type information generation unit 210 Annotation related information storage unit 220 Annotation related information transmission unit 270 First annotation input device 270A Mouse-type input device 272 Pointer depth movement instruction section 274 Pointer up / down / left / right movement instruction section 276 Annotation start / end indicator 278 Annotation deletion instruction section 279 Annotation type indicator Doctor I K patient M annotation P Pointer α Image reference plane

Claims

1. A right-eye camera and a left-eye camera mounted on the endoscope; a first three-dimensional display device; a second three-dimensional display device; An annotation input device that operates a pointer; A three-dimensional annotation drawing system for surgical support comprising: a computer to which images from the right eye camera and the left eye camera are input; The computer includes: a camera image receiving unit that receives a right-eye image signal and a left-eye image signal of a subject imaged by the right-eye camera and the left-eye camera; a background image generating unit that generates a right-eye background image and a left-eye background image based on the right-eye imaging signal and the left-eye imaging signal; a pointer up / down / left / right position information generating unit that generates up / down / left / right position information of the pointer based on an operation signal indicating up / down movement and left / right movement transmitted from the annotation input device; a pointer depth position information generating unit that generates depth position information of the pointer based on an operation signal indicating a depth movement transmitted from the annotation input device; an annotation start / end information generating unit that generates recording start information and recording end information of an annotation in a three-dimensional space based on up / down / left / right position information of the pointer and depth position information of the pointer generated by the operation signal; an annotation-related information storage unit that stores depth position information of the pointer from the generation timing of the recording start information to the generation timing of the recording end information as depth position information of an annotation, and stores up, down, left, right position information of the pointer from the generation timing of the recording start information to the generation timing of the recording end information as up, down, left, right position information of the annotation; a pointer image generating unit that generates a right-eye pointer image and a left-eye pointer image by referring to at least up-down-left-right position information of the pointer; an annotation-related video generating unit that generates an annotation-related video for a right eye and an annotation-related video for a left eye by referring to depth position information of the annotation and up / down / left / right position information of the annotation; a background annotation image synthesis unit that synthesizes the right-eye background image, the left-eye background image, the right-eye pointer image, the left-eye pointer image, and the right-eye annotation-related image, and the left-eye annotation-related image to generate a right-eye final image and a left-eye final image; an image output unit that simultaneously outputs the final right-eye image and the final left-eye image to the first 3D display device and the second 3D display device; A three-dimensional annotation depiction system for surgical support, comprising:

2. The computer further includes a left-right image synthesis unit that generates a 3D final image by superimposing the right-eye final image and the left-eye final image. The three-dimensional annotation depiction system for surgical support according to claim 1.

3. the computer includes a subject depth position information calculation unit that calculates depth position information of the subject based on the right-eye background image and the left-eye background image, the pointer depth position information generating unit generates the depth position information of the pointer based on depth position information of the subject corresponding to up / down / left / right position information of the pointer. A three-dimensional annotation depiction system for surgical support according to any one of claims 1 to 2.

4. the pointer image generation unit generates the right eye pointer image and the left eye pointer image including parallax based on depth position information of the pointer. A three-dimensional annotation depiction system for surgical support according to any one of claims 1 to 3.

5. The annotation input device includes a first annotation input device operated by a user of the first three-dimensional display device, and a second annotation input device operated by a user of the second three-dimensional display device, the computer generates the right-eye annotation-related image and the left-eye annotation-related image by referring to operation signals transmitted from both the first annotation input device and the second annotation input device. A three-dimensional annotation depiction system for surgical support according to any one of claims 1 to 4.

Images