Image processing apparatus, image processing system, and image processing method

Abstract

To provide an image processing apparatus, an image processing system, an image processing method, and an image processing program for causing a user to experience simulation of surgery in a mixed reality space with a simpler configuration.SOLUTION: In an image processing system, a PC generates an image representing an internal structure of a human body before surgery, as an endoscope monitor image on the basis of a first virtual object including the internal structure of the human body before surgery and a real object position attitude being a position attitude of a real object corresponding to an endoscope, outputs the endoscope monitor image and reports success or failure of the surgery depending on whether or not a relation between a position of an affected part in the first virtual object and a position of the real object corresponding to a surgical instrument satisfies a prescribed condition after the execution of a user operation on the real object corresponding to the surgical instrument, in a prescribed state.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present invention relates to image processing technology. [Background technology]

[0002] In recent years, mixed reality (MR) technology has become widespread. Using mixed reality technology, a user wearing a head mounted display (HMD) can be provided with a mixed reality image in which a CG model is placed on a real image, allowing the user to experience a mixed reality world that combines reality and virtuality. To generate a mixed reality image, a method is used in which the position of the HMD is identified using sensors or two-dimensional markers.

[0003] Patent Document 1 describes a technology that uses a two-dimensional marker in an image captured by the HMD to identify the position and orientation of the HMD and provide a mixed reality image. Patent Document 2 describes a technology that uses a magnetic sensor to identify the position and orientation of the HMD and provide a mixed reality image. Patent Document 3 describes a technology that combines a realistic tissue model with a digital environment to provide a tactile sensation close to reality, an immersive environment. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2003-308514 A [Patent Document 2] JP 2003-240532 A [Patent Document 3] Patent Publication No. 2021-043443 Summary of the Invention [Problem to be solved by the invention]

[0005] For example, in Patent Document 3, a realistic tissue model must be prepared, which makes the device configuration complicated. In a mixed reality space, when a virtual object having the internal structure of a human body is used as a virtual object, the organs of the human body that this virtual object has are also virtual objects, so that such organs as virtual objects cannot come into contact with the surgical instruments operated by the user. Therefore, there is no feedback before and after the operation, making it difficult to judge whether the surgical instruments have worked well.

[0006] The present invention provides a technique for allowing a user to experience a surgical simulation in a mixed reality space with a simpler configuration. [Means for solving the problem]

[0007] One aspect of the present invention is characterized by comprising a generation means for generating an image representing the internal structure of the human body before surgery as an endoscope monitor image based on a first virtual object including the internal structure of the human body before surgery and a real object position and orientation that is the position and orientation of a real object corresponding to an endoscope, an output means for outputting the endoscope monitor image, and a notification means for notifying the success or failure of the surgery based on whether a relationship between a position of the affected area in the first virtual object and a position of a real object corresponding to a surgical instrument satisfies a predetermined condition after a user operation is performed on the real object corresponding to the surgical instrument in a predetermined state. Effect of the Invention

[0008] According to the present invention, it is possible to provide a technique for allowing a user to experience a surgical simulation in a mixed reality space with a simpler configuration. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing a connection form between a PC 100 and an HMD 101 in the system. [Diagram 2] FIG. 2 is a block diagram showing an example of the hardware configuration of each of the PC 100 and the HMD 101. [Diagram 3] FIG. 2 is a block diagram showing an example of the functional configuration of the HMD 101 and the PC 100. [Figure 4] FIG. 2 is a diagram showing an example of a software module configuration in the PC 100. [Diagram 5] 4 is a flowchart showing the operations of the PC 100 and the HMD 101. [Figure 6] FIG. 4 is a diagram showing an example of an information configuration. [Figure 7] FIG. 1 is a diagram showing an example of an implementation environment of the system. [Figure 8] 13 is a flowchart showing an example of a process in step S509. [Figure 9] 10 is a flowchart showing an example of the process in step S803. [Figure 10] (a) is an overhead view of the workspace, and (b) is an overhead view of the workspace with a virtual human body object placed therein (mixed reality space). [Figure 11] FIG. 4 is a diagram showing a display example of an endoscope monitor image. [Figure 12] FIG. 1 is a diagram showing an example of a GUI configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims, and not all combinations of features described in the embodiments are essential to the invention. Two or more features among the multiple features described in the embodiments may be arbitrarily combined. In addition, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] [First embodiment] 1, the system according to this embodiment is configured such that an HMD 101, which is an example of a head-mounted display device, and a PC 100, which is an example of an image processing device, can communicate data with each other via a network 150. The network 150 may be a wired network or a wireless network.

[0012] Next, an example of the hardware configuration of each of the PC 100 and the HMD 101 will be described with reference to the block diagram of FIG.

[0013] First, a description will be given of an example of the hardware configuration of the PC 100. The CPU 201 executes various processes using computer programs and data stored in the RAM 203. As a result, the CPU 201 controls the overall operation of the PC 100, and executes or controls various processes that will be described as processes performed by the PC 100.

[0014] The ROM 202 stores setting data for the PC 100, computer programs and data related to the startup of the PC 100, computer programs and data related to the basic operations of the PC 100, and the like.

[0015] The RAM 203 has an area for storing computer programs and data loaded from the ROM 202 or the external memory 211. The RAM 203 also has an area for storing captured images received from the HMD 101 via an external input terminal (e.g., an IEEE 1394 terminal) in the general-purpose bus 212. The RAM 203 also has an area for storing data received from the outside via the communication I / F controller 208. The RAM 203 also has a work area used when the CPU 201 executes various processes. In this way, the RAM 203 can provide various areas as appropriate.

[0016] The video controller 206 is a controller that controls the input and output of information to the HMD 101 and the display 210, and outputs images and text to the HMD 101 and the display 210 via an external output terminal (for example, a digital visual interface).

[0017] Display 210 has a liquid crystal screen or a touch panel screen, and displays the results of processing by CPU 201 as images, characters, etc. The touch panel screen may be a touch panel screen capable of detecting positions touched by multiple fingers, such as a multi-touch screen. Display 210 may also be a projection device such as a projector that projects images and characters.

[0018] An input device 209 is connected to the input controller 205. The input device 209 is a user interface such as a keyboard, a mouse, a touch panel screen, etc., and can input various instructions and information to the PC 100 by the user operating the input device 209.

[0019] The memory controller 207 is a controller that reads and writes computer programs and data from and to the external memory 211. The external memory 211 is a non-volatile memory for storing large amounts of information, such as a hard disk drive device. The external memory 211 stores an OS (operating system), computer programs and data for causing the CPU 201 to execute or control various processes described as processes performed by the PC 100, and the like. The computer programs and data stored in the external memory 211 are loaded into the RAM 203 as appropriate under the control of the CPU 201, and become targets for processing by the CPU 201.

[0020] The external memory 211 may be another type of memory device, such as a flexible disk (FD) or a card-type memory connected to a PCMCIA card slot via an adapter.

[0021] The communication I / F controller 208 performs data communication with an external device via the network 150 (for example, Internet communication using TCP / IP). The communication I / F controller 208 also acquires the measurement results obtained by the optical sensor 104 via Gigabit Ethernet (registered trademark) or the like.

[0022] The general-purpose bus 212 , the CPU 201 , the ROM 202 , the RAM 203 , the video controller 206 , the input controller 205 , the memory controller 207 , and the communication I / F controller 208 are all connected to a system bus 204 .

[0023] Next, an example of the hardware configuration of the HMD 101 will be described.

[0024] The right-eye / left-eye video camera 221 has a right-eye video camera that captures moving images of real space observed from near the right eye of a user wearing the HMD 101 on his / her head, and a left-eye video camera that captures moving images of real space observed from near the left eye of the user. The right-eye video camera is attached to the HMD 101 so as to be located near the right eye of the user wearing the HMD 101 on his / her head. Similarly, the left-eye video camera is attached to the HMD 101 so as to be located near the left eye of the user wearing the HMD 101 on his / her head.

[0025] The right-eye / left-eye display 222 has a right-eye display attached to the HMD 101 so as to be positioned in front of the right eye of a user wearing the HMD 101 on his / her head, and a left-eye display attached to the HMD 101 so as to be positioned in front of the left eye of the user.

[0026] Next, an example of the functional configuration of each of the HMD 101 and the PC 100 is shown in the block diagram of Fig. 3. The functional units shown in Fig. 3 may be implemented by hardware or software (computer program).

[0027] First, an example of the functional configuration of the HMD 101 will be described.

[0028] The acquisition unit 301 acquires captured images captured by the right-eye / left-eye video camera 221. When the right-eye / left-eye video camera 221 captures a moving image, the acquisition unit 301 acquires images of each frame in the moving image as captured images. When the right-eye / left-eye video camera 221 captures still images periodically or irregularly, the acquisition unit 301 acquires the still images as captured images.

[0029] The real image transmitting unit 302 transmits the captured image (real image) acquired by the acquiring unit 301 to the PC 100. The mixed reality image receiving unit 303 receives the mixed reality image transmitted from the PC 100.

[0030] The mixed reality image display unit 304 displays the mixed reality image for the right eye received by the mixed reality image receiving unit 303 on the right-eye display in the right-eye / left-eye display 222. In addition, the mixed reality image display unit 304 displays the mixed reality image for the left eye received by the mixed reality image receiving unit 303 on the left-eye display in the right-eye / left-eye display 222.

[0031] Next, an example of the functional configuration of the PC 100 will be described.

[0032] The real image receiving and storing unit 311 receives a real image transmitted from the HMD 101 , and stores the received real image in a memory device such as the RAM 203 or the external memory 211 .

[0033] The HMD position and orientation specification unit 312 specifies the position and orientation of the HMD 101. There are various methods for specifying the position and orientation of the HMD 101, and the method is not limited to a specific method. For example, the HMD position and orientation specification unit 312 may specify the position and orientation of the HMD 101 based on a real image (for example, a method using a two-dimensional marker in a real image described in Patent Document 1 may be used). Also, for example, the HMD position and orientation specification unit 312 may specify the position and orientation of the HMD 101 based on a measurement result by the optical sensor 104 described above.

[0034] Then, the HMD position and orientation identification unit 312 stores the identified position and orientation of the HMD 101 in a memory device such as the RAM 203 or the external memory 211. For example, the HMD position and orientation identification unit 312 registers a three-dimensional position 612 of the HMD 101 and a three-dimensional orientation 613 of the HMD 101 in a row of ID 611=HMD1 in the position and orientation information 610, as shown in FIG.

[0035] The 3D model storage unit 313 manages the rendering data of various virtual objects in the external memory 211. The rendering data of the virtual object includes, for example, data representing the geometric shape of the virtual object, data representing the position and orientation of the virtual object, and texture data of the virtual object. In this embodiment, the virtual object includes a virtual object simulating a human body (human body virtual object). The human body virtual object is a living body virtual object in which a virtual object representing the surface (skin) of the human body and a virtual object representing the contents of the human body (bones, organs, etc.) are arranged in the same manner as an actual human body. The human body virtual object includes a pre-operative human body virtual object that is a human body virtual object simulating a human body before surgery, and a post-operative human body virtual object that is a human body virtual object simulating a human body after surgery.

[0036] The endoscope position and orientation specification unit 314 specifies the position and orientation of a real object corresponding to the endoscope. The "real object corresponding to the endoscope" is, for example, a mock-up that imitates the shape of an endoscope. There are various methods for specifying the position and orientation of such a real object, and the method is not limited to a specific method. For example, when a position and orientation sensor is attached to the real object corresponding to the endoscope, the endoscope position and orientation specification unit 314 acquires the measurement result by the position and orientation sensor as the position and orientation of the real object. Also, for example, when a two-dimensional marker is attached to the real object corresponding to the endoscope, the endoscope position and orientation specification unit 314 obtains the position and orientation of the real object using the two-dimensional marker in the captured image captured by the right-eye / left-eye video camera 221. Then, the endoscope position and orientation specification unit 314 stores the specified position and orientation in a memory device such as the RAM 203 or the external memory 211. For example, as shown in FIG. 6(a), the endoscope position / posture identification unit 314 registers a three-dimensional position 612 of a physical object corresponding to the endoscope and a three-dimensional posture 613 of the physical object corresponding to the endoscope in the row of ID 611=VA1 in the position / posture information 610.

[0037] The surgical instrument position / orientation identifying unit 315 identifies the position / orientation of a real object corresponding to a surgical instrument. An example of a "real object corresponding to a surgical instrument" is a mock-up that mimics the shape of a surgical instrument. There are various methods for identifying the position / orientation of such a real object, and similar to the method for identifying the position / orientation of a real object corresponding to an endoscope, the method is not limited to a specific method.

[0038] Then, the surgical instrument position / orientation identifying unit 315 stores the identified position / orientation in a memory device such as the RAM 203 or the external memory 211. For example, as shown in Fig. 6(a), the surgical instrument position / orientation identifying unit 315 registers a three-dimensional position 612 of a physical object corresponding to the surgical instrument and a three-dimensional orientation 613 of the physical object corresponding to the surgical instrument in a row of ID 611 = LS1 in the position / orientation information 610.

[0039] The endoscopic image generation unit 316 generates an image of the human body virtual object as seen from the position and orientation, based on the position and orientation of a part (tip of the real object corresponding to the endoscope) of the "real object corresponding to the endoscope" that corresponds to the "camera possessed by the endoscope" and the human body virtual object, as an endoscopic monitor image.

[0040] The composite reality image generating unit 317 generates, as a virtual space image, an image of "a virtual space in which a human body virtual object and an endoscope monitor image are arranged" that can be seen according to the position and orientation of the HMD 101 identified by the HMD position and orientation identifying unit 312. The composite reality image generating unit 317 then generates, as a mixed reality image, a composite image by combining the virtual space image with the real image received by the real image receiving and storing unit 311.

[0041] The mixed reality image transmission unit 318 transmits the mixed reality image generated by the mixed reality image generation unit 317 to the HMD 101 so that the mixed reality image is displayed on the HMD 101.

[0042] Note that the HMD 101 may have one or more of the functional units shown as functional units of the PC 100 in Fig. 3. Also, the PC 100 may be implemented by a plurality of computer devices, and the functional units shown in Fig. 3 may be distributed among the plurality of computer devices.

[0043] Next, an example of a software module configuration in the PC 100 will be described with reference to FIG.

[0044] The PC 100 includes an OS 401, a graphic engine 402, a mixed reality platform 403 (also called an MR platform), and a mixed reality application 404 (also called an MR application or a viewer application). These pieces of software are executed by the CPU 201.

[0045] The OS 401 controls the input and output of the HMD 101 , passes real images obtained from the right-eye / left-eye video cameras 221 to a mixed reality platform 403 , and outputs mixed reality images drawn by the graphics engine 402 to the right-eye / left-eye displays 222 .

[0046] The graphic engine 402 generates a virtual space image using virtual objects stored in the external memory 211, and synthesizes the virtual space image by superimposing it on a real image. The engine used for drawing may be, for example, a widely used graphic engine such as OpenGL or DirectX, or a uniquely developed graphic engine. In this embodiment, OpenGL is used as the graphic library.

[0047] The mixed reality platform 403 identifies the position and orientation of the HMD 101 and aligns the real space with the virtual space. The position and orientation and alignment techniques can be realized using known techniques disclosed in Japanese Patent Application Laid-Open No. 2002-32784, Japanese Patent Application Laid-Open No. 2006-072903, Japanese Patent Application Laid-Open No. 2007-166427, etc.

[0048] It is also possible to provide a position sensor in the HMD 101, without using a two-dimensional marker, and specify the position and orientation of the HMD 101 based on the position measured by triangulation using this position sensor.

[0049] The mixed reality application 404 receives the position and orientation of the HMD 101, shape information of the 3D model, and position and orientation information from the mixed reality platform 403, and issues a drawing command for the 3D model to the graphic engine 402. At this time, a command is issued using the OpenGL API, in which identification information and position and orientation information of the 3D model to be drawn are set.

[0050] Next, the operations of the PC 100 and the HMD 101 will be described with reference to the flowchart of FIG.

[0051] In step S501, the HMD 101 performs various initialization processes related to startup.

[0052] In step S502, the right-eye / left-eye video camera 221 in the HMD 101 captures images of the real space according to the position and orientation of the HMD 101 to obtain captured images of the real space (a captured image for the right eye and a captured image for the left eye). The processes in and after step S502 are repeatedly performed for each captured image captured by the right-eye / left-eye video camera 221.

[0053] In step S503, the HMD 101 transmits to the PC 100 the captured image (real image) acquired in step S502.

[0054] In step S504, the CPU 201 receives the captured image transmitted from the HMD 101 via the general-purpose bus 212.

[0055] In step S505, the CPU 201 stores the captured image received in step S504 in a memory device such as the RAM 203 or the external memory 211.

[0056] In step S506, the CPU 201 acquires the position and orientation of the HMD 101. As described above, the method for acquiring the position and orientation of the HMD 101 is not limited to a specific method.

[0057] In step S507, the CPU 201 stores the position and orientation of the HMD 101 acquired in step S506 in a memory device such as the RAM 203 or the external memory 211.

[0058] In step S508, the CPU 201 constructs a human body virtual object (for example, a pre-surgery human body virtual object) based on rendering data of the human body virtual object stored in the external memory 211. The CPU 201 then places the human body virtual object in a position and orientation that is preset as the placement position and orientation of the human body virtual object. The CPU 201 then generates, as a virtual space image, an image of the "virtual space in which the human body virtual object is placed" that is visible from the position and orientation of the HMD 101. The CPU 201 then generates, as a mixed reality image, a composite image that combines the captured image acquired in step S504 and the virtual space image.

[0059] In step S509, the CPU 201 generates an endoscope monitor image, and generates an image by combining the generated endoscope monitor image with the mixed reality image generated in step S508 as a second mixed reality image. The process in step S509 will be described later.

[0060] In step S510, the CPU 201 controls the video controller 206 to transmit the second mixed reality image generated in step S509 to the HMD 101.

[0061] In step S511, the HMD 101 receives the second mixed reality image transmitted in step S510.

[0062] In step S512, the HMD 101 displays the second mixed reality image received in step S511 on the right-eye / left-eye display 222.

[0063] Here, an example of an implementation environment of the system according to this embodiment will be described with reference to FIG.

[0064] An origin marker 106 that defines the origin and three axes (x-axis, y-axis, z-axis) of the world coordinate system is provided on a tabletop that serves as a work area for performing a surgical simulation. The PC 100 recognizes space using spatial feature points of a "captured image of real space including the origin marker 106" captured by the HMD 101. Therefore, even if the origin marker 106 cannot be confirmed from the captured image, the PC 100 can obtain the position and orientation of the HMD 101 (the relative position and orientation with respect to the origin marker 106) through spatial recognition.

[0065] A mockup 105, which is a real object corresponding to a surgical instrument, is provided with target markers 109a and 110a, which are two-dimensional markers, so that the mockup 105 can be confirmed from any angle. The PC 100 can obtain the position and orientation of the mockup 105 (relative position and orientation with respect to the origin marker 106) using the target markers 109a and 110a in the captured image.

[0066] The mockup 105 is also provided with an operation unit such as a button that can be operated by the user, and the operation unit is connected to the PC 100 via a network 150. When the user operates the operation unit (performs an operation such as operating a lever or button of a surgical instrument), the PC 100 is notified of this via the network 150.

[0067] A mock 107, which is a real object corresponding to an endoscope, is provided with target markers 109b, 110b, which are two-dimensional markers, so that the mock 107 can be confirmed from any angle. The PC 100 can obtain the position and orientation of the mock 107 (relative position and orientation with respect to the origin marker 106) using the target markers 109b, 110b in the captured image. A range 113 indicates the imaging range of the camera of the endoscope corresponding to the mock 107.

[0068] The board 111 is provided with a plurality of holes through which the guides 112 can be inserted and removed. The mocks 105 and 107 are fixed and attached to the guides 112 inserted into any of the holes. The mocks 105 and 107 are passed through the guides 112 fixed on the board 111 that determines the trocar installation position, thereby fixing the position of the trocar through which the human body virtual object passes, and the position becomes the insertion parts 1016a and 1016b of the human body virtual object 1011 as shown in FIG. 10(b). FIG. 10(a) is an overhead view of the above-mentioned work area. FIG. 10(b) is an overhead view of the state (mixed reality space) in which the human body virtual object is placed in the work area.

[0069] Next, an example of the process in step S509 will be described with reference to the flowchart in FIG.

[0070] In step S801, the CPU 201 acquires the positions and orientations of the surgical instrument mockup and the endoscope mockup. As described above, there are various methods for acquiring the positions and orientations of the mockups, and the method is not limited to a specific method. The CPU 201 then stores the acquired positions and orientations in a memory device such as the RAM 203 or the external memory 211.

[0071] In step S802, the CPU 201 obtains the position and orientation of a portion of the mockup of the surgical instrument corresponding to the tip of the surgical instrument (hereinafter referred to as the first position and orientation (first position and first orientation)) based on the position and orientation of the mockup. For example, as shown in FIG. 6(a), in the position and orientation information 610 previously stored in the external memory 211, a relative three-dimensional position 612 of the tip of the surgical instrument from the reference position of the surgical instrument and a relative three-dimensional orientation 613 of the tip of the surgical instrument from the reference orientation of the surgical instrument are registered in advance in the row of ID611=LS1-HEAD. Therefore, in this case, the CPU 201 obtains, as the first position, a three-dimensional position obtained by adding "the relative three-dimensional position 612 of the tip of the surgical instrument from the reference position of the surgical instrument" to the position of the mockup of the surgical instrument obtained in step S801. Similarly, the CPU 201 obtains a three-dimensional orientation as a first orientation by adding the "relative three-dimensional orientation 613 of the tip of the surgical instrument from the reference orientation of the surgical instrument" to the orientation of the mock surgical instrument acquired in step S801.

[0072] Further, the CPU 201 obtains the position and orientation of the portion of the mock of the endoscope corresponding to the camera of the endoscope (hereinafter referred to as the second position and orientation (second position and second orientation)) based on the position and orientation of the mock. For example, as shown in FIG. 6A, in the row of ID 611=VA1-RANGE that identifies the virtual imaging range 113 of the camera of the endoscope in the position and orientation information 610, a relative three-dimensional position 612 of the camera from the reference position of the endoscope and a relative three-dimensional orientation 613 of the camera from the reference orientation of the endoscope are registered in advance. Therefore, in this case, the CPU 201 obtains a three-dimensional position obtained by adding "the relative three-dimensional position 612 of the camera from the reference position of the endoscope" to the position of the mock of the endoscope obtained in step S801 as the second position. Similarly, the CPU 201 obtains a three-dimensional orientation obtained by adding "the relative three-dimensional orientation 613 of the camera from the reference orientation of the endoscope" to the orientation of the mock of the endoscope obtained in step S801 as the second orientation.

[0073] In step S803, the CPU 201 generates an image of the human body virtual object seen from the second position and posture as an endoscope monitor image based on the second position and posture and the human body virtual object. The process of step S803 will be described later.

[0074] In step S804, the CPU 201 places the endoscope monitor image generated in step S803 in a virtual space at a position and orientation that is preset as the placement position and orientation of the endoscope monitor image, generates the "placed endoscope monitor image" that is visible from the position and orientation of the HMD 101, and generates an image by combining the generated image with the mixed reality image generated in step S508 as a second mixed reality image.

[0075] The method for setting the arrangement position and orientation of the endoscope monitor image is not limited to a specific setting method. For example, the arrangement position and orientation of the endoscope monitor image may be set using a GUI (Graphical User Interface) as exemplified in FIG.

[0076] The setting unit 1211 includes a box for inputting a profile name of the arrangement position and orientation of the endoscope monitor image, a box for inputting the position (X, Y, Z) at which the endoscope monitor image is arranged, a box for inputting the orientation (direction) (X, Y, Z) at which the endoscope monitor image is arranged, a box for inputting the size of the endoscope monitor image (horizontal size X in FIG. 12), a button for inputting an instruction to save the information group inputted in these boxes as a profile in the external memory 211, and a button for inputting an instruction to open a profile already saved in the external memory 211. Note that the information set by operating the operation unit 1211 is an example. For example, as shown in FIG. 6(b), a profile 670 including ID 671=profile name, position 672=position (X, Y, Z) of the endoscope monitor image, and orientation 673=orientation (X, Y, Z) of the endoscope monitor image is created and stored in the external memory 211.

[0077] Such a GUI is displayed on the display 210 by, for example, executing a corresponding computer program by the CPU 201. The user operates the input device 209 to input operations to the GUI (input to a box or an instruction to press a button). The CPU 201 executes processing according to the instructions input by the operation input.

[0078] In this way, by saving the placement position, orientation, size, etc. of the endoscopic monitor image together with the profile name, it is possible to reproduce an endoscopic monitor in the same environment as an actual surgical site, and to reproduce a position and orientation of the endoscopic monitor that is easy to see.

[0079] In step S805, the CPU 201 determines whether the surgical instrument operation flag (initial value is OFF) is ON. If the result of this determination is that the surgical instrument operation flag is ON, the process proceeds to step S809, and if the surgical instrument operation flag is OFF, the process proceeds to step S806.

[0080] In step S806, the CPU 201 judges whether or not the distance between a position that is preset as the position of the affected area in the human body virtual object and the first position (the tip position of the surgical instrument) is within a first predetermined distance. For example, as shown in Fig. 6(c), the external memory 211 stores in advance affected area position information 680 in which an ID 681 of the affected area and a position 682 of the affected area are registered for each affected area. In this case, the CPU 201 judges whether or not the distance between any of the positions registered in the affected area position information 680 and the first position (the tip position of the surgical instrument) is within a first predetermined distance.

[0081] If the result of this determination is that the distance between the position that is preset as the position of the affected part in the human body virtual object and the first position is within the first predetermined distance, the process proceeds to step S807; if the distance between the position that is preset as the position of the affected part in the human body virtual object and the first position is not within the first predetermined distance, the process proceeds to step S510.

[0082] In step S807, CPU 201 determines whether the operation unit of the mock surgical instrument has been operated by the user (by operating a button or lever). When the user operates the operation unit of the mock surgical instrument, the operation unit transmits a signal indicating the content of the operation to PC 100 via network 150, and CPU 201 receives the signal via communication I / F controller 208. Therefore, when CPU 201 receives the signal, it determines that "the operation unit of the mock surgical instrument has been operated by the user (by operating a button or lever)," and when it does not receive the signal, it does not determine that "the operation unit of the mock surgical instrument has been operated by the user (by operating a button or lever)."

[0083] If it is determined that "the operating part of the mock surgical instrument has been operated by the user (by operating a button or lever)" as a result of such a determination, processing proceeds to step S808; if it is not determined that "the operating part of the mock surgical instrument has been operated by the user (by operating a button or lever)", processing proceeds to step S510.

[0084] Here, the operation of the operating unit of the mock surgical instrument by the user corresponds to, for example, the operation of an automatic suturing treatment using a linear stapler. In the case of a linear stapler, a certain treatment time is required until the automatic suturing is completed after the lever or button operation, and the linear stapler needs to be held in that position during the treatment time. The end of the automatic suturing treatment can be reproduced by holding the linear stapler in a predetermined position (affected area) for a certain treatment time.

[0085] In step S808, the CPU 201 sets a surgical instrument operation flag to ON, which indicates that the surgical instrument is in an operating state. If the CPU 201 arranges a virtual object (surgical instrument virtual object) that resembles the shape of the surgical instrument according to the position and orientation of the surgical instrument, the mixed reality image includes an image of the surgical instrument virtual object that is visible according to the position and orientation of the HMD 101. In this case, from this point onwards, the operation of the tip of the mock surgical instrument (automatic suturing in the case of a linear stapler) continues to be displayed on the endoscope monitor image until it is determined in step S810 that the operation of the operation unit of the mock surgical instrument has ended.

[0086] On the other hand, in step S809, the CPU 201 judges whether or not the distance between the position preset as the position of the affected part in the human body virtual object and the first position (the tip position of the surgical instrument) is within the first predetermined distance, as in step S806. If the result of this judgment is that the distance between the position preset as the position of the affected part in the human body virtual object and the first position is within the first predetermined distance (predetermined state), the process proceeds to step S810, and if the distance between the position preset as the position of the affected part in the human body virtual object and the first position is not within the first predetermined distance, the process proceeds to step S813. This judgment branch is a condition for considering the surgery successful only when the elapsed time since the user operated the mockup of the surgical instrument is equal to or longer than a predetermined time in a state in which the distance between the position of the affected part in the human body virtual object and the first position (the tip position of the surgical instrument) is within the first predetermined distance. In this embodiment, the first predetermined distance is 15 mm, but the first predetermined distance can be changed as appropriate depending on the content of the surgery. The predetermined state may be a state in which the first position approaches a position that is preset as the position of the affected area from a predetermined posture (angle) and is within the first predetermined distance.

[0087] In step S810, the CPU 201 determines whether or not the predetermined operation of the surgical instrument has been completed (predetermined condition). If the time that has elapsed since the surgical instrument operation flag was turned ON exceeds a predetermined time, the CPU 201 determines that the predetermined operation of the surgical instrument has been completed, and if the time that has elapsed since the surgical instrument operation flag was turned ON does not exceed the predetermined time, the CPU 201 does not determine that the predetermined operation of the surgical instrument has been completed.

[0088] If it is determined that the predetermined operation of the surgical instrument has been completed as a result of this determination, the process proceeds to step S811, and if it is not determined that the predetermined operation of the surgical instrument has been completed, the process proceeds to step S510.

[0089] In step S811, the CPU 201 sets the post-operative human body virtual object to be used as the human body virtual object in the next step S803. In this embodiment, until the process of step S811 is executed, the pre-operative human body virtual object is set to be used as the human body virtual object in step S803.

[0090] Therefore, if the distance between the position of the affected area in the pre-operative human body virtual object and the position of the surgical instrument mock is within a first predetermined distance and the elapsed time since the user operated the surgical instrument mock exceeds a predetermined time, in the next step S803, an endoscopic monitor image is generated based on the post-operative human body virtual object and the second position and orientation.

[0091] In step S812, the CPU 201 synthesizes a character string or an image indicating that the surgery was successful at a specified position in the second mixed reality image as a success indication. The success indication may be synthesized on the endoscope monitor image or outside the endoscope monitor image. In addition, the success indication may simply be a display of the endoscope monitor image generated from the post-surgery human body virtual object in step S803, or may be a sound effect or voice indicating success, or a light turning on or blinking. Then, the CPU 201 sets the surgical instrument operation flag to OFF.

[0092] Here, a display example of an endoscope monitor image will be described with reference to Fig. 11. For example, as shown in Fig. 10(b), an endoscope monitor image 1015 is arranged at a predetermined position and orientation in mixed reality space. In an actual endoscope monitor image, the mock endoscope moves, so it is difficult to continue to display only the same part as in Fig. 11, but for simplicity's sake, the same part is displayed in Fig. 11.

[0093] An endoscope monitor image 1015-1 shows a state in which mock surgical instruments have not yet been inserted into tumor sites 1102 and 1103, which are diseased sites in a pulmonary artery 1101.

[0094] The endoscope monitor image 1015-2 shows a state in which a portion 1105 corresponding to the tip of the surgical instrument in the surgical instrument virtual object is displayed within the endoscope monitor image 1015-1.

[0095] An endoscope monitor image 1015-3 shows a state in which a hint display 1104 (for example, a red circle) which is an image that highlights the periphery of the resection part (affected part) is displayed within the endoscope monitor image 1015-1. When the user operates the input device 209 to designate a button 1205 in the GUI of Fig. 12, the CPU 201 turns on the hint function and sets it so that the hint display 1104 is displayed. When the user does not operate the input device 209 to designate the button 1205, the CPU 201 turns off the hint function and sets it so that the hint display 1104 is not displayed.

[0096] The red circle, which is a hint display, may be displayed with different color intensity and size depending on the distance between the tip of the surgical instrument (mock surgical instrument) and the affected area (tumor area).

[0097] An endoscope monitor image 1015-4 shows a state in which a portion 1105 corresponding to the tip of the surgical instrument in the surgical instrument virtual object is displayed on the endoscope monitor image 1015-3.

[0098] An endoscope monitor image 1015-5 shows the result of combining the endoscope monitor image 1015-4 with the success display 1106 in step S812.

[0099] 12 has a pull-down button 1201 for displaying a list of candidate trocar positions and a pull-down button 1202 for displaying a list of candidate tumor positions, which are diseased areas. When the user designates either pull-down button by operating the input device 209, a list of candidate positions is displayed, and the user can then specify a desired position from the candidate positions included in the list.

[0100] The position of the trocar is set in accordance with the position of the guide 112, and by setting it together with the position of the tumor, a surgical pattern can be selected from among a plurality of surgical patterns. The pulmonary artery resection button 1204 is a toggle button, and by turning this button ON / OFF, the endoscope monitor image 1015-5 and the endoscope monitor image 1015-1 can be switched between for display. In other words, the pulmonary artery resection button 1204 can be used to switch between the endoscope monitor images before and after surgery.

[0101] When the enter button 1203 is pressed, the contents set using the pull-down buttons 1201 , 1202 , and buttons 1204 and 1205 are stored in the external memory 211 .

[0102] In step S813, the CPU 201 synthesizes a character string or an image indicating that the surgery has failed at a specified position in the second mixed reality image as a failure indication. The failure indication may be synthesized on the endoscope monitor image. The CPU 201 then sets the surgical instrument operation flag to OFF.

[0103] Next, an example of the process in step S803 above will be described with reference to the flowchart in FIG.

[0104] In step S901, the CPU 201 acquires the first position and posture and the second position and posture calculated in step S802.

[0105] In step S902, the CPU 201 acquires a human body virtual object from the external memory 211. Before executing step S811, the CPU 201 acquires a pre-operative human body virtual object, and after executing step S811, the CPU 201 acquires a post-operative human body virtual object. Note that the pre-operative human body virtual object / post-operative human body virtual object may be registered in the external memory 211 for each trocar position and tumor position, and the pre-operative human body virtual object / post-operative human body virtual object corresponding to the trocar position and tumor position set in the GUI of Fig. 12 may be used.

[0106] In step S 903 , the CPU 201 acquires the affected area position information stored in the external memory 211 .

[0107] In step S904, the CPU 201 determines whether the hint function is ON. The setting method of the hint function is not limited to a specific setting method.

[0108] As a result of this determination, if the hint function is ON, the process proceeds to step S906; if the hint function is OFF, the process proceeds to step S905; if the hint function is set as "ON only at close range," the process proceeds to step S907.

[0109] In step S905, the CPU 201 places the human body virtual object acquired in step S902 in a placement position and orientation that is preset as the position and orientation for placing the human body virtual object, and generates an image of the placed human body virtual object seen from the second position and orientation as an endoscope monitor image.

[0110] In step S906, the CPU 201 places a hint display (for example, hint display 1104 in FIG. 11), which is a virtual object that highlights the periphery of the affected area, at the position of the affected area included in the affected area position information acquired in step S903, places the human body virtual object acquired in step S902 in a placement position and orientation that is preset as the position and orientation for placing the human body virtual object, and generates an image of the placed human body virtual object and the hint display seen from the second position and orientation as an endoscope monitor image.

[0111] When the affected area hint function is turned on, the affected area is highlighted among multiple organs when a mock endoscope is inserted, making it easier to find the affected area in the simulation and allowing for quicker confirmation of the affected area during actual surgery.

[0112] In step S907, the CPU 201 determines whether the distance between the position of the affected area included in the affected area position information acquired in step S903 and the first position (the tip position of the surgical instrument) is within a second predetermined distance.

[0113] If the result of this determination is that the distance between the position of the affected area included in the affected area position information acquired in step S903 and the first position (the tip position of the surgical instrument) is within the second predetermined distance, the process proceeds to step S908, and if the distance between the position of the affected area included in the affected area position information acquired in step S903 and the first position (the tip position of the surgical instrument) is not within the second predetermined distance, the process proceeds to step S905. Note that the value of the second predetermined distance is set to a value larger than the first predetermined distance, and may be a predetermined value or may be a value input by the user on a setting screen (not shown) or the like.

[0114] In step S908, the CPU 201 generates an image of the human body virtual object and the hint display seen from the second position and posture as an endoscope monitor image, similar to step S906 described above, but changes the hint display according to the distance between the position of the affected area included in the affected area position information acquired in step S903 and the first position (the tip position of the surgical instrument).

[0115] There are various methods for changing the hint display depending on the distance between the position of the affected area contained in the affected area position information acquired in step S903 and the first position (the tip position of the surgical instrument), and the method is not limited to a specific method.

[0116] For example, the color intensity or size of the hint display (circle) may be changed depending on the distance between the position of the affected area included in the affected area position information acquired in step S903 and the first position (the tip position of the surgical instrument).

[0117] Thus, according to this embodiment, an image representing the internal structure of the human body before surgery is generated as an endoscopic monitor image based on a first virtual object including the internal structure of the human body before surgery and a first position and posture of a real object corresponding to the endoscope. When the endoscopic monitor image is output, if the distance between the position of the affected area in the first virtual object and the position of the real object corresponding to the surgical instrument is within a first predetermined distance and the elapsed time since a user operation was performed on the real object corresponding to the surgical instrument exceeds a predetermined time, an image representing the internal structure of the human body after surgery is generated as an endoscopic monitor image based on a second virtual object including the internal structure of the human body after surgery and the first position and posture.

[0118] As a result, according to this embodiment, a simulation of surgery can be smoothly performed using a virtual human body object with a simple configuration. In addition, a simulation of surgical instruments during surgery can be performed, allowing the user to learn the feel of operating the surgical instruments.

[0119] Furthermore, multiple surgical patterns (trocar position and affected area position) can be set, and the position, orientation, and size of the endoscopic monitor image can be freely changed, improving the reproducibility of various surgical simulations.

[0120] Furthermore, the hint function for displaying the affected area makes it easy to perform a surgical simulation, enabling smooth re-creation of procedures during the actual surgery.

[0121] Note that, although this embodiment is described assuming the simulation of thoracoscopic surgery, mainly lung surgery, the present invention can also be used for simulating surgery, such as laparoscopic surgery, which uses an endoscopic camera and surgical instruments (such as a linear stapler).

[0122] [Second embodiment] In the first embodiment, the HMD 101 and the PC 100 are separate devices, but the above-mentioned functions of the PC 100 may be implemented by the HMD 101.

[0123] In the first embodiment, the endoscope monitor image is placed in a virtual space, an image of the placed endoscope monitor image seen from the second position and orientation is generated, and the image is synthesized with the mixed reality image, but the present invention is not limited to this, and the endoscope monitor image may be placed at any position on the mixed reality image. Note that the method of setting the two-dimensional position at which the endoscope monitor image is placed in the mixed reality image and the size of the endoscope monitor image are not limited to a specific setting method.

[0124] In addition, in the first embodiment, the number of HMDs 101 connected to the PC 100 is one, but this is not limited to the above, and two or more HMDs 101 may be connected to the PC 100. In this case, the PC 100 performs the above processing for each HMD 101.

[0125] In addition, the arrangement position and orientation of the endoscope monitor image may not be fixed, but may be dynamically changed. For example, the orientation of the endoscope monitor image may be dynamically changed so that the endoscope monitor image always faces the HMD 101 directly.

[0126] Furthermore, the output destination of the second mixed reality image is not limited to the HMD 101. For example, the CPU 201 may control the video controller 206 to display the second mixed reality image on the display 210.

[0127] As described above, it goes without saying that the object of the present invention can be achieved by supplying a recording medium on which a computer program that realizes the functions of the above-mentioned embodiments is recorded to a system or device, and having the computer (or CPU or MPU) of that system or device read and execute the computer program stored on the recording medium.

[0128] In this case, the computer program itself read out from the recording medium will realize the novel functions of the present invention, and the recording medium storing the computer program will constitute the present invention.

[0129] Examples of recording media for supplying computer programs include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EEPROMs, silicon disks, solid state drives, etc.

[0130] Furthermore, it goes without saying that the functions of the above-mentioned embodiments are not only realized by a computer executing a computer program that has been read out, but also include cases where an OS (operating system) running on the computer performs some or all of the actual processing based on the instructions of the computer program, and the functions of the above-mentioned embodiments are realized through that processing.

[0131] Furthermore, it goes without saying that this also includes cases where a computer program read from a recording medium is written into a memory provided on an expansion board inserted into a computer or a function expansion unit connected to the computer, and then a CPU or the like provided on the expansion board or function expansion unit performs some or all of the actual processing based on the instructions of the code of the computer program, thereby realizing the functions of the above-mentioned embodiments.

[0132] The present invention may be applied to a system consisting of multiple devices, or to an apparatus consisting of a single device. Needless to say, the present invention can also be applied to a case where the present invention is achieved by supplying a computer program to a system or apparatus. In this case, the effect of the present invention can be enjoyed by the system or apparatus by reading a recording medium storing a computer program for achieving the present invention into the system or apparatus.

[0133] Furthermore, by downloading and reading a computer program for achieving the present invention from a server, database, etc. on a network using a communication computer program, the system or device can enjoy the effects of the present invention.

[0134] The numerical values, processing timing, processing order, processing subject, data (information) structure / acquisition method / send destination / send source / storage location, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to these examples.

[0135] In addition, a part or all of the embodiments described above may be used in appropriate combination. In addition, a part or all of the embodiments described above may be used selectively.

[0136] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention.

Claims

1. a generating means for generating an image representing the internal structure of the human body before surgery as an endoscope monitor image based on a first virtual object including the internal structure of the human body before surgery and a real object position and orientation which are the position and orientation of a real object corresponding to the endoscope; an output means for outputting the endoscope monitor image; a notification means for notifying the success or failure of the surgery based on whether or not a relationship between a position of an affected area in the first virtual object and a position of a real object corresponding to a surgical instrument satisfies a predetermined condition after a user operation is performed on the real object corresponding to the surgical instrument in a predetermined state; 13. An image processing device comprising:

2. 2. The image processing device according to claim 1, wherein the predetermined state is a state in which a distance between a position of an affected area in the first virtual object and a position of a real object corresponding to a surgical instrument is within a first predetermined distance.

3. The image processing apparatus according to claim 1 , wherein the predetermined condition is that an elapsed time from when a user performs an operation on a real object corresponding to the surgical instrument exceeds a predetermined time.

4. 2. The image processing apparatus according to claim 1, wherein the generating means generates a composite image by combining an image of real space captured by a head-mounted display device and the endoscope monitor image.

5. 2. The image processing device according to claim 1, wherein the notification means displays an image representing the internal structure of the human body after surgery as an endoscope monitor image based on a second virtual object including the internal structure of the human body after surgery and the position and orientation of the real object.

6. The image processing device according to claim 1, characterized in that the notification means displays an image including a failure display indicating that the surgery has failed when a relationship between a position of the affected area in the first virtual object and a position of a real object corresponding to a surgical instrument no longer satisfies the specified condition after a user operation is performed on the real object corresponding to the surgical instrument in the specified state.

7. 2 . The image processing apparatus according to claim 1 , wherein the generating means acquires, as the physical object position and orientation, a position and orientation of a portion of a physical object corresponding to the endoscope, which corresponds to a camera of the endoscope.

8. 2. An image processing apparatus according to claim 1, wherein the position of the physical object corresponding to the surgical instrument is the position of a portion of the physical object that corresponds to a tip of the surgical instrument.

9. The image processing apparatus according to claim 1 , wherein the image showing the internal structure of the human body before surgery includes an indication of a position of an affected area in the human body before surgery.

10. 10. The image processing device according to claim 9, wherein the generating means changes the display in accordance with a distance between a position of the affected area in the first virtual object and a position of a real object corresponding to a surgical instrument when the distance is within a second predetermined distance.

11. 2. The image processing apparatus according to claim 1, further comprising a means for setting at least one of a position, a posture and a size of the endoscope monitor image.

12. A head mounted display device; The image processing device according to claim 1 , A system comprising:

13. An image processing method performed by an image processing device, comprising: a generation step in which a generation means of the image processing device generates an image representing the internal structure of the human body before surgery as an endoscope monitor image based on a first virtual object including the internal structure of the human body before surgery and a real object position and orientation which are the position and orientation of a real object corresponding to the endoscope; an output step in which an output means of the image processing device outputs the endoscope monitor image; a notification step of notifying the success or failure of the surgery based on whether or not a relationship between a position of an affected area in the first virtual object and a position of a real object corresponding to a surgical instrument satisfies a predetermined condition after a user operation is performed on the real object corresponding to the surgical instrument in a predetermined state by a notification means of the image processing device; 13. An image processing method comprising:

14. A computer program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 11.

Images