Laparoscopic image operation method, laparoscopic image operation system and laparoscopic image operation program

Abstract

To avoid stop of treatment in laparoscopic treatment and improve outcome of a patient.SOLUTION: A laparoscopic image operation method comprises: capturing a video stream of a laparoscopic image of a patient (2) using a laparoscope (12) inserted into the patient (2) during laparoscopic treatment; supplying the captured laparoscopic image to a video processor (14) which is configured to add additional information as an overlay to the captured laparoscopic images; rendering a representation of a 3D model (18) of a target organ or structure using a renderer (16), and merging the rendered representation of the 3D model (18) with the captured laparoscopic image to generate a composite image (20); and displaying the composite image (20) on a monitor (22).SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a laparoscopic image manipulation method, a laparoscopic image manipulation system, and a laparoscopic image manipulation program. [Background technology]

[0002] During laparoscopic surgery, a video camera is inserted into the patient through a trocar and the abdominal cavity is visualized on a 2D, 3D or near 3D laparoscopic video monitor. This minimally invasive approach generally benefits the patient in terms of less trauma, less blood loss and shorter hospital stay compared to traditional open surgical approaches, but at the expense of the surgeon's loss of perspective and loss of opportunity for tissue palpation.

[0003] Perspective is important to understand the spatial relationships of anatomical structures, e.g., the distance of a tumor from a major blood vessel in space, and to determine whether the patient can be operated on via a laparoscopic approach or if an open approach is indeed required. To compensate for the lack of perspective, a 3D model reconstructed from a computed tomography (CT) scan of the patient is often used in the operating room and displayed on a second monitor next to the laparoscopic main monitor. Another approach is to use a 3D printed patient-specific model of the target organ or structure in the operating room. The target structure can be, for example, a blood vessel or vascular structure, a muscle structure, a tendon structure, a cartilage structure, a fascial structure, etc.

[0004] However, when the additional information is displayed on an additional screen or a 3D printed model, the surgeon is forced to lose eye contact with the main surgical situation as shown in the laparoscopic video stream when accessing the additional information. To avoid a dangerous situation, the surgeon usually takes a short break and stops the procedure. However, this disrupts the procedure and leads to an extension of the procedure time. The latter in turn can lead to a worsening of the patient's outcome. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] U.S. Pat. No. 1,083,5344 Summary of the Invention [Problem to be solved by the invention]

[0006] An object of the present disclosure is to provide a laparoscopic image manipulation method, a laparoscopic image manipulation system, and a laparoscopic image manipulation program that can avoid treatment stoppages during laparoscopic treatment and improve patient outcomes. [Means for solving the problem]

[0007] Such an object may be solved by a method of laparoscopic image manipulation comprising: capturing a video stream of laparoscopic images of a patient using a laparoscope inserted in the patient during a laparoscopic procedure, providing the captured laparoscopic images to a video processor configured to add additional information as an overlay on the captured laparoscopic images, rendering a representation of a 3D model of the target organ or structure using a renderer, generating a composite image by merging the rendered representation of the 3D model with the captured laparoscopic images, and displaying the composite image on a monitor, which may be a main surgical monitor.

[0008] The laparoscopic image manipulation method displays a rendering of a 3D model prepared prior to a laparoscopic procedure together with the captured laparoscopic image on a monitor, in particular a main surgical monitor. The main surgical monitor is the primary monitor on which all data and surgical images most relevant to the surgeon tasked with performing the procedure on the patient are displayed. Displaying the rendering of the 3D model within the laparoscopic image on the main surgical monitor means that the surgeon does not have to break eye contact with the laparoscopic image when reviewing the rendering of the 3D model. This eliminates the need for the surgeon to stop the procedure to be able to review the rendering of the 3D model displayed on a separate display or as a physical 3D model of the target organ or structure.

[0009] The monitors, particularly the main surgical monitor, may be conventional displays such as computer monitors, or they may be head-up or visor displays worn by the surgeon over the eyes.

[0010] In one embodiment, at least one of the orientation, size and position of the rendering of the 3D model in the composite image is controlled by a manual controller connected to a renderer. The renderer may be implemented as rendering software running on a video processor or on a separate computer. The manual controller may be a known controller such as a mouse, a trackball, an Xbox® controller, a PlayStation® controller, a Nintendo Switch® controller, a joystick controller, etc. The manual controller may be controlled by the surgeon performing the procedure or an assistant while tracking the orientation of the target organ or structure in the laparoscopic image. Manual manipulation of the orientation and / or size of the rendering of the 3D model allows the surgeon the freedom to evaluate the target organ or structure from various angles that are not necessarily accessible with a laparoscope.

[0011] In an embodiment, information of the individual frames of the video stream of laparoscopic images, in particular the image resolution and / or frame rate, is input to the renderer, which is configured to render a representation of the 3D model according to the input information of the individual frames. By synchronizing the rendered representation of the 3D model, also referred to as the rendering, with the individual frames of the video stream of laparoscopic images, generation of the composite image can proceed in real time and without the need to reprocess the rendering to fit the laparoscopic images.

[0012] For laparoscopic 3D images generated using stereo laparoscopy, it is envisioned that when the model is rendered in 3D and a composite image is formed, the left and right renderings of the 3D model are merged with a larger disparity than the structures in the left and right laparoscopic images. By choosing a larger disparity or displacement, the rendering of the 3D model appears to float above the image of the target organ or structure and surrounding tissues.

[0013] In an embodiment, the 3D model of the target organ or structure is derived from previous CT and / or MRI scan data of the patient. Such previous CT or MRI scans of the patient may be performed in a radiology department of a hospital and processed by the hospital or an external provider to generate a 3D model of the target organ or structure. The 3D model of the target organ or structure may be uploaded to a renderer configured to render a two-dimensional representation of the 3D model according to a selected orientation of the 3D model in space. The 3D model may also be modified in terms of visualization, for example to reduce deformations that occurred during the execution of previous scans.

[0014] This object can also be solved by a laparoscopic image manipulation system including a laparoscope, a video processor, a controller and a monitor, in particular a surgical main monitor, wherein the laparoscope is configured to capture a video stream of laparoscopic images of a patient and to provide the captured laparoscopic images to the video processor, one of the controller, the video processor or a separate computer executes rendering software configured to render a representation of a 3D model of the target organ or structure, the video processor is configured to generate a composite image by merging the rendered representation of the 3D model with the captured laparoscopic image, and the monitor is configured to display the composite image.

[0015] The laparoscopic image manipulation system embodies the same features, characteristics and advantages as the laparoscopic image manipulation method described above, providing the laparoscopic surgeon with a composite image displaying a 3D model of the target organ or structure, prepared from previous CT or MRI scan data within a laparoscopic image of the target organ or structure and surrounding tissue, providing a side-by-side view of the model and the target organ or structure. The surgeon does not need to take his eyes off the actual laparoscopic image, thus avoiding loss of eye contact with the image, including possible loss of orientation in the surgical field, avoiding interruptions to the laparoscopic procedure, ensuring faster laparoscopy and better patient outcomes.

[0016] The laparoscopic image manipulation system may include a manual controller having a data link to at least one of a controller running rendering software, a video processor, and a separate computer, the rendering software being configured to change at least one of the orientation, size, and position of a rendering of a 3D model of a target organ or structure in a composite image in response to signals from the manual controller. Such a manual controller may be a known controller such as a mouse, a trackball, an Xbox® controller, a PlayStation® controller, a joystick controller, etc. The manual controller may be controlled by a surgeon or an assistant performing a procedure while tracking the orientation of the target organ or structure in the laparoscopic image.

[0017] In an embodiment, the laparoscopic image manipulation system may further include a frame grabber configured to capture the laparoscopic video stream frame by frame and generate a composite image by merging a rendered representation of the 3D model with the captured laparoscopic image frame by frame.

[0018] In a further embodiment, the system components, particularly the controller, the video processor and the separate computer, are configured to perform a laparoscopic image manipulation method according to the above disclosure.

[0019] The above object may also be achieved by a laparoscopic image manipulation program stored on a non-volatile medium, which is specifically designed to carry out the steps of the above-mentioned laparoscopic image manipulation method when executed on a system component of a laparoscopic image manipulation system according to the above disclosure. Such a laparoscopic image manipulation program embodies the software aspect of the present invention. Different portions of the computer software may be stored and executed on different components of the laparoscopic image manipulation system according to their respective functions.

[0020] Further features of the invention will become apparent from the description of the embodiments according to the invention together with the claims and the included drawings. The embodiments according to the invention can realize individual features or a combination of several features.

[0021] The present invention is described below on the basis of exemplary embodiments, without limiting the general spirit of the invention, and explicit reference is made to the drawings for disclosure of all details according to the invention not described in more detail in the text. [Brief description of the drawings]

[0022] [Figure 1] FIG. 1 illustrates one embodiment of a laparoscopic image manipulation system. [Diagram 2] 13A-13C illustrate an embodiment of a composite image according to an embodiment of a method for laparoscopic image manipulation. [Diagram 3] FIG. 13 illustrates a further embodiment of a composite image. [Figure 4] 1 illustrates an embodiment of a method for laparoscopic image manipulation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] In the drawings, elements of the same or similar type or corresponding parts are given the same reference numbers so as not to have to reintroduce the items.

[0024] 1 illustrates one embodiment of a laparoscopic image manipulation system 10. The laparoscopic image manipulation system 10 is configured for use in laparoscopic examination of a patient 2 with a laparoscope 12, which is provided with either an imaging sensor or a camera head (not shown) at the tip of the proximal end of the laparoscope. Images of a video stream generated by the laparoscope 12 are provided to a video processor 14. The output of the video processor 14 is provided to a rendering computer 16 configured to render a representation of the 3D model 18 at a particular orientation and size using the input of the 3D model 18, which is then merged by overlay or other means with the laparoscopic image, either in the video processor 14 or the rendering computer 16 or in a central controller (not shown), thereby creating a composite image 20, which is displayed on a monitor, which may be a main surgical monitor 22. The laparoscopic image manipulation system 10 may also include a frame grabber device (not shown) and / or a manual controller 17. The manual controller 17 may be used to control the orientation, size and / or position of the rendering of the 3D model 18 in the composite image. The manual controller 17 may be controlled by the surgeon performing the laparoscopy or by the surgeon's assistant.

[0025] FIG. 2 shows an embodiment of a composite image according to an embodiment of a laparoscopic image manipulation method. The composite image 20 is essentially the same as that shown in FIG. 1, but with a different color representation as grayscale hues to better emphasize the present disclosure. The composite image 20 includes a laparoscopic image 24 provided by the laparoscope 12 embedded in a frame that provides additional information and control icons 28. In the upper left corner of the laparoscopic image 24, a rendering of a 3D model of a target organ or structure 30 is superimposed on the laparoscopic image 24. The target organ or structure 30, as well as the rendering 26 of the 3D model of the target organ or structure, are provided with assistant lines that the surgeon's assistant or surgeon can use to manipulate the orientation of the rendering to match the orientation of the actual target organ or structure 30 in the laparoscopic image 24.

[0026] As can be seen in FIG. 2, the 3D model rendering 26 of the target organ or structure is clearly different from the laparoscopic image 24. The rendering 26 is rendered at a different scale than the target organ or structure 30 in the laparoscopic image 24 and appears to float above the organ seen in the laparoscopic image 24. However, the scale may be changed using the manual controller 17 to match the target organ or structure in the video image or even larger. The surface texture may also be intentionally made to have a more artificial appearance to make it immediately recognizable as the model rendering 26 of the target organ or structure 30. The purpose is to give the surgeon a very clear and rapid means of orientation regarding the target organ or structure 30 to provide the surgeon with information obtained from a previous CT or MRI scan about the target organ or structure 30 and any lesions, tumors or other abnormalities to be addressed during the laparoscopy.

[0027] In the case of laparoscopic 3D images generated using stereo laparoscopy, it is envisioned that when the model is rendered in 3D to form a composite image, the rendering of the 3D model will actually appear to float above the image of the target organ or structure 30 and surrounding tissues.

[0028] Using the manual controller 17, the surgeon or assistant may adjust the orientation, size and / or location of the 3D model rendering 26 within the composite image. The selection of the orientation of the 3D model rendering 26 may be informed by a desire to view the 3D model from a perspective not available to the laparoscope in the laparoscopic image 24, or by a desire to have the orientation of the 3D model rendering 26 match the orientation of the target organ or structure 30 in the laparoscopic image 24. The selection of the size of the 3D model rendering 26 may be informed by a desire to view details by magnifying the 3D model rendering 26, or by a desire to minimize the obstruction of the laparoscopic image 24 by reducing the magnification of the rendering 26 after inspection of the details is completed. The location of the 3D model rendering 26 within the laparoscopic image 24 may be selected to minimize obstruction of the laparoscopic image 24, and in particular the target organ or structure 30 therein. However, the rendering may also be made semi-transparent and superimposed directly on top of the original target organ or structure 30.

[0029] 3 illustrates a further embodiment of a composite image in which a rendering of a 3D model is superimposed on a laparoscopic image. As in FIG. 2, the 3D model of the target organ or structure is displayed directly on the main surgical monitor 22 and can be used in a variety of ways to assist the surgeon in intraoperative orientation. For example, it can be used to visualize subsurface structures that are not visible in the laparoscopic video image. Such subsurface structures are shown in the screenshot of FIG. 3.

[0030] 4 shows an embodiment of a laparoscopic image manipulation method. The basic method includes capturing a video stream of laparoscopic images (step S10) and feeding the laparoscopic images to a video processor (step S20). In parallel, a rendering of a 3D model of the target organ or structure, which may include subsurface structures such as blood vessels, is generated (step S30), and a composite image is generated by merging the rendering of the 3D model with the previously captured laparoscopic images (step S40). The composite image and additional information and / or control icons are then displayed on a monitor, in particular the surgical main monitor (step S50).

[0031] The rendering of the 3D model in step S30 may have several inputs. This may include retrieving 3D model information prepared using a previous CT or MRI scan of the target organ or structure and processed to be useful for rendering (step S32). The 3D model information may be input once at the start of the laparoscopy. By extracting frame information from the laparoscopic video stream on a frame-by-frame basis (step S22), synchronization between the rendering of the 3D model and the laparoscopic image with which the rendering is merged may be improved. Furthermore, the orientation, size and / or position of the rendering of the 3D model within the laparoscopic image may be manually controlled using the manual controller 17 as described above (step S34).

[0032] All specified features, including those obtained only from the drawings, and individual features disclosed in combination with other features, are considered to be important to the invention both alone and in combination. The embodiments according to the invention can be realized by individual features or by a combination of several features. Features combined with the expression "particularly" or "especially" should be treated as preferred embodiments. [Explanation of symbols]

[0033] 2 patients 10 Laparoscopic image manipulation system 12 Laparoscopy 14 Video Processor 16 Rendering Computer 17 Manual Controller 18 3D models 20 Composite Image 22 Main monitor for surgery 24 Laparoscopic Images 26 Rendering 3D Models 28 Additional Information 30 Target organs or structures

Claims

1. 1. A method for laparoscopic image manipulation, comprising: capturing a video stream of laparoscopic images of a patient using a laparoscope inserted into the patient during a laparoscopic procedure; providing the captured laparoscopic image to a video processor configured to add additional information as an overlay on the captured laparoscopic image; Rendering a representation of a 3D model of a target organ or structure using a renderer and generating a composite image by merging the rendered representation of the 3D model with the captured laparoscopic image; displaying the composite image on a monitor; 23. A method for laparoscopic image manipulation comprising:

2. The method of claim 1 , wherein at least one of an orientation, a size, and a position of the rendered representation of the 3D model in the composite image is controlled by a manual controller connected to the renderer.

3. The laparoscopic image manipulation method of claim 1, wherein image resolution and / or frame rate information of individual frames of the video stream of the laparoscopic image is input to the renderer, and the renderer is configured to render a representation of the 3D model according to the input information of the individual frames.

4. The method of laparoscopic image manipulation according to claim 1 , wherein the 3D model of the target organ or structure is derived from previous CT and / or MRI scan data of the patient.

5. 1. A laparoscopic image manipulation system comprising a laparoscope, a video processor, a controller, and a monitor, the laparoscope is configured to capture a video stream of laparoscopic images of a patient and provide the captured laparoscopic images to the video processor; one of the controller, the video processor, or a separate computer executes rendering software configured to render a representation of a 3D model of a target organ or structure; the video processor is configured to generate a composite image by merging a rendered representation of the 3D model with the captured laparoscopic image; The monitor is configured to display the composite image.

6. 6. The laparoscopic image manipulation system of claim 5, further comprising a manual controller having a data link to at least one of the controller executing the rendering software, the video processor, and the separate computer, the rendering software being configured to modify at least one of an orientation, a size, and a position of a representation of the rendered 3D model of the target organ or structure in the composite image in response to a signal from the manual controller.

7. 6. The laparoscopic image manipulation system of claim 5, further comprising a frame grabber configured to capture a video stream of the laparoscopic images frame by frame and generate the composite image by merging a rendered representation of the 3D model with the laparoscopic images captured frame by frame.

8. The laparoscopic image manipulation system of claim 5 , wherein the system components, the controller, the video processor, and the separate computer, are configured to perform the laparoscopic image manipulation method of claim 1 .

9. A laparoscopic image manipulation program stored on a non-volatile medium, comprising: A laparoscopic image manipulation program that causes a computer to execute the laparoscopic image manipulation method according to claim 1.

Images