3D visualization enhancements for depth perception and collision avoidance in endoscopic systems

Abstract

A series of images is obtained from the endoscope. A three-dimensional reconstruction is performed on the series of images to reconstruct the anatomical structure shown in the series of images. Graphics such as meshes are rendered on the series of images based on the three-dimensional reconstruction, resulting in an augmented endoscope video feed being shown on a display.

Description

Technical Field

[0001] This disclosure relates generally to the field of surgical robotics, and more specifically to generating visual enhancements for depth perception or collision avoidance. Background Technology

[0002] Minimally invasive surgery (MIS), such as laparoscopic surgery, involves techniques designed to minimize tissue damage during surgical procedures. For example, a laparoscopic procedure typically involves making multiple small incisions inside the patient (e.g., in the abdomen) and introducing one or more instruments and at least one endoscopic camera through these incisions. The surgical procedure is then performed with the aid of visualization provided by the camera, using the introduced instruments.

[0003] Generally, medical interventions (MIS) offer multiple beneficial effects, such as reducing patient scarring, alleviating patient pain, shortening patient recovery time, and reducing medical costs associated with patient recovery. In some implementations, MIS can be performed using a surgical robotic system, which includes one or more robotic arms for manipulating surgical tools based on commands from an operator. For example, the operator can provide commands for manipulating the surgical tools while viewing images provided by a camera and displayed to the user on a monitor.

[0004] As described above, a minimally invasive surgical instrument (MIS) may include inserting an endoscope into the patient to provide images of the patient's internal anatomy during surgery. Minimally invasive surgical instruments are inserted into the patient within the endoscopic view. The endoscopic view allows the surgeon to see the instruments, enabling them to be moved and manipulated, for example, to perform cutting, grasping, or suturing.

[0005] During remote operations, current 3D endoscopic views can provide a sense of depth and distance using binocular cues. This allows users to judge the relative positions of the workspace and tools. However, this system can become ineffective. Depending on the endoscopic viewpoint, lighting conditions, and texture in the workspace, users may struggle to measure distances to objects and tools. This presents a challenge for both new and experienced users. This problem can be exacerbated when using a standard 2D display due to the lack of spatial cues. Whether displayed on a 3D stereoscopic display or a standard 2D display, the uncertainty in distances shown in the endoscopic view can slow down surgical procedures—users may perform procedures more slowly to maintain accuracy or reduce the risk of unwanted contact between tools or between tools and patient anatomy. Summary of the Invention

[0006] This system and method can enhance endoscopic visualization during remote or manually performed procedures. It can present graphics on existing endoscopic views, including patterns (e.g., 3D grid lines or mesh) that show or enhance the visibility of the shapes and contours of surfaces detected in the endoscopic image feed. The graphics can include a geometric reference for the location of surgical tools. This reference helps the observer discern the position of the tools relative to the patient's tissues, organs, and other internal anatomy. This system can enhance the user's depth perception of the endoscopic scene and the relative distance between the tools and the environment. The endoscopic scene is the environment captured by the endoscope's image sensors. For example, when inserted into a patient, the endoscopic scene can include the patient's anatomy, such as tissues, organs, muscles, bones, etc.

[0007] In some implementations, a method for improving depth perception of endoscopic views is performed. This method includes acquiring a series of images obtained from the endoscope; in other words, an endoscopic video feed. Three-dimensional reconstruction is performed on the series of images to reconstruct the anatomical structures shown in the series of images. Based on the 3D reconstruction, graphics (e.g., patterns, grids, etc.) are rendered on the series of images, resulting in an enhanced endoscopic video feed being displayed on a monitor. This method can be performed using a surgical robotic system and / or manual minimally invasive surgical tools. In this way, user performance can be improved and unintentional collisions, such as between tools or between tools and internal anatomical structures, can be reduced.

[0008] Graphical visualization enhancements can be displayed on a 3D display (e.g., a stereoscopic display) that is presented above or "on top of" the endoscopic view. 3D scene reconstructions can also be utilized in virtual reality or augmented reality settings (e.g., with head-mounted displays) for remote operation, simulation, or training scenarios. Additionally or alternatively, enhancements (e.g., geometric mesh lines and positional references) can also be implemented on a standard 2D screen to provide monocular cues of the 3D scene. Users can fully or partially enable or disable graphical overlays via user input devices such as handheld controllers, graphical user interfaces, voice recognition, or other equivalent input devices.

[0009] The foregoing summary does not include an exhaustive list of all embodiments of this disclosure. It is contemplated that this disclosure encompasses all suitable combinations of the various embodiments outlined above, as well as those described in the detailed description below and specifically pointed out in the claims section, and the systems and methods to be practiced. Some combinations may have specific advantages not specifically stated. Attached Figure Description

[0010] Figure 1 Examples of surgical robotic systems in the operating room according to some implementation schemes are shown.

[0011] Figure 2 The process for providing enhanced endoscopic video feed is illustrated according to some implementation schemes.

[0012] Figure 3 Examples illustrate systems for providing enhanced endoscopic video feeds according to some implementation schemes.

[0013] Figure 4 Examples illustrate enhanced endoscope feeding according to some implementation schemes.

[0014] Figure 5 An example illustrates an enhanced endoscopic feed with a graphic indicating the position of one or more tools, according to some implementation schemes.

[0015] Figure 6 and Figure 7 An enhanced endoscope feed with a warning system is shown according to some implementation schemes. Detailed Implementation

[0016] Non-limiting examples of various embodiments and variations of the present invention are described herein and illustrated in the accompanying drawings.

[0017] See Figure 1 This is a drawing view of an exemplary surgical robotic system 1 in a surgical setting. System 1 includes a user console 2, a control tower 3, and one or more surgical robotic arms 4 at a surgical robotic platform 5 (e.g., a table, bed, etc.). The arms 4 can be mounted to the table or bed where the patient lies, such as... Figure 1 As shown in the examples, they can also be mounted on a trolley separate from the table or bed. System 1 can incorporate any number of devices, tools, or accessories for performing surgery on patient 6. For example, system 1 may include one or more surgical tools 7 for performing surgical procedures. Surgical tool 7 may be an end effector attached to the distal end of surgical arm 4 for performing surgical procedures.

[0018] Each surgical tool 7 can be manually manipulated, robotically manipulated, or both during surgery. For example, surgical tool 7 can be a tool for accessing, viewing, or manipulating the internal anatomy of patient 6. In one aspect, surgical tool 7 is a gripper capable of grasping the patient's tissues. Surgical tool 7 can be configured to be manually controlled by bedside operator 8, robotically controlled via actuated movement of its attached surgical robotic arm 4, or both. The robotic arm 4 is shown as table-mounted, but in other configurations, arm 4 can be mounted to a trolley, ceiling, or sidewall, or to another suitable structural support.

[0019] A remote operator 9 (such as a surgeon or other human operator) can use the user console 2 to remotely manipulate the arm 4 and its attached surgical instruments 7, referred to herein as remote manipulation. The user console 2 may be located in the same operating room as the rest of the system 1, such as... Figure 1 As shown. However, in other environments, the user console 2 may be located in an adjacent or nearby room, or it may be located in a remote location, such as in a different building, city, or country. The user console 2 may include a seat 10, foot controls 13, one or more handheld user input devices (UIDs) 14, and at least one user display 15 configured to display a view, for example, of a surgical site within a patient 6. In the exemplary user console 2, a remote operator 9 sits in the seat 10 and views the user display 15 while manipulating the foot controls 13 and the handheld UID 14 to remotely control the arm 4 and the surgical instruments 7 mounted on the distal end of the arm 4.

[0020] In some variations, the bedside operator 8 can operate the system 1 in a "bedside" mode, where the bedside operator 8 (the user) is positioned to one side of the patient 6 and simultaneously manipulates robotically driven tools (end-effectors attached to arm 4), holding a handheld UID 14 in one hand and a manual laparoscopic tool in the other. For example, the bedside operator's left hand can manipulate the handheld UID to control the robotically driven tools, while the bedside operator's right hand can manipulate the manual laparoscopic tool. In this particular variation of system 1, the bedside operator 8 can perform both robot-assisted minimally invasive surgery and manual laparoscopic surgery on the patient 6.

[0021] During the exemplary procedure (surgical operation), patient 6 is prepared for surgery and aseptically covered with a sterile drape to administer anesthesia. Initial access to the surgical site can be manually performed (to facilitate access to the surgical site) while the arms of robotic system 1 are in a retracted or withdrawn configuration. Once access is complete, initial positioning or preparation of robotic system 1, including its arms 4, can be performed. The surgery then continues, with remote operator 9 at user console 2 using foot controls 13 and UID 14 to manipulate various end effectors and, possibly, imaging systems to perform the surgery. Artificial assistance can also be provided at the operating table or surgical table by a bedside person (e.g., bedside operator 8) wearing sterile surgical gowns, who can perform tasks on one or more arms of robotic arms 4, such as tissue retraction, manual repositioning, and tool changes. Non-sterilized personnel may also be present to assist remote operator 9 at user console 2. When a procedure or surgical operation is completed, System 1 and User Console 2 can be configured or set to a certain state to facilitate the completion of postoperative procedures, such as cleaning or disinfection, and the input or printing of health records via User Console 2.

[0022] In one embodiment, the remote operator 9 holds and moves UID 14 to provide input commands, thereby moving the robotic arm actuator 17 in the robotic system 1. UID 14 may be communicatively coupled to the rest of the robotic system 1, for example, via a console computer system 16. UID 14 may generate spatial state signals corresponding to the movement of UID 14, such as the position and orientation of the UID's handheld housing, and the spatial state signals may be input signals for controlling the movement of the robotic arm actuator 17. The robotic system 1 may use control signals derived from the spatial state signals to control the proportional movement of the actuator 17. In one embodiment, a console processor of the console computer system 16 receives the spatial state signals and generates corresponding control signals. Based on these control signals controlling how the actuator 17 is energized to move a segment or connector of the arm 4, the movement of a corresponding surgical tool attached to the arm may simulate the movement of UID 14. Similarly, the interaction between the remote operator 9 and UID 14 may generate, for example, a clamping control signal that causes the jaws of the gripper of the surgical tool 7 to close and clamp the tissue of the patient 6.

[0023] The surgical robot system 1 may include a plurality of UIDs 14, wherein a corresponding control signal is generated for each UID that controls the actuators and surgical instruments (end-effectors) of a respective arm 4. For example, a remote operator 9 may move a first UID 14 to control the movement of an actuator 17 located in the left robotic arm, wherein the actuator responds by moving links, gears, etc. in the arm 4. Similarly, movement of a second UID 14 by the remote operator 9 controls the movement of another actuator 17, which in turn moves other links, gears, etc. of the robot system 1. The robot system 1 may include a right arm 4 fixed to a bed or table on the right side of the patient, and a left arm 4 located on the left side of the patient. The actuators 17 may include one or more motors controlled to drive the joints of the arm 4 to rotate, for example, relative to the patient, changing the orientation of the endoscope or gripper of the surgical instrument 7 attached to the arm. The movement of a plurality of actuators 17 in the same arm 4 may be controlled by spatial state signals generated from a particular UID 14. The UID 14 may also control the movement of the corresponding surgical instrument gripper. For example, each UID 14 can generate a corresponding gripping signal to control the movement of an actuator (e.g., a linear actuator) that opens or closes the jaws of a gripper at the distal end of the surgical tool 7 to grip tissue in the patient 6.

[0024] In some respects, communication between platform 5 and user console 2 can be achieved via control tower 3, which translates user commands received from user console 2 (and more specifically from console computer system 16) into robot control commands transmitted to arm 4 on robot platform 5. Control tower 3 can also transmit status and feedback from platform 5 back to user console 2. The communication connection between robot platform 5, user console 2, and control tower 3 can be via wired and / or wireless links, using any suitable data communication protocol from a variety of data communication protocols. Any wired connection can optionally be integrated into the floor and / or walls or ceiling of the operating room. Robot system 1 can provide video output to one or more displays, including displays within the operating room and remote displays accessible via the Internet or other networks. Video output (video feed) can also be encrypted to ensure privacy, and all or part of the video output can be stored on a server or electronic healthcare record system.

[0025] exist Figure 2 The diagram illustrates a method or process 40 for improving depth perception of an endoscope feed. In operation 41, this process includes acquiring a series of images from the endoscope. The endoscope may have a single image sensor or a stereo camera (with two or more lenses and associated image sensors). As described in other sections, if structured light is used for 3D reconstruction, the endoscope may include a light source emitting structured light.

[0026] In operation 42, the process includes performing three-dimensional reconstruction on a series of images to reconstruct the anatomical structures shown in the series of images. One or more techniques, such as structured lighting, machine learning, and / or stereo reconstruction, can be used to detect the shape or surface of objects, such as tools and anatomical structures, captured in the series of images.

[0027] In operation 43, the process includes rendering graphics on a series of images based on 3D reconstruction, resulting in an enhanced endoscopic video feed being displayed on a monitor. For example, a pattern such as a grid can be projected onto one or more detection surfaces of anatomical structures determined based on the 3D reconstruction.

[0028] This process can be performed repeatedly and in real time (e.g., when a series of images are captured by the endoscope), thereby providing improved depth perception and reducing the risk of collisions during the execution of surgical procedures (or their simulation). In some embodiments, the process can be activated and deactivated by user input. In some embodiments, the process can be automatically activated based on sensed activity, such as sensed changes in light or movement of the endoscope.

[0029] In some implementations, the method can be performed using hand tools, such as a manually controlled endoscope and other manually controlled surgical instruments, for example, inserted into the patient via a cannula. Additionally or alternatively, the method can be performed using a surgical robotic system (such as reference...). Figure 1 The system described is used to execute.

[0030] exist Figure 3 The image shows a system 58 for enhancing endoscopic video feeds. An endoscope 60 has one or more image sensors that generate video feeds (image sequences). When the endoscope is inserted into a patient, the patient's internal anatomy is displayed in the image sequence. Internal anatomy may include tissues, organs, veins, capillaries, arteries, muscles, etc. The images may also include one or more surgical tools (e.g., endoscopes, scalpels, grippers, or needles).

[0031] The graphics enhancement processor 62 performs three-dimensional reconstruction on the endoscopic images to detect anatomical structures as well as the surface and shape of objects such as surgical tools. In some embodiments, such a processor can be integrated as... Figure 1 The surgical robotic system shown may be integrated, for example, with the user console 2 or control tower 3, or as a separate, stand-alone computing device. Different techniques can be used to perform 3D reconstruction, such as those described below. It should be noted that other 3D reconstruction methods not discussed in this disclosure may be implemented to reconstruct endoscopic scenes without departing from the scope of this disclosure.

[0032] In some implementations, 3D reconstruction involves analyzing structured light captured in a series of images to reconstruct anatomical structures. For example, a projector can illuminate a scene captured by an endoscope with a 2D pattern, which may have spatially varying intensity patterns. The projector can be integrated with an endoscope or a separate tool inserted into the patient. Surfaces on which the light pattern falls will alter the shape of the light pattern detected by the endoscopic camera. This structured light falling on the surface can be analyzed to detect the shape of surfaces in the scene, thereby reconstructing the patient's internal anatomy and the 3D surface shapes of objects such as tools present in the scene.

[0033] In some implementations, the endoscope may include a stereo camera having at least two lenses and corresponding image sensors at different advantageous locations. Three-dimensional reconstruction can be performed by establishing stereo correspondences between images captured by each image sensor of the stereo camera to reconstruct the surface shape of the patient's internal anatomy in three dimensions. Similarly, objects such as tools present in a scene can be reconstructed in the same manner.

[0034] For example, known computer vision algorithms can be applied to image streams from stereo cameras in endoscopes. When two or more cameras in an endoscope acquire images from different vantage points, binocular stereo vision of the endoscopic images can be utilized. Corresponding feature points (e.g., common markers or “features” captured in the two image streams) can be extracted from the two image streams for reconstruction. The 3D positions of these feature points can be calculated based on the parallax of the images and the geometric relationship between the two viewpoints, thereby establishing and using stereo correspondences between the image streams to reconstruct the anatomical structures and objects captured in the images.

[0035] In some implementation schemes, Figure 1 The surgical robotic system may include this endoscope with a stereo camera. The endoscope can be operated manually or attached as a tool 7 to one or more surgical robotic arms 4.

[0036] In some implementations, the relative position and orientation of the tools with respect to the endoscopic view can be obtained from geometric calculations of a surgical robotic arm and tool actuators that cause movement within one or more surgical tools shown in a series of images. For example, refer to... Figure 3 Position information 66 of the tools shown in the endoscope feed can be obtained from motor position and sensors that influence, encode, and / or sense the position of the surgical robotic arm and the tools attached to the distal ends of those arms. Telemetry describing the position (e.g., engagement values) of the surgical robotic arm and the attached tools can be obtained, which can be transformed using kinematic data of the surgical robotic arm to determine the three-dimensional orientation and position data of the surgical robotic arm and tools. In some embodiments, the position information may be obtained from a user console, control tower, or other sources. Figure 1 Other components described are obtained.

[0037] Position information 66 obtained from the surgical robotic system can be mapped to the endoscopic view to improve the accuracy of 3D reconstruction of the endoscopic scene. The system can compare the derived positions of tools and anatomical structures to evaluate and improve the 3D reconstruction by providing confirmatory or contradictory data points. Additionally or alternatively, tool positions can be determined by processing the endoscopic feed using computer vision algorithms, by identifying tools in the endoscopic feed, and / or by other 3D reconstruction methods known or described in this disclosure.

[0038] Reference Figure 3In some implementations, the 3D reconstruction performed at the graphics enhancement processor 62 includes applying a machine learning model to a series of images to reconstruct anatomical structures. The machine learning model may include artificial neural networks, such as convolutional neural networks, feedforward neural networks, or recurrent neural networks. Training data can be used to train the model to detect and reconstruct 3D surfaces present in the endoscopic scene. The trained neural network can map information from the endoscopic images to the underlying 3D shape.

[0039] In some implementations, 3D reconstruction includes estimating the geometry of the environment based on 3D registration (e.g., point set registration) and reconstruction using endoscopic images. Additional geometric information from scans of other sensors (e.g., MRI or CT scans) can also be used to improve the accuracy of 3D reconstruction.

[0040] The graphics enhancement processor 62 renders graphics on a series of images based on 3D reconstruction, resulting in an enhanced endoscopic video feed being displayed on the display 64. For example, the graphics may include patterns projected onto one or more examination surfaces of an anatomical structure, such as grids (also described as meshes), lines, dots, or polygons. Furthermore, although shown as rectangles in the figures, grids may include other shapes, such as triangles and other polygons. The display may include a stereoscopic display, a 2D display, and / or a head-mounted display, such as a virtual reality or augmented reality head-mounted device.

[0041] In some implementations, the display and Figure 1 The surgical robotic system shown integrates features such as a display 15 on a user console. In some embodiments, this display is a stereoscopic display. In some embodiments, the stereoscopic display is head-mounted. This display facilitates the execution of remote surgery. Additionally or alternatively, the display may exist on the tableside as a freestanding 2D display or as a head-mounted device to help guide tools manually controlled by the bedside operator 8.

[0042] Figure 4 An example of enhanced endoscopic feeding is shown. A mesh or grid is presented on the detection surface (e.g., an object or anatomical structure) in the endoscopic scene. In this way, the visibility of the shape and location of the anatomical structure's surface is improved.

[0043] Figure 5Another example of enhanced endoscopic feed is shown. In this enhanced endoscopic feed, the graphics include one or more lines indicating the position of a surgical instrument captured in a series of images. For example, lines (e.g., dashed lines) may indicate one or more planes originating from the axis of the tool. Other lines (e.g., thick solid lines) indicate the intersections of these planes with the surfaces of anatomical structures. These graphics serve as a geometric reference indicating the distance between the tool and the anatomical structure along the planes. The pattern projected onto the surface of the user's anatomical structure further helps the user discern the distance between the tool and the anatomical structure relative to the lines of intersection. Thus, depth perception can be improved, and the risk of collisions can be reduced.

[0044] In some implementations, users can achieve location references, such as quantitative measurements of the location, orientation, or distance of any element of the endoscopic scene, the patient's anatomy, and the tools. Digital text can be displayed alongside the features. For example, such as... Figure 6 As shown, text or graphics can be presented in the enhanced endoscopy feed, indicating the distance between two tools (e.g., the total shortest distance between end effectors, tool axes, and / or tools). Similarly, the relative positions of objects in a scene can be presented as text in the enhanced endoscopy feed.

[0045] In some implementation schemes, such as Figure 6 As shown, visual or audio warnings can be implemented for the potential risk of collisions between various elements in the scene. For example, if an tool is determined to be within a threshold proximity to another tool, a visual or audio warning is given.

[0046] Similarly, such as Figure 7 As shown, text or graphics can indicate the distance between a tool and an anatomical structure captured in a series of images. Organs or regions of the anatomical structure can be identified (e.g., through user input or configurable settings) as "restricted" or "of interest," such that the distance between the display tool and the region of interest is determined. If the tool is determined to be within a threshold proximity to a restricted or "of interest" region, the system can provide a visual or audio warning. In some aspects, a warning can be issued if the tool is inactive and determined to be within a threshold proximity to an organ or designated region. These thresholds can be specified by various means, such as being configurable as a setting, specified via user input, and / or hard-coded in programmable memory.

[0047] In some implementations, based on 3D reconstruction, computer vision algorithms, and / or positional information of surgical tools received from a surgical robotic system, the system can determine that one of the aforementioned thresholds has been met. In this case, a text warning may be flashed to the display and / or an audible warning may be provided using a speaker, indicating, for example, that the tools are within an "x" distance of each other. In some cases, lines or other graphics showing the shortest paths between tools (or between a tool and an anatomical structure) may be presented. This can inform the user how to move the tools to increase separation.

[0048] The various embodiments and components described herein can be embodied, at least in part, in software. That is, the process can be implemented by a processor executing a sequence of instructions contained in a storage medium, such as a non-transitory machine-readable storage medium (e.g., DRAM or flash memory). In various embodiments, hard-wired circuitry can be used in combination with software instructions to implement the techniques described herein. Therefore, the techniques are not limited to any particular combination of hardware circuitry and software, or any particular source of instructions executed by the audio processing system.

[0049] In this specification, certain terms are used to describe the characteristics of various embodiments. For example, in some cases, the terms “module,” “processor,” “unit,” “model,” “system,” “device,” and “component” refer to hardware and / or software configured to perform one or more processes or functions. For example, examples of “hardware” include, but are not limited to, integrated circuits such as processors (e.g., digital signal processors, microprocessors, application-specific integrated circuits, microcontrollers, etc.). Therefore, as those skilled in the art will understand, different combinations of hardware and / or software can be implemented to perform the processes or functions described by the foregoing terms. Of course, hardware may alternatively be implemented as a finite state machine or even combinational logic. Examples of “software” include application programs, applets, routines, or even executable code in the form of a series of instructions. As mentioned above, software can be stored on any type of machine-readable medium.

[0050] For purposes of explanation, the foregoing description uses specific nomenclature to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that specific details are not required to practice the invention. The foregoing description of specific embodiments of the invention has been provided for illustrative and descriptive purposes. These are not intended to be exhaustive or to limit the invention to the specific forms disclosed; various modifications and alterations can be made to this disclosure in light of the foregoing teachings. The embodiments were chosen and described in order to best explain the principles of the invention and its practical application, thereby enabling others skilled in the art to best utilize the invention and its various embodiments with various modifications suitable for the contemplated particular uses.

Claims

1. A method for improving depth perception of endoscopic views, comprising: Obtain a series of images of internal anatomical structures obtained from an endoscope; Perform three-dimensional reconstruction on the series of images to reconstruct the surface shape of a portion of the internal anatomical structure shown in the series of images; Based on the three-dimensional reconstruction rendering graphics, the graphics include patterns provided on the surface shape of the portion of the internal anatomy shown on the series of images, thereby resulting in enhanced endoscopic video feed; as well as The enhanced endoscopic video feed is displayed on the monitor.

2. The method according to claim 1, wherein, The pattern includes at least one of grids, lines, and dots.

3. The method according to claim 1, wherein, The graphic includes one or more lines indicating the location of the surgical instruments captured in the series of images.

4. The method according to claim 1, wherein, The graphics indicate at least one of the following: the distance between two or more tools, or the distance between a tool and the internal anatomical structure captured in the series of images.

5. The method of claim 1, further comprising providing a visual warning or an audio warning via a speaker on the display if the tool is determined to be within a threshold of (a) another tool or (b) a designated area of ​​the internal anatomy.

6. The method according to claim 1, wherein, The endoscope includes a stereo camera having at least two lenses and corresponding image sensors at different advantageous positions, and the three-dimensional reconstruction includes establishing a stereo correspondence between the images from the stereo camera to reconstruct the surface shape of a portion of the internal anatomy.

7. The method according to claim 1, wherein, Performing 3D reconstruction involves obtaining positional information from a surgical robotic arm or tool actuator to determine the position of one or more surgical tools, the surgical robotic arm or the tool actuator causing movement in one or more surgical tools shown in the series of images.

8. The method according to claim 1, wherein, The three-dimensional reconstruction includes analyzing structured light captured in the series of images to reconstruct the surface shape of a portion of the internal anatomy.

9. The method according to claim 1, wherein, The 3D reconstruction includes applying a machine learning model to the series of images to reconstruct the surface shape of a portion of the internal anatomical structure.

10. The method according to claim 1, wherein, The series of images includes one or more manually operated surgical tools.

11. The method according to claim 1, wherein, The display is a stereoscopic display.

12. A system for improving depth perception of endoscopic views, comprising: One or more surgical robotic arms; Endoscope; monitor; as well as The processor is configured to perform the following operations: Three-dimensional reconstruction is performed on a series of images obtained from the endoscope, including the patient's internal anatomy, to reconstruct the surface shape of a portion of the internal anatomy shown in the series of images; as well as The graphics include patterns projected onto one or more detection surfaces on the series of images, representing the surface shape of a portion of the internal anatomy determined based on the three-dimensional reconstruction, thereby resulting in enhanced endoscopic video. as well as The enhanced endoscopic video is displayed on the monitor.

13. The system according to claim 12, wherein, Performing 3D reconstruction includes obtaining position information from the one or more surgical robotic arms or tool actuators to determine the position of the one or more surgical tools, the one or more surgical robotic arms or the tool actuators causing movement in the one or more surgical tools attached to the one or more surgical robotic arms and shown in the series of images.

14. The system according to claim 12, wherein, The pattern includes at least one of grids, lines, and dots.

15. The system according to claim 12, wherein, The graphic includes one or more lines showing the location of surgical tools coupled to the one or more surgical robotic arms captured in the series of images.

16. The system according to claim 12, wherein, The graphics indicate at least one of the following: the distance between two or more surgical instruments, or the distance between a surgical instrument and the internal anatomical structure captured in the series of images.

17. The system of claim 12, further comprising providing a visual or audio warning if the surgical tool is determined to be within a threshold of (a) another surgical tool or (b) a designated area of ​​the internal anatomy.

18. The system according to claim 12, wherein, The endoscope includes a stereo camera having at least two lenses and corresponding image sensors at different advantageous positions, and the three-dimensional reconstruction includes establishing a stereo correspondence between the images from the stereo camera to reconstruct the surface shape of a portion of the internal anatomy.

19. The system according to claim 12, wherein, The display includes at least one of a stereoscopic display, a 2D display, and a head-mounted display.

Images