CAMERA CALIBRATION FOR SURGICAL SYSTEMS

Abstract

The present disclosure describes a system and method for camera calibration. The system includes a memory and a control unit communicatively coupled to the memory. The control unit moves an endoscope through a cannula and stops the endoscope in the cannula. The control unit also uses the endoscope at the position to acquire images of a reference corresponding to the cannula and adjusts a parameter of the endoscope based on the reference in the images.

Description

RELATED REGISTRATION

[0001] This application claims priority over the preliminary US patent application with serial number 63 / 687,825, which was filed on August 28, 2024, and which is hereby incorporated into this document by reference. SPECIALIZATION

[0002] The present disclosure relates generally to medical systems (e.g., digital reference point systems, anatomy detection systems, clinical guidance systems, and surgical systems). More specifically, the present disclosure relates to a medical system that calibrates the stereo cameras of an endoscope. STATE OF THE ART

[0003] Physicians use computerized medical systems to perform various medical tasks. For example, physicians can use computerized surgical systems to perform operations on patients, even remotely. These surgical systems use endoscopes with stereo cameras (e.g., a left camera and a right camera) to provide physicians with different views of the surgical site during the procedure. The images from the cameras are also used to take measurements at the surgical site (e.g., to measure depth or distance to an anatomical structure). Due to temperature, pressure, and / or other conditions present during the operation of an endoscope, the cameras can become distorted or drift, introducing misalignment between the images from the cameras.This leads to inaccurate measurements taken using the images from the cameras. SUMMARY

[0004] The present disclosure describes a system and method for camera calibration. According to one embodiment, the system includes a memory and a control unit communicatively coupled to the memory. The control unit moves an endoscope through a cannula and stops the endoscope at a position within the cannula. Using the endoscope, the control unit also acquires images of a reference corresponding to the cannula at that position and adjusts a parameter of the endoscope based on the reference in the images.

[0005] According to another embodiment, a method includes moving an endoscope through a cannula and stopping the endoscope at a position within the cannula. The method further includes acquiring images of a reference corresponding to the cannula at that position using the endoscope and adjusting a parameter of the endoscope based on the reference in the images. Other embodiments include a non-volatile, machine-readable medium that stores instructions which, when executed by a processor, cause the processor to perform the method.

[0006] The preceding general description and the following detailed description are in their essence exemplary and explanatory, and are intended to provide an understanding of the present disclosure without limiting its scope. In this respect, further aspects, features, and advantages of the present disclosure will be apparent to a person skilled in the art from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1A to 1C illustrate an exemplary medical system. Fig. 2A illustrates an example medical system. Fig. 2B illustrates an exemplary medical instrument system within the medical system of Fig. 2A. Fig. 2C illustrates an exemplary part of the medical instrument system of Fig. 2B. Fig. Figure 3 illustrates an example operation for taking measurements. Fig. Figures 4A to 4D illustrate exemplary operations for adjusting an endoscope camera. Fig. Figure 5 illustrates an example operation for adjusting an endoscope camera. Fig. Figures 6A to 6C illustrate exemplary operations for determining camera misalignment. Fig. Figure 7 is a flowchart of an exemplary procedure for adjusting an endoscope camera. DETAILED DESCRIPTION

[0007] Physicians use computerized medical systems to perform medical tasks. For example, physicians can use computerized surgical systems to perform operations on patients, even remotely. These surgical systems can use endoscopes with stereo cameras to provide physicians with different views of surgical sites during procedures (e.g., through a camera that records video of the surgical site). Images from the endoscopies can also be used to take measurements at the surgical sites. For example, a digital ruler application can use the images from the endoscope to measure the distances between points at the surgical sites or the distances to points at the surgical sites. As another example, during a fluorescence imaging procedure, a fluorescent dye or several fluorescent dyes can be injected into tissue.Different tissue depths can absorb different dyes or different amounts of dye, resulting in different tissue depths being illuminated in different ways (e.g., in different colors, shades, or tones). A digital ruler application or fluorescence imaging application can use the images of the illuminated tissue captured by the endoscope to measure the depth of different tissue areas with a similar appearance.

[0008] During endoscope operation, temperature, pressure, and other environmental conditions can cause the endoscope's camera lenses to drift, introducing misalignment into the stereo images. This misalignment can cause certain points in the images to shift by a small number of pixels. While this misalignment may be minor and easily overlooked, it can render measurements taken using the images inaccurate. For example, if the endoscope's existing parameters (e.g., extrinsic parameters) are used to convert the pixel coordinates of specific points in the images into three-dimensional coordinates in global space, the misalignment can lead to errors or inaccuracies in the three-dimensional coordinates. These inaccurate measurements then make it more difficult for surgeons to operate safely at the surgical site.For example, an inaccurate distance or depth measurement between points at an operating site can cause a surgeon to move a surgical instrument too far, which can result in the instrument making a larger cut or incision than necessary.

[0009] The present disclosure describes a medical system (e.g., a surgical system) that detects a misalignment between the cameras of an endoscope and adjusts parameters (e.g., camera calibration parameters) of the endoscope to correct the misalignment. As the endoscope is advanced through a cannula (e.g., a tube) to a surgical site, the system generally stops the endoscope at a position within the cannula (e.g., at a first mark printed on the inside of the cannula, at a predetermined stopping point in the cannula, etc.). Using its cameras, the endoscope acquires images of a reference point corresponding to the cannula (e.g., a second mark of known size printed on the inside of the cannula at a predetermined distance from the first mark, an opening at the end of the cannula, etc.).The system analyzes the images to determine pixel misalignments between them. Since the system knows the size of the reference image, it can calculate distance misalignments from these pixel misalignments. The system then adjusts the endoscope parameters (e.g., extrinsic parameters) to correct or compensate for these distance misalignments.

[0010] In certain configurations, the medical system offers several technical advantages. For example, by adjusting the endoscope's parameters, the system compensates for distance misalignments, thus enabling the use of endoscope images for taking measurements (e.g., depth measurements, distance measurements, etc.) at the surgical site. As another example, the system improves the accuracy of measurements taken using endoscope images, thereby enhancing patient health and safety.

[0011] In some examples, one or more components of the medical system can be implemented as a computer-assisted surgical system. However, it is understood that the medical system can be implemented in any type of medical system (e.g., digital reference point systems, anatomy detection systems, and clinical guidance systems). Fig. Figure 1A shows an example of a computer-assisted surgical system 100 that can implement some of the features described herein.

[0012] The surgical system 100 can include a manipulator assembly 102, a user control device 104, and an auxiliary device 106, all of which are communicatively coupled. The surgical system 100 can be used by a medical team to perform a computer-assisted medical procedure or other similar surgery on the body of a patient 108 or on another body if this is useful in a particular implementation. The medical team can include a first user 110-1 (such as a surgeon if it is a surgical procedure), a second user 110-2 (such as a patient-side assistant), a third user 110-3 (such as another assistant, a nurse, a resident, etc.).) and a fourth user 110-4 (such as an anesthesiologist in the case of a surgical procedure), all of whom can be collectively referred to as user 110, and each of whom can control, interact with, or otherwise use the surgical system 100. There may also be more, fewer, or additional users during a medical procedure if this is useful for a particular implementation. For example, the team composition may differ for different medical or non-medical procedures and include users with different roles.

[0013] Fig. While Figure 1A illustrates a minimally invasive medical procedure, such as a minimally invasive surgical procedure, it is understood that Surgical System 100 can be used similarly to perform open medical procedures or other types of operations. For example, operations such as exploratory imaging procedures, sham medical procedures used for training purposes, and / or other operations can also be performed.

[0014] The manipulator assembly 102 can include one or more manipulator arms 112 (e.g., manipulator arms 112-1 to 112-4), to which one or more instruments can be coupled. The instruments can be used for a computer-assisted surgical procedure on the patient 108 (e.g., by being at least partially inserted into the patient 108 and manipulated within the patient 108). Although the manipulator assembly 102 is shown and described herein as including four manipulator arms 112, the manipulator assembly 102 can include a single manipulator arm 112 or any other number of manipulator arms if this is useful in a particular implementation. The example of Fig. Figure 1A illustrates the manipulator arms 112 as robotic manipulator arms, but in some examples, one or more instruments can be partially or fully manually controlled, such as by being held and manually operated by a person. These partially or fully manually controlled instruments can be used in conjunction with, or as an alternative to, a computer-controlled instrumentation coupled to the manipulator arms 112.

[0015] During medical surgery, the user control device 104 can facilitate teleoperation control of the manipulator arms 112 and instruments attached to the manipulator arms 112 by the user 110-1. For this purpose, the user control device 104 can provide the user 110-1 with images of a surgical area associated with the patient 108, acquired by an imaging device. The manipulator arms 112, or any instruments coupled to the manipulator arms 112, can mimic the dexterity of the hand, wrist, and fingers of the user 110-1 through multiple degrees of freedom. In this way, the user 110-1 can intuitively perform a procedure (e.g., making an incision, suturing, etc.) using one or more of the manipulator arms 112, or any instruments coupled to the manipulator arms 112.

[0016] The auxiliary device 106 may include one or more computer devices that perform auxiliary functions to support the procedure, such as providing insufflation, electrocautery energy, illumination, or other energy for imaging devices, image processing, or coordinating components of the surgical system 100. In some examples, the auxiliary device 106 may include a display monitor 114 that shows one or more user interfaces or graphic or textual information to support the procedure. In some cases, the display monitor 114 may be a touchscreen display that provides input functionality to the user. Augmented content provided by a region-based augmentation system may be similar to or different from the content associated with the display monitor 114 or one or more display devices in the operating area (not shown).

[0017] The manipulator assembly 102, user control device 104, and auxiliary device 106 can be communicatively coupled to one another in any suitable manner. For example, the manipulator assembly 102, user control device 104, and auxiliary device 106 can be communicatively coupled by means of control lines 116, which can represent any wired or wireless communication link if this is advantageous in a particular implementation. For this purpose, the manipulator assembly 102, user control device 104, and auxiliary device 106 can each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, and so on.

[0018] Fig. Figure 1B illustrates an exemplary manipulator assembly 102. As shown in Fig. As shown in Figure 1B, the manipulator assembly 102 includes a base 118, a manipulator arm 112-1, a manipulator arm 112-2, a manipulator arm 112-3, and a manipulator arm 112-4. Each manipulator arm 112-1, 112-2, 112-3, and 112-4 is pivotally coupled to the base 118. While the base 118 may include casters to facilitate mobility, in some embodiments the manipulator assembly 102 is fixed to a floor, ceiling, operating table, supporting structure, or the like.

[0019] In a typical procedure, two of the manipulator arms 112-1, 112-2, 112-3, or 112-4 hold surgical instruments, and a third arm holds a stereo endoscope. The remaining manipulator arms are available to allow the insertion of other instruments at the work site. Alternatively, the remaining manipulator arms can be used to insert another endoscope or other imaging device, such as an ultrasound transducer, into the work site.

[0020] Each of the manipulator arms 112-1, 112-2, 112-3, and 112-4 can be formed from connecting elements that are coupled to one another and manipulated by actuated joints. Each of the manipulator arms 112-1, 112-2, 112-3, and 112-4 can include a setup arm and a device manipulator. The setup arm positions the device it holds so that a pivot point occurs at its entry point into the patient. The device manipulator can then manipulate the device it holds so that the device can be pivoted around the pivot point, inserted into and withdrawn from the entry point, and rotated around its shaft axis. Each of the manipulator arms 112-1, 112-2, 112-3 and 112-4 can include sensors (e.g. kinematic sensors, position sensors, accelerometers, etc.) that detect or track movement of the manipulator arms 112-1, 112-2, 112-3 and 112-4.These sensors can, for example, detect how far or how fast a manipulator arm 112-1, 112-2, 112-3 or 112-4 moves in a particular direction.

[0021] Fig. Figure 1C illustrates an exemplary user control device 104. The user control device 104 includes a stereo view display 120 so that the user can view the surgical site in stereo from the images captured by the stereoscopic camera of the manipulator assembly 102. Left and right eyepieces 122 and 124 are provided in the stereo view display 120 so that the user can see a left and right display screen within the display 120 with the user's left and right eyes, respectively. While the surgeon typically views an image of the surgical site on a suitable view device or display, they perform the surgical procedures on the patient by manipulating the master control input devices, which in turn control the movements of the robotic instruments.

[0022] Furthermore, the user control device 104 includes left and right input devices 126 and 128, which the user can grasp with their left and right hands, respectively, to manipulate devices (e.g., surgical instruments) held by the manipulator arms 112-1, 112-2, 112-3, and 112-4 of the manipulator assembly 102, preferably in six or more degrees of freedom (“DOF”). Foot pedals 130 with toe and heel controls are provided on the user control device 104, enabling the user to control the movement and / or actuation of devices connected to the foot pedals.

[0023] A processing unit 132 is provided in the user control device 104 for control and other purposes. The processing unit 132 performs various functions in the surgical system 100. One function performed by the processing unit 132 may be to shift and transmit the mechanical movement of the input devices 126 and 128 to actuate their corresponding joints in the associated manipulator arms 112-1, 112-2, 112-3, and 112-4, enabling the surgeon to effectively operate devices such as surgical instruments. Another function of the processing unit 132 may be to implement the procedures, cross-coupling control logic, and control units or processors described herein. The auxiliary device 106 may include a processing unit 132 that performs the functions or operations described herein.The processing device 132 can include a processor and a memory that perform the functions described herein.

[0024] The processor may include any electronic circuitry, in particular one or a combination of microprocessors, microcontrollers, application-specific integrated circuits (ASICs), application-specific instruction set processors (ASIPs), and / or state machines, communicatively coupled to a memory and controlling the operation of the user control device 104 and / or the auxiliary device 106. The processor may have an 8-bit, 16-bit, 32-bit, 64-bit, or any other suitable architecture. The processor may include an arithmetic logic unit (ALU) for performing arithmetic and logical operations, processor registers that provide operands to the ALU and store the results of ALU operations, and a control unit that retrieves instructions from memory and executes them by directing the coordinated operations of the ALU, registers, and other components.The processor may include other hardware that runs software to control and process information. The processor executes software stored in memory to perform one of the functions described herein. The processor controls the operation and management of the user control device 104 or the auxiliary device 106 by processing information (e.g., information received from the user control device 104, the manipulator assembly 102, the auxiliary device 106, and / or memory). The processor is not limited to a single processing device and may include multiple processing devices, either contained in the same device or computer or distributed across multiple devices or computers.The processor is deemed to be performing a set of functions or operations if the multiple processing devices collectively perform the set of functions or operations, even if different processing devices perform different functions or operations in the set.

[0025] Fig. Figure 2A illustrates an exemplary computer-assisted surgical system 200 that can implement some of the features described herein. The surgical system 200 can be used, for example, in surgical, diagnostic, therapeutic, biopsy, or non-medical procedures. As shown in Fig. As shown in Figure 2A, the surgical system 200 (which may be a robot-assisted surgical system) includes one or more manipulator assemblies 202 for operating one or more medical instrument systems 204 when performing various procedures on a patient P positioned on a table T in a medical setting. For example, the manipulator assembly 202 may drive the catheter or end-effector movement, apply a treatment to a target tissue, and / or manipulate control elements. The manipulator assembly 202 may be a remotely operated, non-remotely operated, or hybrid assembly of both remotely operated and non-remotely operated components, with selected degrees of freedom of movement that may be motorized and / or remotely operated, and selected degrees of freedom of movement that may be non-motorized and / or non-remotely operated.An operator input system 206, which may be located inside or outside the medical environment, generally includes one or more control units for controlling the manipulator assembly 202. The manipulator assembly 202 supports a medical instrument system 204 and may optionally include a variety of actuators or motors that drive inputs on the medical instrument system 204 in response to commands from a control system 212. The actuators may optionally include drive systems that, when coupled to the instrument system 204, move the medical instrument system 204 into a natural or surgically created anatomical opening. Other drive systems may move the distal end of the medical instrument in multiple degrees of freedom, including three degrees of linear motion (e.g., linear motion along the Cartesian x, y, and z axes) and three degrees of rotational motion (e.g.,Rotation around the Cartesian x, y, and z axes). The manipulator assembly 202 can support various other systems for rinsing, treatment, or other purposes. Such systems may include fluid systems (e.g., reservoirs, heating / cooling elements, pumps, and valves), generators, lasers, probes, and ablation components.

[0026] The surgical system 200 also includes a display system 210 for showing an image or representation of the surgical site and a medical instrument system 204. The image or representation is generated by an imaging system 209, which may include an endoscopic imaging system. The display system 210 and operator input system 206 can be configured so that an operator O can control the medical instrument system 204 and the operator input system 206 with the perception of telepresence. A graphical user interface can be displayed on the display system 210 and / or on a display system of an independent planning workstation.

[0027] In some examples, the imaging system 209 includes an endoscopic imaging system with components that are integrally or detachably coupled to the medical instrument system 204. However, in other examples, a separate imaging device, such as an endoscope attached to a separate manipulator assembly, can be used with the medical instrument system 204 to image the surgical site. The imaging system 209 can be implemented as hardware, firmware, software, or a combination thereof, interacting with or otherwise being executed by one or more computer processors, which may include the processors 214 of the control system 212.

[0028] The surgical system 200 also includes a sensor system 208. The sensor system 208 may include a position / location sensor system (e.g., an actuator encoder or an electromagnetic (EM) sensor system) and / or a shape sensor system (e.g., an optical fiber shape sensor) for determining the position, orientation, velocity magnitude, velocity vector, attitude, and / or shape of the medical instrument system 204. These sensors may also detect the position, orientation, or attitude of the patient P on the table T. For example, the sensors may detect whether the patient P's face is facing up or down. As another example, the sensors may detect the direction in which the patient P's head is facing. The sensor system 208 may also include temperature, pressure, force, or contact sensors, or similar devices.

[0029] The surgical system 200 may also include a control system 212, which includes at least one memory 216 and at least one computer processor 214 for controlling the interaction between the medical instrument system 204, the operator input system 206, the sensor system 208, and the display system 210. The control system 212 includes programmed instructions (e.g., a non-volatile, machine-readable medium that stores the instructions) for implementing a procedure using the surgical system 200, including navigating, directing, mapping, depositing, or retracting surgical features, applying treatments to target tissue (e.g., by applying energy), or similar actions.

[0030] Furthermore, the control system 212 can include a virtual visualization system to provide operator O with navigational assistance when controlling the medical instrument system 204 during an image-guided surgical procedure. Virtual navigation using the virtual visualization system can be based on a reference to an acquired preoperative or intraoperative dataset of anatomical pathways. The virtual visualization system processes images of the surgical site acquired using imaging technology such as computed tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and / or similar technologies.The 212 control system uses a preoperative image to locate the target tissue (using visual imaging techniques and / or by receiving user input) and to create a preoperative plan that includes an optimal initial treatment site. The preoperative plan may include, for example, a planned size to which an expandable device will be expanded, a treatment duration, a treatment temperature, and / or multiple placement sites.

[0031] The processor 214 is any electronic circuit, in particular one or a combination of microprocessors, microcontrollers, application-specific integrated circuits (ASICs), application-specific instruction set processors (ASIPs), and / or state machines, communicatively coupled to the memory 216 and controlling the operation of the control system 212. The processor 214 can have an 8-bit, 16-bit, 32-bit, 64-bit, or any other suitable architecture. The processor 214 can include an arithmetic logic unit (ALU) for performing arithmetic and logical operations, processor registers that provide operands to the ALU and store the results of ALU operations, and a control unit that retrieves instructions from memory and executes them by directing the coordinated operations of the ALU, registers, and other components.The processor 214 can include other hardware that runs software to control and process information. The processor 214 executes software stored in a memory 216 to perform one of the functions described herein. The processor 214 controls the operation and management of the control system 212 by processing information (e.g., information received from the manipulator assembly 202, the operator input system 206, and the memory 216). The processor 214 is not limited to a single processing device and can include multiple processing devices, either contained within the same device or computer, or distributed across multiple devices or computers.The processor 214 is deemed to be performing a set of functions or operations when the multiple processing units collectively perform the set of functions or operations, even if different processing units perform different functions or operations in the set.

[0032] Memory 216 can store data, operating software, or other information for Processor 214, either permanently or temporarily. Memory 216 can include one or a combination of volatile or non-volatile local or remote devices suitable for storing information. For example, Memory 216 can include Random Access Memory (RAM), Read-Only Memory (ROM), magnetic storage devices, optical storage devices, or any other suitable information storage device, or a combination of these devices. Software represents any suitable set of instructions, logic, or code contained in a computer-readable storage medium. For example, software can be contained in Memory 216, a floppy disk, a CD, or a USB flash drive.In particular embodiments, the software may include an application that can be executed by the processor 214 to perform one or more of the functions described herein. The memory 216 is not limited to a single memory and may include multiple memories contained in the same device or computer or distributed across multiple devices or computers. The memory 216 is considered to store a set of data, operating software, or information if the multiple memories collectively store the set of data, operating software, or information, even if different memories store different portions of the data, operating software, or information in the set.

[0033] Fig. Figure 2B illustrates an exemplary medical instrument system 204 within the surgical system 200. In some embodiments, the medical instrument system 204 is used in an image-guided medical procedure. For example, the medical instrument system 204 can be used for exploratory non-teleoperation procedures or in procedures using traditional manually controlled medical instruments, such as endoscopy.

[0034] The medical instrument system 204 includes an elongated flexible device 220, such as a flexible catheter or an endoscope (e.g., gastroscope, bronchoscope), coupled to a drive unit 222. The elongated flexible device 220 includes a flexible body 224 with a proximal end 226 and a distal end or tip portion 228. In some embodiments, the flexible body 224 has an outer diameter of approximately 14 to 20 millimeters. Other outer diameters of the flexible body may be larger or smaller. The flexible body 224 has an approximate length to reach specific parts of the anatomy, such as the lungs, sinuses, pharynx, or the upper or lower gastrointestinal tract, when the flexible body 224 is inserted into a patient's oral or nasal cavity.

[0035] The medical instrument system 204 includes a tracking system 230 for determining the position, orientation, velocity magnitude, velocity vector, position, and / or shape of the distal end 228 and / or one or more segments 232 along the flexible body 224 using one or more sensors and / or imaging devices. The entire length of the flexible body 224 between the distal end 228 and the proximal end 226 is effectively divided into the segments 232. The tracking system 230 is implemented as hardware, firmware, software, or a combination thereof, which interacts with or is otherwise executed by one or more computer processors, which may include the processors 214 of the control system 212.

[0036] The tracking system 230 tracks the distal end 228 and / or one or more of the segments 232 using a shape sensor 234. In some embodiments, the tracking system 230 tracks the distal end 228 using a position sensor system 236, such as an electromagnetic (EM) sensor system. In some examples, the position sensor system 236 measures six degrees of freedom (e.g., three position coordinates x, y, and z and three orientation angles specifying pitch, yaw, and roll of a base point) or five degrees of freedom (e.g., three position coordinates x, y, and z and two orientation angles specifying pitch and yaw of a base point).

[0037] The flexible body 224 includes one or more channels 238, which are dimensioned and designed to accommodate one or more medical instruments 240. In some embodiments, the flexible body 224 includes two channels 238 for separate instruments 240, but a different number of channels 238 can also be provided. Fig. 2C illustrates an exemplary part of the medical instrument system 204 of Fig. 2B. As in Fig. As can be seen in Figure 2C, the medical instrument 240 extends through the flexible body 224. In some embodiments, the medical instrument 240 can be used for procedures and aspects of procedures, such as surgery, biopsy, ablation, mapping, imaging, illumination, irrigation, or aspiration. The medical instrument 240 is deployed through the channel 238 of the flexible body 224 and is used at a target location within the anatomy. The medical instrument 240 includes, for example, imaging devices, biopsy instruments, ablation instruments, catheters, laser ablation fibers, and / or other surgical, diagnostic, or therapeutic tools. The medical tools include end effectors that have a single working element, such as a scalpel, a blunt blade, a lens, an optical fiber, an electrode, and / or the like.Other end effectors include, for example, forceps, graspers, balloons, needles, scissors, clip applicators, and / or the like. Other end effectors also include electrically activated end effectors such as electrosurgical electrodes, transducers, sensors, imaging devices, and / or the like. The medical instrument 240 is advanced from the opening of the channel 238 to perform the procedure and then retracted into the channel when the procedure is complete. The medical instrument 240 is removed from the proximal end 226 of the flexible body 224 or from another optional instrument access opening (not shown) along the flexible body 224. The medical instrument 240 can also be used with an image acquisition device (e.g., an endoscopic camera) within the elongated flexible device 220. Alternatively, the medical instrument 240 itself can be the image acquisition device.

[0038] The medical instrument 240 additionally houses cables, connecting links, or other actuating controls (not shown) that extend between the proximal and distal ends to control the bending of the distal end of the medical instrument 240. The flexible body 224 also houses cables, connecting links, or other steering controls (not shown) that extend between the drive unit 222 and the distal end 228 to control the bending of the distal end 228, as shown, for example, by the dashed line drawings 242 of the distal end 228. In some examples, at least four cables are used to provide independent up-down steering for controlling a pitching motion of the distal end 228 and left-right steering for controlling a yaw motion of the distal end 228.In embodiments where the medical instrument system 204 is actuated by a robot-assisted assembly, the drive unit 222 can include drive inputs that are detachably coupled to and powered by drive elements such as actuators of the teleoperation assembly. In some embodiments, the medical instrument system 204 includes gripping features, manual actuators, or other components for manual control of the movement of the medical instrument system 204. Information from the tracking system 230 can be sent to a navigation system 244, where the information is combined with information from the visualization system 246 and / or the preoperatively obtained models to provide the physician or other operator with real-time positional information.

[0039] Fig. Figures 3 to 7 illustrate exemplary operations performed by a computer system within a medical system (e.g., the surgical system 100 of Fig. 1A or the surgical system 200 of Fig. 2A). In general, the computer system (which may be implemented in the user control device 104 and / or the auxiliary device 106 of the surgical system 100 using the processing device 132 and / or in the control system 212 of the surgical system 200 using the processor 214 and the memory 216) detects misalignments between cameras of an endoscope and adjusts the parameters of the endoscope to correct or compensate for the misalignments.

[0040] The computer system can be described as performing certain operations (e.g., stopping the endoscope or slider, taking images, etc.) that may involve other components, such as the endoscope, the slider, the endoscope, etc. In these cases, it is understood that the control unit performs these operations by transmitting signals to the other components, causing them to carry out the operations.

[0041] Fig. Figure 3 illustrates an example of Operation 300 for taking measurements. The computer system performs Operation 300. As shown in Fig. As can be seen in Figure 3, an endoscope 302 (which may be recording images) is involved in the operation 300. The endoscope 302 can be a stereo endoscope, including a left camera 304 and a right camera 306. These cameras 304 and 306 can be offset from each other and record images of an object 308. For example, the endoscope 302 can be positioned at an operating site, and the cameras 304 and 306 can record images of an anatomical object at the operating site. In the example of Fig. 3 The left camera 304 takes a left image 310 of the object 308 and the right camera 306 takes a right image 312 of the object 308.

[0042] Although the left camera 304 and the right camera 306 are positioned close to each other on the endoscope 302, they are located at different physical positions and have different orientations. This results in the left camera 304 and the right camera 306 capturing different images of the object 308. For example, the left image 310 and the right image 312 may show different perspectives of the object 308. A portion of the object 308 may appear in a particular set of pixels in the left image 310, and the same portion of the object 308 may appear in the right image 312 in a slightly different set of pixels.

[0043] The computer system can use the left image 310, the right image 312, and parameters 314 to determine measurements 316 at the surgical site. For example, the computer system can measure the depth and / or distance using the left image 310, the right image 312, and parameters 314. The parameters 314 can include extrinsic parameters (e.g., extrinsic camera calibration parameters) and intrinsic parameters (e.g., intrinsic camera calibration parameters) of the endoscope 302. The extrinsic parameters can specify the position of the left camera 304 and the position of the right camera 306. For example, the extrinsic parameters can include a rotation matrix and / or a translation vector that specify the position and / or orientation of the left camera 304 and / or the right camera 306. The intrinsic parameters can specify how the left camera 304 and the right camera 306 capture the left image 310 and the right image 312.For example, the intrinsic parameters can include an optical axis, focal length, principal point, oblique coefficients, etc.

[0044] The computer system uses parameters 314 to determine measurement 316 from the left image 310 and the right image 312. For example, the computer system can use the intrinsic and extrinsic parameters to convert the pixels in the two-dimensional (2D) planes of the left image 310 and / or the right image 312 into three-dimensional (3D) coordinates in the world. The computer system can then use the 3D coordinates to determine measurements 316 such as depths and distances.

[0045] As previously discussed, the endoscope 302 may be exposed to different temperatures and / or pressures at different operating locations during operation. These temperatures and pressures may cause displacement or other physical distortions of the left camera 304 and / or right camera 306. As a result of these distortions, the left image 310 and / or the right image 312 produced by the left camera 304 and / or the right camera 306 may also be distorted. For example, the object 308 may occupy a different set of pixels in the left image 310 and / or the right image 312, which may lead to inaccuracies in the measurement 316.If the computer system uses left image 310 and right image 312 to calculate measurement 316, then the displacement and / or distortion in left image 310 and / or right image 312 may cause the computer system to calculate a depth and / or distance that is greater or less than the actual depth or distance. If a user moves a surgical instrument at the surgical site based on the inaccurate depth and / or distance, this may result in damage or injury at the surgical site.

[0046] Fig. Figures 4A to 4D illustrate example operations for adjusting an endoscope camera. The computer system generally performs these operations to determine distortions in the endoscope's cameras before the endoscope reaches the surgical site. The computer system can then adjust the endoscope's parameters to compensate for these distortions. If the computer system subsequently uses the adjusted parameters to calculate measurements (e.g., distances, depths, etc.) at the surgical site from the images captured by the endoscope, it produces more accurate measurements, which can improve the patient's health.

[0047] Fig. Figure 4A illustrates an example operation 400 performed by the computer system to move the endoscope 302 through a cannula 402. The computer system generally navigates the endoscope 302 to a surgical site by moving it through a cannula 402, which may resemble a tube. An operator may position the endoscope 302 on a carriage 404 located inside the cannula 402. The computer system can then move the carriage 404 through the cannula 402 to advance the endoscope 302 through it. When the endoscope 302 reaches the end of the cannula 402, it can emerge from the cannula 402 into the surgical site.

[0048] Fig. Figure 4B illustrates an example operation 420 performed by the computer system to calibrate the endoscope 302. The cannula 402 has markings printed on its inner wall. As the endoscope 302 moves through the cannula 402 on the carriage 404, it may encounter these markings. In the example of Fig. 4B The endoscope 302 encounters a first mark 422 printed on the inner wall of the cannula 402. Additionally, a second mark 424 (e.g., a square or rectangle, an arrangement of dots, an April day, etc.), serving as a reference, is printed on the inner wall of the cannula 402 at a predetermined or preset distance from the first mark 422. The second mark 424 can generally be any visual reference point that can provide subpixel coordination in image space. When the endoscope 302 encounters the first mark 422, the computer system can stop the carriage 404 and / or the endoscope 302 to prevent the endoscope 302 from advancing further down the cannula 402. For example, if the computer system detects the first marker 422 at a particular position in a video or image recorded by the endoscope 302, the computer system can stop the carriage 404 and / or the endoscope 302.In this way, the computer system stops the endoscope 302 at a predetermined or preset distance from the second marker 424.

[0049] In some embodiments, several second markings 424 are printed on the inner wall of the cannula 402. For example, the second markings 424 (e.g., several AprilTags) can be printed in such a way that they form a ring on the inner wall of the cannula 402. Additionally, any type of ink can be used to print the second marking 424. For example, an ultraviolet marking tracer ink can be used to print the second marking 424.

[0050] The computer system then uses the endoscope 302 to capture images of the second marker 424 from the predetermined or preset distance. In the example of Fig. 4B The computer system uses the cameras of the endoscope 302 to capture a left image 426 and a right image 428 of the second marker 424. The left image 426 can be captured by a left camera of the endoscope 302, and the right image 428 can be captured by a right camera of the endoscope 302.

[0051] The computer system then performs an adjustment 430 on the left image 426 and an adjustment 432 on the right image 428. The adjustments 430 and 432 can correct the distortion of the left image 426 and the right image 428. For example, due to the shape of the lens on the left and right cameras, the second mark 424 shown in the left image 426 and the right image 428 may exhibit distortions that introduce additional curvature. The adjustments 430 and 432 can remove some of this curvature, thereby straightening the lines and producing more accurate images of the second mark 424. In some embodiments, the adjustments 430 and 432 can also shift or move the second mark 424 in the left image 426 and / or the right image 428 to accommodate the different positions and / or orientations of the left and right cameras.In this way, the computer system places the left image 426 and the right image 428 into the same image plane.

[0052] As an example, the computer system may detect the edges and / or corners of the second mark 424 in the left image 426 and the right image 428. The computer system then straightens the edges and / or corners of the second mark 424. In some cases, the computer system may not straighten or correct the other parts of the second mark 424.

[0053] In embodiments where multiple second markings 424 are printed on the inner wall of the cannula, left image 426 and right image 428 may show multiple second markings 424. The computer system can detect the edges and / or corners of these second markings 424 in left image 426 and right image 428. These edges and / or corners can provide the computer system with sufficient information to calibrate the endoscope.

[0054] Fig. Figure 4C illustrates an example operation 440 performed by the computer system to adjust the parameters of the endoscope. The computer system begins with the left image 426 and the right image 428 of the second marker 424 after adjusting the left image 426 and the right image 428. The computer system compares the left image 426 and the right image 428 to determine a misalignment 442 between the left image 426 and the right image 428. For example, the computer system can determine the pixels 444 occupied by the second marker 424 in the left image 426 and the right image 428. The computer system can compare these pixels 444 to determine a difference in the pixels 444 occupied by the second marker 424 between the left image 426 and the right image 428. The misalignment 442 can indicate this difference.For example, the misalignment 442 can specify a horizontal difference in pixels 444 and a vertical difference in pixels 444. Consequently, the misalignment 442 can specify the number of pixels 444 by which the second marker 424 in the left image 426 is offset from the second marker 424 in the right image 428.

[0055] In some embodiments, the computer system converts the misalignment 442 into a translational misalignment and a rotational misalignment. The translational misalignment can be a translational component of the misalignment 442, and the rotational misalignment can be a rotational component of the misalignment 442. For example, the translational component can specify the magnitude (e.g., measured in pixels) of the misalignment 442 along a directional axis of the image space (e.g., along a horizontal or vertical axis of the image space). The rotational component can specify an angular component of the misalignment 442 (e.g., an angular offset from the horizontal or vertical axis of the image space).

[0056] The computer system is also provided with a size 446 of the second mark 424. For example, the size 446 can be a parameter or input provided to the computer system when operating using the cannula. The size 446 can specify a physical size of the second mark 424 printed on the cannula. For example, the size 446 can specify the physical dimensions (e.g., length and width) of the second mark 424. The computer system calculates a size per pixel 448 using the size 446 and the pixels 444. For example, the pixels 444 can specify a number of pixels in a horizontal direction and a number of pixels in a vertical direction occupied by the second mark 424 in the left image 426 and / or the right image 428. The computer system can divide the size 446 by the number of pixels 444 to determine the size per pixel 448.For example, the computer system can divide a horizontal dimension, specified by the size 446, by the number of pixels in the horizontal direction, and the computer system can divide a vertical dimension, specified by the size 446, by the number of pixels in the vertical direction.

[0057] The computer system then determines a misalignment distance of 450 using the misalignment 442 and the size per pixel 448. The computer system can multiply the misalignment 442 (which is the pixel misalignment) by the size per pixel 448 to produce the misalignment distance of 450. For example, the computer system can multiply a number of pixels of a horizontal misalignment, specified by the misalignment 442, by a horizontal size per pixel, specified by the size per pixel 448, to produce a horizontal misalignment distance, and the computer system can multiply a number of pixels of a vertical misalignment, specified by the misalignment 442, by a vertical size per pixel, specified by the size per pixel 448, to produce a vertical misalignment distance.Consequently, the misalignment distance 450 is a physical distance represented by the pixel misalignment between the second marker 424 in the left image 426 and the second marker 424 in the right image 428. The size per pixel 448 effectively converts the misalignment 442 in the image or pixel space into the misalignment distance 450 in the world or in global space.

[0058] In embodiments where the computer system has determined the translational misalignment and the rotational misalignment, the computer system can determine the misalignment distance 450 by multiplying the translational misalignment by the size per pixel 448 and by the cosine of the rotational misalignment.

[0059] The computer system determines an adjustment 452 of the endoscope's parameters 314 to correct for the misalignment distance 450. The adjustment 452 may include adjustments to the endoscope's extrinsic parameters. By adjusting these extrinsic parameters, the computer system calibrates how the 2D pixel coordinates in the images acquired by the endoscope are converted into global 3D coordinates to account for or correct for the misalignment distance 450. In this way, when the computer system uses parameters 314 to measure distances or depths from the images acquired by the endoscope, the measured distances or depths are accurate and correct for the misalignment distance.

[0060] In some embodiments, the computer system compares the misalignment distance 450 with one or more thresholds 454 (e.g., a horizontal threshold and a vertical threshold) to determine whether the computer system should adjust the parameters 314. If the misalignment distance 450 falls below the threshold 454, the computer system may determine that the misalignment distance 450 is within tolerance and maintain the parameters 314. If the misalignment distance 450 exceeds the threshold 454, the computer system may perform the adjustment 452 of the parameters 314 to correct the misalignment distance 450.

[0061] In this way, the computer system adjusts the endoscope's parameters (e.g., extrinsic parameters) to correct for any physical distortions that may have occurred in the endoscope's cameras. By making these adjustments, the computer system produces more accurate measurements (e.g., distance and / or depth measurements) using the images captured by the endoscope. These more accurate measurements can reduce the likelihood of injury or damage during a procedure at the surgical site.

[0062] In some embodiments, the computer system implements a threshold value 454 that indicates whether the endoscope should be used for measuring distances or depths. For example, if the misalignment distance 450 exceeds the threshold value 454, the computer system may determine that the endoscope cannot be calibrated to correct the misalignment and that a different endoscope should be used. If the endoscope continues to be used, the computer system may prevent the loading or use of measurement applications (such as a digital ruler application or a fluorescence imaging application).

[0063] Fig. Figure 4D illustrates an example operation 460 performed by the computer system to calibrate the endoscope 302. Generally, the computer system can perform operation 460 even if no markings are present on the inner wall of the cannula.

[0064] As in Fig. As can be seen in Figure 4D, the carriage 404 can move the endoscope 302 through the cannula 402. The carriage 404 can stop the endoscope 302 at a position 462 in the cannula 402. Position 462 can be a predetermined position located a predetermined or preset distance from one end of the cannula 402. In some cases, position 462 can be specified by software in the computer system (e.g., a software stop), and the software can stop the carriage 404 when it reaches position 462. There is an opening 464 (e.g., a circular opening) at the end of the cannula 402 that serves as a reference. When the carriage 404 stops at position 462, the endoscope 302 can be a predetermined or preset distance from the opening 464.

[0065] The computer system then uses the endoscope 302 to capture images of the opening 464 from the predetermined or preset distance. In the example of Fig. In 4D, the computer system uses the cameras of the endoscope 302 to capture a left image 466 and a right image 468 of the opening 464. The left image 466 can be captured by a left camera of the endoscope 302, and the right image 468 can be captured by a right camera of the endoscope 302.

[0066] The computer system then performs an adjustment 470 on the left image 466 and an adjustment 472 on the right image 468. The adjustments 470 and 472 can correct the distortion of the left image 466 and the right image 468. For example, due to the shape of the lenses on the left and right cameras, the aperture 464 shown in the left image 466 and the right image 468 may exhibit distortions that introduce additional curvature. The adjustments 470 and 472 can remove some of this curvature, making the boundary straighter and smoother, and producing more accurate images of the aperture 464. In some embodiments, the adjustments 470 and 472 can also shift or move the aperture 464 in the left image 466 and / or the right image 468 to accommodate the different positions and / or orientations of the left and right cameras.In this way, the computer system places the left image 466 and the right image 468 into the same image plane.

[0067] For example, the computer system can detect the boundary of aperture 464 in the left image 466 and the right image 468. The computer system then smooths the boundary of aperture 464. In some cases, the computer system may not straighten or smooth the other parts of aperture 464.

[0068] Then the computer system can perform the in Fig. The operation 440 shown in Figure 4C can be performed using left image 466 and right image 468 (e.g., instead of left image 426 and right image 428) to adjust the endoscope 302. For example, the computer system may detect a misalignment between left image 466 and right image 468. The computer system can then determine an adjustment to the parameters of the endoscope 302 that will reduce or correct the misalignment. The computer system can use a predetermined or preset distance between the endoscope 302 and the end of the cannula 402 and the size of the opening 464 to determine the parameter adjustment. In this way, the computer system can correct a misalignment in the endoscope 302 even if there are no markings on the inner wall of the cannula 402. Instead, the computer system can use the opening 464 at the end of the cannula 402 as a replacement for the markings.

[0069] In some embodiments, the computer system can stop the carriage 404 and the endoscope 302 at several locations within the cannula 402. Each of these locations can be at a different distance from the opening 464. For example, the computer system can stop the endoscope 302 at location 462 and then at the end of the cannula 402 itself. The endoscope 302 can then acquire images of the opening 464 at the end of the cannula 402. The computer system can use the multiple sets of images of the opening 464 to calibrate the endoscope 302.

[0070] Fig. Figure 5 illustrates an example operation 500 for adjusting an endoscope camera. Generally, the computer system performs operation 500 to adjust the luminance of the endoscope. The computer system begins with the left image 426 from the left camera of the endoscope and the right image 428 from the right camera of the endoscope. The computer system analyzes the left image 426 and the right image 428 to determine a contrast 502 and / or an intensity 504 of the left image 426 and the right image 428. The contrast 502 is a measurement that indicates the difference in luminance or color that makes an object in the left image 426 and the right image 428 visible against a background of a different luminance or color. The intensity 504 is a measurement that indicates the amount of light reflected by an object in the left image 426 and the right image 428.Both of these measurements can indicate how easy it is to perceive objects in the left image 426 and the right image 428, and to perceive parts of the left image 426 and the right image 428 (e.g., borders of the left image 426 and the right image 428).

[0071] The computer system determines an adjustment 506 of the endoscope's luminance 508 based on the contrast 502 and intensity 504. The computer system can compare the contrast 502 and / or intensity 504 with one or more threshold values ​​that indicate whether the luminance 508 should be increased or decreased. For example, if the contrast 502 and / or intensity 504 fall below certain threshold values, the computer system can determine the adjustment 506 to increase the luminance 508. If the contrast 502 and / or intensity 504 exceed certain threshold values, the computer system can determine the adjustment 506 to decrease the luminance 508. The magnitude of the adjustment 506 can be determined by the differences between the contrast 502 and / or intensity 504 and their respective threshold values. The greater the differences, the greater the magnitude of the adjustment 506, and vice versa.

[0072] Adjusting the luminance 508 of the endoscope can adjust the amount of light emitted by a light source (e.g., a light-emitting diode) positioned on the endoscope. Increasing the luminance 508 increases the amount of light emitted, and decreasing the luminance 508 decreases the amount of light emitted. By emitting more or less light, the computer system can increase or decrease the contrast 502 and / or intensity 504 in the images captured by the endoscope. In this way, the computer system can facilitate the differentiation of objects appearing in the images and the distinction between parts of the images.

[0073] Fig. Figures 6A to 6C illustrate exemplary operations for determining camera misalignment. Generally, the computer system performs these operations after the endoscope has passed through the cannula to the surgical site to determine whether further endoscope calibrations are necessary.

[0074] Fig. Figure 6A shows an example of operation 600 performed by the computer system. As in Fig. As can be seen in Figure 6A, the endoscope 302 has passed through the cannula to an operating site 602. An object 604 (e.g., an anatomical object) is located at the operating site 602. The endoscope 302 is directed towards the object 604. The cameras of the endoscope 302 record the left image 606 and the right image 608 of the object 604.

[0075] Fig. Figure 6B shows an example left image 606 or right image 608. As in Fig. As can be seen in Figure 6B, image 606 / 608 shows object 604. Additionally, image 606 / 608 includes a border 610 near the periphery of the image. Depending on the properties of the lenses in the endoscope cameras, the border 610 can be round, elliptical, or circular. Portions of image 606 / 608 between the border 610 and the periphery of image 606 / 608 may be black.

[0076] Fig. Figure 6C shows an example operation 620 performed by the computer system to determine whether further adjustments are needed to calibrate the endoscope. The computer system begins by analyzing the frame 610 in an image from the endoscope (e.g., the left image 606). The computer system compares the image to a reference image 622. More precisely, the computer system compares the frame 610 in the image to a reference frame 624 in the reference image 622. The reference image 622 may have been acquired by the endoscope when the endoscope was confirmed as calibrated. The computer system then stored the reference image 622 for future use.

[0077] The computer system compares the frame 610 with the reference frame 624 to determine a misalignment 626. For example, the computer system can determine whether the frame 610 occupies the same pixels in the image as the reference frame 624 in the reference image 622. The misalignment 626 can specify a number of pixels (e.g., a number of pixels in a horizontal direction and / or a number of pixels in a vertical direction) by which the frame 610 in the image differs from the reference frame 624 in the reference image 622.

[0078] The computer system then compares the misalignment 626 with one or more threshold values ​​628. For example, the computer system can compare the number of pixels in the horizontal direction specified by the misalignment 626 with a horizontal threshold value, and the computer system can compare the number of pixels in the vertical direction specified by the misalignment 626 with a vertical threshold value. If the misalignment 626 falls below the threshold values ​​628, the computer system can determine that no further calibration of the endoscope is required.

[0079] If the misalignment 626 exceeds the threshold values ​​628, the computer system determines that further calibration should be performed. The computer system generates a warning 630, which may include a message indicating that further calibration of the endoscope should be performed. The computer system communicates warning 630 to the user to warn them that calibration is required. The user can respond by retracting the endoscope into the cannula to recalibrate the endoscope using the [unclear text - likely a reference to a specific tool or device]. Fig. Recalibrate the operations shown in 4A to 4C.

[0080] Fig. Figure 7 is a flowchart of an exemplary method 700 for adjusting an endoscope camera. In certain embodiments, the method 700 is performed by a computer system (located in the user control device 104 and / or the auxiliary device 106 of the surgical system 100 using the [unclear]). Fig. 1A to 1C shown processing device 132 and / or in the control system 212 of the surgical system 200 using the processor 214 and the memory 216, which are in the Fig. (shown in 2A to 2C, which can be implemented). By performing procedure 700, the computer system calibrates an endoscope.

[0081] In Block 702, the computer system moves the endoscope through a cannula (which may resemble a tube) toward a surgical site. The cannula may have various markings printed on its inner wall. As the endoscope moves through the cannula, it may encounter these markings. The computer system can determine when the endoscope has encountered a marking by detecting the marking in an image or video produced by the endoscope. The computer system can control a carriage on which the endoscope is positioned to move or stop the endoscope within the cannula.

[0082] In block 704, the computer system stops the endoscope. For example, the computer system can stop the endoscope according to a first marker. The first marker is printed on the inner wall of the cannula and can indicate a stopping point for the endoscope. For example, the first marker can be a line or a box. The computer system can stop the endoscope (e.g., stop the carriage) when it detects from the image or video of the endoscope that it is positioned near or at the first marker. As another example, the computer system can stop the endoscope according to a software stop. The computer system can detect when the endoscope is at a position at a predetermined or preset distance from the end of the cannula and stop the endoscope at that predetermined or preset distance.

[0083] In Block 706, the computer system uses the endoscope to capture images of a reference. For example, the reference could be a second mark printed on the inner wall of the cannula when the endoscope is stopped. The second mark could be an April Tag, and the computer system could know its size (e.g., its physical dimensions). Additionally, the second mark could be printed at a predetermined or preset distance from the first mark. As another example, the reference could be an opening (e.g., a circular opening) at the end of the cannula. The endoscope could include a stereo camera (e.g., left and right cameras) that produces multiple images (e.g., a left image and a right image) of the reference.

[0084] In block 708, the computer system adjusts an endoscope parameter based on the images of the reference taken by the endoscope. For example, the computer system can compare the images of the reference (e.g., the left image and the right image) to determine any pixel misalignment between the reference and the images. Since the computer system knows the physical size of the reference, it can use this size to convert the pixel misalignment into a physical misalignment distance. The computer system then adjusts the endoscope parameter to correct for this misalignment distance. After adjusting the parameter, the computer system can then use it to convert the 2D coordinates of the pixels in the images from the endoscope into global 3D coordinates. The computer then uses these global 3D coordinates to determine measurements (e.g., measured distances and / or depths).

[0085] In summary, a medical system (e.g., a surgical system) detects misalignment between the cameras of an endoscope and adjusts the endoscope's parameters to correct the misalignment. Generally, as the endoscope is advanced through a cannula (e.g., a tube) toward a surgical site, the system stops the endoscope within the cannula. The endoscope then captures images of a reference point (e.g., a mark on the inner wall of the cannula, an opening at one end of the cannula, etc.) using its cameras. The system analyzes the images to determine pixel misalignments between the images. Because the system knows the size of the reference point, it can calculate distance misalignments from these pixel misalignments. The system then adjusts the endoscope's parameters (e.g., extrinsic parameters) to correct or compensate for these distance misalignments.

[0086] This description and the accompanying drawings, which illustrate aspects, embodiments, or modules, should not be considered limiting. Various mechanical, compositional, structural, electrical, and operational modifications may be made without deviating from the inventive concept and the scope of this description and the claims. In some cases, well-known circuits, structures, or techniques are not shown or described in detail so as not to obscure other features. Identical numbers in two or more figures represent the same or similar elements.

[0087] This description sets forth specific details that describe some embodiments in accordance with the present disclosure. Numerous specific details are set forth to provide a thorough understanding of the embodiments. However, it will be obvious to the person skilled in the art that some embodiments are practically feasible without some or all of these specific elements. The specific embodiments disclosed herein are intended to be illustrative, but not limiting. The person skilled in the art will recognize other elements which, although not specifically described here, are within the scope and inventive concept of this disclosure.Furthermore, to avoid unnecessary repetition, one or more features shown and described in connection with one embodiment may also be incorporated into other embodiments, unless specifically described otherwise or if one or more features would render an embodiment inoperable.

[0088] Furthermore, the terminology used in this description should not be restrictive. For example, spatial terms such as "below," "under," "lower," "above," "upper," "proximal," "distal," and similar terms may be used to describe the relationship of one element or feature to another illustrated in the figures. These spatial terms are intended to encompass various positions (i.e., locations) and orientations (i.e., rotational placements) of the elements or their operation, in addition to the position and orientation shown in the figures. For instance, if the contents of one of the figures were reversed, elements described as being "below" or "underneath" other elements or features would be "above" or "above" other elements or features. Therefore, the exemplary term "below" can encompass both positions and orientations of above and below.A device can be oriented differently (rotated 90 degrees, or in other orientations), and the spatial descriptive terms used here can then be interpreted accordingly. Similarly, descriptions of movements along and around different axes also include various special element positions and orientations. Furthermore, the singular forms "a," "an," and "the" should also include the plural forms unless the context indicates otherwise. And while the terms "includes," "comprehensive," "includes," and similar terms indicate the presence of mentioned features, steps, operations, elements, and / or components, they do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.Components described as coupled can be directly coupled electrically or mechanically, or they can be indirectly coupled via one or more intermediate components.

[0089] Elements that are described in detail with reference to one embodiment or module may, if practically feasible, also be included in other embodiments or modules in which they are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment and not with reference to a second embodiment, it may nevertheless be claimed as included in the second embodiment.Therefore, to avoid unnecessary repetition in the following description, one or more elements shown or described in connection with one embodiment or application may also be incorporated into other embodiments or aspects, unless specifically described otherwise, unless the one or more elements would render one or more embodiments inoperable, or unless two or more of the elements provide conflicting functions.

[0090] In some cases, well-known procedures, interventions, components and circuits were not shown or described in detail in order to avoid unnecessarily obscuring other aspects of the embodiments.

[0091] This disclosure describes various devices, elements, and fractions of computerized devices and elements with respect to their state in three-dimensional space. As used herein, the term "position" refers to the location of an element or fraction of an element in three-dimensional space (e.g., three translational degrees of freedom along Cartesian x, y, and z coordinates). As used herein, the term "orientation" refers to the rotational placement of an element or fraction of an element (three rotational degrees of freedom—e.g., roll, pitch, and yaw). As used herein, the term "shape" refers to a set of positions or orientations measured along an element.As used herein, and for a device with repositionable arms, the term “proximal” refers to a direction towards the base of the computerized device along its kinematic chain, and “distal” refers to a direction away from the base along the kinematic chain.

[0092] Aspects of this disclosure are described with reference to computerized systems and devices, which may include remotely operated, remotely controlled, autonomous, semi-autonomous, robotic, and / or similar systems and devices. Furthermore, aspects of this disclosure are described with respect to an embodiment using a medical system such as the DA VINCI SURGICAL SYSTEM or ION SYSTEM, which has been commercially developed by Intuitive Surgical, Inc. of Sunnyvale, California. However, those knowledgeable in this field will understand that the aspects disclosed herein may be performed and implemented in a variety of ways, including in robotic and, where appropriate, non-robotic embodiments. Techniques described with reference to surgical instruments and surgical procedures may also be used in other contexts.Therefore, the instruments, systems, and procedures described herein may be used for humans, animals, parts of human or animal anatomy, industrial systems, general robotic or teleoperation systems. Further examples include the use of the instruments, systems, and procedures described herein for non-medical purposes, including industrial uses, general robotic uses, capturing or manipulating non-tissue workpieces, cosmetic enhancements, imaging of human or animal anatomy, collecting data from human or animal anatomy, setting up or dismantling systems, training medical and non-medical personnel, and / or similar applications. Additional exemplary applications include, but are not limited to, the following:The use includes procedures on tissue removed from human or animal anatomy (with or without reintroduction) and on human or animal cadavers. Furthermore, these techniques can also be used for medical treatments or diagnostic procedures, which may involve surgical aspects.

[0093] Although illustrative embodiments have been shown and described, the foregoing disclosure considers a wide range of modifications, changes, and substitutions, and in some cases, some features of the embodiments can be used without correspondingly using other features. Many variations, alternatives, and modifications would be apparent to a person skilled in the art. Therefore, the scope of the disclosure is to be limited only by the following claims, and it is appropriate that the claims be interpreted broadly and in a manner consistent with the scope of the embodiments disclosed herein. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 63 / 687,825

[0001]

Claims

[1] A system for adjusting an endoscope, the system comprising: a storage facility; and a control unit that is communicatively coupled to the memory, wherein the control unit: the endoscope is moved through a cannula; the endoscope stops at a position in the cannula; using the endoscope to take images of a reference at the position corresponding to the cannula; and A parameter of the endoscope was adjusted based on the reference in the images. [2] System according to claim 1, wherein stopping the endoscope comprises stopping a slide on which the endoscope is positioned. [3] System according to one of claims 1-2, wherein the control unit further prevents the endoscope from moving further through the cannula until the endoscope has taken the images. [4] System according to any one of claims 1-3, wherein adjusting the parameter comprises: Correcting the distortion of the images to produce undistorted images; Comparing the corrected images to determine any misalignment in the endoscope; Determine, based on a reference size and the misalignment in the endoscope, an adjustment; and Adjusting the parameter. [5] System according to claim 4, wherein the parameter is an extrinsic parameter of the endoscope. [6] System according to any one of claims 1-5, wherein the control unit further: at least one of which is determined by a contrast or intensity of the reference in the images; and The luminosity of the endoscope is determined based on at least one of the contrast or intensity. [7] System according to one of claims 1-6, wherein the reference comprises a mark on an inner wall of the cannula. [8] System according to one of claims 1-7, wherein the reference comprises an opening at one end of the cannula. [9] System according to one of claims 1-8, wherein the holding of the endoscope at the position in the cannula is carried out according to a marking on an inner wall of the cannula and at a predetermined distance from the reference. [10] System according to any one of claims 1-9, wherein the control unit further: a frame of an image taken by the endoscope after the endoscope was positioned at a surgical site outside the cannula; a difference is determined between the image's border and a reference border; and Based on the difference exceeding a threshold, a warning is generated indicating that the endoscope should be calibrated. [11] System according to any one of claims 1-10, wherein the endoscope comprises a first camera and a second camera, wherein the images comprise a first image taken by the first camera and a second image taken by the second camera, and wherein adjusting the parameter comprises: Determining a pixel misalignment between the first image and the second image; Convert, based on a reference size, the pixel misalignment into a translational misalignment and a rotational misalignment; and Adjusting the parameter based on translational misalignment and rotational misalignment. [12] System according to one of claims 1-11, wherein the control unit further measures a distance based on the adjusted parameter. [13] A method for adjusting an endoscope, the method comprising: Moving the endoscope through a cannula; Holding the endoscope in one position in the cannula; To take, using the endoscope at the position, images of a reference corresponding to the cannula; and Adjusting a parameter of the endoscope based on the reference in the images. [14] Method according to claim 13, wherein stopping the endoscope comprises stopping a slide on which the endoscope is positioned. [15] Method according to one of claims 13-14, further comprising preventing the endoscope from moving further through the cannula until the endoscope has taken the images. [16] Method according to any one of claims 13-15, wherein adjusting the parameter comprises: Correcting the distortion of the images to produce undistorted images; Comparing the corrected images to determine any misalignment in the endoscope; Determining an adjustment based on a reference measurement and the misalignment in the endoscope; and Adjusting the parameter. [17] Method according to claim 16, wherein the parameter is an extrinsic parameter of the endoscope. [18] A method according to any one of claims 13-17, further comprising: Determine at least one of the contrasts or intensities of the reference in the images; and Adjusting the luminosity of the endoscope based on at least one of the contrast or intensity. [19] Method according to one of claims 13-18, wherein the reference comprises a mark on an inner wall of the cannula. [20] A non-volatile, machine-readable medium that stores instructions for adjusting an endoscope which, when executed by a processor, cause the processor to: the endoscope is moved through a cannula; the endoscope stops at a position in the cannula; using the endoscope to take images of a reference at the position corresponding to the cannula; and A parameter of the endoscope was adjusted based on the reference in the images.

Images