Camera calibration for surgical systems

Abstract

The invention relates to camera calibration for surgical systems. The present disclosure describes systems and methods for camera calibration. The system includes a memory and a controller communicatively coupled to the memory. The controller moves the endoscope through the cannula and stops the endoscope in the cannula. The controller also captures an image of a reference corresponding to the cannula using the endoscope at the location, and adjusts a parameter of the endoscope based on the reference in the image.

Description

[0001] Related applications

[0002] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 687,825, filed on August 28, 2024, which is incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to medical systems (e.g., digital reference systems, anatomical structure detection systems, clinical guidance systems, and surgical systems). Specifically, this disclosure relates to medical systems for calibrating stereoscopic imaging devices for endoscopes. Background Technology

[0004] Physicians use computer-assisted medical systems to perform various medical tasks. For example, physicians can use computer-assisted surgical systems to perform (or even remotely perform) procedures on patients. These surgical systems use endoscopes with stereoscopic camera units (e.g., left and right cameras) to provide the physician with various views of the surgical site during the procedure. Images from the camera units are also used to make measurements at the surgical site (e.g., measuring depth or distance to anatomical structures). Due to temperature, pressure, and / or other conditions present when the endoscope is operated on, the camera units may be distorted or offset, introducing misalignment between the images from the camera units. Consequently, measurements made using images from the camera units become inaccurate. Summary of the Invention

[0005] This disclosure describes a system and method for calibrating a camera device. According to one embodiment, the system includes a memory and a controller communicatively coupled to the memory. The controller moves an endoscope through a cannula and stops the endoscope at a position within the cannula. The controller also uses the endoscope at this position to capture an image of a reference corresponding to the cannula and adjusts endoscope parameters based on the reference in the image.

[0006] According to another embodiment, a method includes: moving an endoscope through a cannula, and stopping the endoscope at a position within the cannula. The method further includes: using the endoscope at this position to capture an image of a reference corresponding to the cannula, and adjusting parameters of the endoscope based on the reference in the image. Other embodiments include: a non-transitory machine-readable medium storing instructions that, when executed by a processor, cause the processor to perform the method.

[0007] The foregoing general description and the following detailed description are exemplary and illustrative in nature and are intended to provide an understanding of this disclosure without limiting its scope. In this regard, additional aspects, features, and advantages of this disclosure will be apparent to those skilled in the art from the following detailed description. Attached Figure Description

[0008] Figures 1A to 1C An example medical system is shown.

[0009] Figure 2A An example medical system is shown.

[0010] Figure 2B It shows Figure 2A Example medical device systems in medical systems.

[0011] Figure 2C It shows Figure 2B Example section of a medical device system.

[0012] Figure 3 An example operation for performing measurements is shown.

[0013] Figures 4A to 4D An example operation for adjusting the endoscopic camera device is shown.

[0014] Figure 5 An example operation for adjusting the endoscopic camera device is shown.

[0015] Figures 6A to 6C An example operation for determining that a camera device is misaligned is shown.

[0016] Figure 7 This is a flowchart of an example method for adjusting an endoscopic camera device. Detailed Implementation

[0017] Physicians use computer-assisted medical systems to perform medical tasks. For example, physicians can use computer-assisted surgical systems to perform (or even remotely perform) procedures on patients. These surgical systems can use endoscopes with stereoscopic imaging capabilities to provide physicians with various views of the surgical site during the procedure (e.g., capturing video of the surgical site via the imaging capabilities). Images from the endoscope can also be used to make measurements at the surgical site. For example, digital ruler applications can use images from the endoscope to measure distances between points within the surgical site or to points within the surgical site. As another example, during a fluorescence imaging procedure, one or more fluorescent dyes can be injected into the tissue. Different depths of the tissue may receive different dyes or different amounts of dye, resulting in tissue at different depths being illuminated in different ways (e.g., different colors, different shadows, or hues). Digital ruler applications or fluorescence imaging applications can use images of the illuminated tissue captured by the endoscope to measure the depth of different portions of tissue with similar appearances.

[0018] When operating an endoscope, temperature, pressure, and other environmental conditions can cause lens shift in the endoscope's camera, introducing misalignment into the stereoscopic image from the camera. Misalignment can cause some points in the image to move by a few pixels. This misalignment may be small and easily overlooked, but it can lead to inaccurate measurements made using the image. For example, when existing parameters of the endoscope (e.g., extrinsic parameters) are used to convert the pixel coordinates of some points in the image into three-dimensional coordinates in global space, misalignment can cause the three-dimensional coordinates to include errors or inaccuracies. These inaccurate measurements make it more difficult for the surgeon to operate safely at the surgical site. For example, inaccurate distance or depth measurements between points at the surgical site may cause the surgeon to move the surgical instrument too far, potentially resulting in a larger cut or incision than necessary.

[0019] This disclosure describes a medical system (e.g., a surgical system) that detects misalignment between the imaging devices of an endoscope and adjusts endoscope parameters (e.g., imaging device calibration parameters) to correct the misalignment. Typically, as the endoscope is advanced toward the surgical site via a cannula (e.g., a tube), the system stops the endoscope at a position within the cannula (e.g., at a first mark printed on the inside of the cannula, at a predetermined stop position within the cannula, etc.). The endoscope uses its imaging devices to capture an image of a reference corresponding to the cannula (e.g., a second mark of known size printed on the inside of the cannula at a predetermined distance from the first mark, an opening at the end of the cannula, etc.). The system analyzes the images to determine pixel misalignment between the images. Because the system knows the size of the reference, it can calculate distance misalignment based on the pixel misalignment. The system then adjusts endoscope parameters (e.g., external parameters) to correct or compensate for the distance misalignment.

[0020] In some implementations, the medical system offers several technical advantages. For example, by adjusting the parameters of the endoscope, the system compensates for distance misalignment, allowing images from the endoscope to be used for measurements at the surgical site (e.g., depth measurements, distance measurements, etc.). As another example, the system improves the accuracy of measurements performed using images from the endoscope, which improves patient health and safety.

[0021] In some examples, one or more components of a medical system may be implemented as a computer-assisted surgical system. However, it should be understood that the medical system can be implemented as any type of medical system (e.g., digital reference system, anatomical structure detection system, and clinical guidance system). Figure 1A An example computer-assisted surgical system 100 is shown that can implement some of the features described herein.

[0022] Surgical system 100 may include manipulator component 102, user control device 104, and auxiliary device 106, all of which are communicatively coupled to each other. A medical team may utilize surgical system 100 to perform computer-assisted medical procedures or other similar operations on the body of patient 108 or on any other body that may serve a particular implementation. The medical team may include a first user 110-1 (e.g., a surgeon during the procedure), a second user 110-2 (e.g., a patient-side assistant), a third user 110-3 (e.g., another assistant, nurse, intern, etc.), and a fourth user 110-4 (e.g., an anesthesiologist during the procedure), all of whom may be collectively referred to as user 110, and each user may control surgical system 100, interact with surgical system 100, or otherwise become a user of surgical system 100. More, fewer, or alternative users may be present during the medical procedure, as may be required to serve a particular implementation. For example, the team composition may differ for different medical or non-medical procedures and may include users with different roles.

[0023] although Figure 1A The illustration depicts a minimally invasive medical procedure, such as a minimally invasive surgical procedure, but it will be understood that the surgical system 100 can be similarly used to perform open medical procedures or other types of operations. For example, it can also perform operations such as exploratory imaging, simulated medical procedures for training purposes, and / or other operations.

[0024] Manipulator assembly 102 may include one or more instruments that can be coupled to one or more manipulator arms 112 (e.g., manipulator arms 112-1 to 112-4). These instruments can be used for computer-assisted surgical procedures performed on patient 108 (e.g., by insertion into and manipulation within patient 108 at least partially). While manipulator assembly 102 is depicted and described herein as comprising four manipulator arms 112, manipulator assembly 102 may include a single manipulator arm 112 or any other number of manipulator arms as may serve a particular implementation. Although Figure 1A The example shows manipulator arm 112 as a robotic manipulator arm, but in some examples, one or more instruments may be partially or fully manually controlled, for example by being held and manually controlled by a person. These partially or fully manually controlled instruments may be used in conjunction with computer-aided instruments coupled to manipulator arm 112 or as alternative instruments.

[0025] During medical procedures, the user control device 104 facilitates remote operation control of the manipulator arm 112 and instruments attached to it by the user 110-1. To this end, the user control device 104 can provide the user 110-1 with images of the operating area associated with the patient 108, such as images captured by an imaging device. The manipulator arm 112 or any instrument coupled to it can mimic the dexterity of the user 110-1's hand, wrist, and fingers across multiple degrees of freedom. In this way, the user 110-1 can visually perform procedures (e.g., incision procedures, suturing procedures, etc.) using one or more of the manipulator arms 112 or any instruments coupled to them.

[0026] The assistive device 106 may include one or more computing devices that perform assistive functions to support the process, such as providing air, electrocautery, illumination, or other energy to components of the imaging device, image processing, or the coordinated surgical system 100. In some examples, the assistive device 106 may include a display monitor 114 that displays one or more user interfaces, or graphical or textual information, to support the process. In some cases, the display monitor 114 may be a touchscreen display that provides user input functionality. Enhanced content provided by a region-based enhancement system may be similar to or different from the content associated with the display monitor 114 or one or more display devices in the operating area (not shown).

[0027] The controller assembly 102, user control device 104, and auxiliary device 106 can be communicatively coupled to each other in any suitable manner. For example, the controller assembly 102, user control device 104, and auxiliary device 106 can be communicatively coupled via a control line 116, which can represent any wired or wireless communication link that can serve a particular implementation. For this purpose, the controller assembly 102, user control device 104, and auxiliary device 106 can each include one or more wired or wireless communication interfaces, such as one or more LAN interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0028] Figure 1B An example manipulator component 102 is shown. (See example...) Figure 1B As shown, the manipulator assembly 102 includes a base 118, manipulator arms 112-1, 112-2, 112-3, and 112-4. Each manipulator arm 112-1, 112-2, 112-3, and 112-4 is pivotally coupled to the base 118. Although the base 118 may include casters to allow for easy movement, in some embodiments, the manipulator assembly 102 is fixedly mounted to the floor, ceiling, console, structural frame, etc.

[0029] In a typical procedure, two of the manipulator arms 112-1, 112-2, 112-3, or 112-4 hold surgical instruments, and a third manipulator arm holds a stereoscopic endoscope. The remaining manipulator arms are available to allow the introduction of other instruments at the work site. Alternatively, the remaining manipulator arms can be used to introduce another endoscope or another image-capturing device, such as an ultrasound transducer, into the work site.

[0030] Each of manipulator arms 112-1, 112-2, 112-3, and 112-4 may be formed by links coupled together and manipulated by actuable joints. Each of manipulator arms 112-1, 112-2, 112-3, and 112-4 may include a mounting arm and a device manipulator. The mounting arm positions the device it holds such that a pivot point is located at its entry port into the patient's body. The device manipulator can then manipulate the device it holds such that the held device can pivot about the pivot point, be inserted into and retracted from the entry port, and rotate about its axis. Each of manipulator arms 112-1, 112-2, 112-3, and 112-4 may include sensors (e.g., kinematic sensors, position sensors, accelerometers, etc.) for detecting or tracking the movement of manipulator arms 112-1, 112-2, 112-3, and 112-4. For example, these sensors can detect how far or how fast the manipulator arms 112-1, 112-2, 112-3, or 112-4 move in a certain direction.

[0031] Figure 1C An example user control device 104 is shown. User control device 104 includes a stereoscopic vision display 120, allowing a user to stereoscopically observe the surgical site from images captured by a stereoscopic camera device of manipulator assembly 102. A left eyepiece 122 and a right eyepiece 124 are provided in the stereoscopic vision display 120, allowing the user to view the left and right displays within the display 120 with their left and right eyes, respectively. While typically observing images of the surgical site on a suitable observer or display, the surgeon performs the surgical procedure on the patient by manipulating a main control input device, which in turn controls the movement of robotic instruments.

[0032] User control device 104 also includes a left input device 126 and a right input device 128, which a user can grasp with his / her left and right hands, respectively, to manipulate a device (e.g., a surgical instrument) held by the manipulator arms 112-1, 112-2, 112-3, and 112-3 of manipulator assembly 102, preferably in six or more degrees of freedom (“DOF”). A foot pedal 130 with toe and heel controls is provided on user control device 104, allowing the user to control the movement and / or actuation of devices associated with the foot pedal.

[0033] A processing unit 132 is provided in the user control device 104 for control and other purposes. The processing unit 132 performs various functions within the surgical system 100. One function performed by the processing unit 132 may be to convert and transmit the mechanical movements of input devices 126 and 128 to actuate their corresponding joints in their associated manipulator arms 112-1, 112-2, 112-3, and 112-4, enabling the surgeon to effectively manipulate devices such as surgical instruments. Another function of the processing unit 132 may be to implement the methods described herein, cross-coupled control logic, and controllers or processors. The auxiliary device 106 may include the processing unit 132 performing the functions or actions described herein. The processing unit 132 may include a processor and memory performing the functions described herein.

[0034] The processor may include any electronic circuit system, including but not limited to one or a combination of a microprocessor, microcontroller, application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), and / or state machine, communicatively coupled to memory and controlling the operation of user control device 104 and / or auxiliary device 106. The processor may be 8-bit, 16-bit, 32-bit, 64-bit, or have any other suitable architecture. The processor may include: an arithmetic logic unit (ALU) for performing arithmetic and logical operations; processor registers for supplying operands to the ALU and storing the results of ALU operations; and a control unit for fetching instructions from memory and executing the instructions by directing the coordinated operation of the ALU, registers, and other components. The processor may include other hardware that operates the software to control and process information. The processor executes software stored in memory to perform any of the functions described herein. The processor controls the operation and management of user control device 104 or auxiliary device 106 by processing information (e.g., information received from user control device 104, manipulator assembly 102, auxiliary device 106, and / or memory). A processor is not limited to a single processing device and may include multiple processing devices contained in the same device or computer or distributed across multiple devices or computers. If multiple processing devices jointly perform a set of functions or actions, the processor is considered to perform that set of functions or actions, even if different processing devices perform different functions or actions within that set.

[0035] Figure 2A An example computer-assisted surgical system 200 is shown that can implement some of the features described herein. The surgical system 200 can be used for, for example, surgery, diagnosis, treatment, biopsy, or non-medical procedures. Figure 2AAs shown, a surgical system 200 (which may be a robot-assisted surgical system) includes one or more manipulator assemblies 202 for operating one or more medical device systems 204 while performing various procedures on a patient P positioned on a table T in a medical environment. For example, the manipulator assembly 202 may drive the movement of a catheter or end effector, apply treatment to target tissue, and / or manipulate control members. The manipulator assembly 202 may be a teleoperated assembly, a non-teleoperated assembly, or a hybrid teleoperated and non-teleoperated assembly, having selectable degrees of freedom of motion that can be motorized and / or teleoperated, and selectable degrees of freedom of motion that can be de-motorized and / or non-teleoperated. An operator input system 206, which may be internal or external to the medical environment, typically includes one or more control devices for controlling the manipulator assembly 202. The manipulator assembly 202 supports the medical device system 204 and may optionally include multiple actuators or motors that drive inputs on the medical device system 204 in response to commands from the control system 212. The actuator may optionally include a drive system that, when coupled to the medical device system 204, can advance the medical device system 204 into a natural orifice or surgically created anatomical orifice. Other drive systems can move the distal end of the medical device in multiple degrees of freedom, which may include three linear degrees of freedom (e.g., linear motion along the x, y, and z Cartesian axes) and three rotational degrees of freedom (e.g., rotation about the x, y, and z Cartesian axes). The manipulator assembly 202 may support a variety of other systems for irrigation, treatment, or other purposes. Such systems may include fluid systems (e.g., reservoirs, heating / cooling elements, pumps, and valves), generators, lasers, interrogators, and ablation components.

[0036] The surgical system 200 also includes a display system 210 for displaying images or representations of the surgical site, and a medical device system 204. The image or representation is generated by an imaging system 209, which may include an endoscopic imaging system. The display system 210 and operator input system 206 can be configured such that an operator can use remote presence sensing to control the medical device system 204 and operator input system 206. A graphical user interface can be displayed on the display system 210 and / or on a separately planned workstation display system.

[0037] In some examples, imaging system 209 includes an endoscopic imaging system having components integrally or removably coupled to medical device system 204. However, in some examples, a separate imaging device, such as an endoscope, attached to a separate manipulator assembly may be used with medical device system 204 to image a surgical site. Imaging system 209 may be implemented as hardware, firmware, software, or a combination thereof, interacting with or otherwise executed by one or more computer processors, which may include processor 214 of control system 212.

[0038] Surgical system 200 also includes sensor system 208. Sensor system 208 may include position / positioning sensor systems (e.g., actuator encoders or electromagnetic (EM) sensor systems) and / or shape sensor systems (e.g., fiber optic shape sensors) for determining the position, orientation, rate, velocity, pose, and / or shape of medical device system 204. These sensors may also detect the position, orientation, or pose of patient P on the table T. For example, sensors may detect whether patient P is facing down or up. As another example, sensors may detect the direction in which patient P's head is pointing. Sensor system 208 may also include temperature, pressure, force, or contact sensors, etc.

[0039] The surgical system 200 may also include a control system 212, which includes at least one memory 216 and at least one computer processor 214 for controlling the medical device system 204, the operator input system 206, the sensor system 208, and the display system 210. The control system 212 includes programmed instructions (e.g., a non-transitory machine-readable medium storing the instructions) to perform procedures using the surgical system 200, including for navigation, steering, imaging, engaging feature deployment or retraction, applying treatment to target tissue (e.g., via energy application), etc.

[0040] The control system 212 may also include a virtual visualization system to provide navigational assistance to the operator O when controlling the medical device system 204 during an image-guided surgical procedure. Virtual navigation using the virtual visualization system may be based on reference to acquired preoperative or intraoperative datasets of the anatomical pathway. The virtual visualization system processes images of the surgical site imaged using imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermal imaging, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, etc. The control system 212 uses preoperative images to locate target tissue (using visual imaging techniques and / or by receiving user input) and create a preoperative plan that includes an optimal first positioning for performing the treatment. The preoperative plan may include, for example, planned dimensions for expanding the expandable device, treatment duration, treatment temperature, and / or multiple deployment positions.

[0041] Processor 214 is any electronic circuit system, including but not limited to: one or a combination of a microprocessor, microcontroller, application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), and / or state machine, communicatively coupled to memory 216 and controlling the operation of control system 212. Processor 214 may be 8-bit, 16-bit, 32-bit, 64-bit, or have any other suitable architecture. Processor 214 may include: an arithmetic logic unit (ALU) for performing arithmetic and logical operations; processor registers for supplying operands to the ALU and storing the results of ALU operations; and a control unit for fetching instructions from memory and executing the instructions by directing the coordinated operation of the ALU, registers, and other components. Processor 214 may include other hardware that operates software to control and process information. Processor 214 executes software stored on memory 216 to perform any of the functions described herein. Processor 214 controls the operation and management of control system 212 by processing information (e.g., information received from manipulator component 202, operator input system 206, and memory 216). Processor 214 is not limited to a single processing device, and may include multiple processing devices contained in the same device or computer or distributed across multiple devices or computers. If multiple processing devices jointly perform a set of functions or actions, processor 214 is considered to perform that set of functions or actions, even if different processing devices perform different functions or actions within that set.

[0042] Memory 216 may permanently or temporarily store data, operating software, or other information of processor 214. Memory 216 may include any or a combination of volatile or non-volatile local or remote devices suitable for storing information. For example, memory 216 may include random access memory (RAM), read-only memory (ROM), magnetic storage devices, optical storage devices, or any other suitable information storage devices or combinations thereof. Software refers to a collection of any suitable instructions, logic, or code contained in a computer-readable storage medium. For example, software may be contained in memory 216, a disk, a CD, or a flash drive. In certain embodiments, software may include an application executable by processor 214 to perform one or more of the functions described herein. Memory 216 is not limited to a single memory and may include multiple memories contained in the same device or computer or distributed across multiple devices or computers. If multiple memories jointly store a set of data, operating software, or information, memory 216 is considered to store that set of data, operating software, or information, even if different memories store different portions of the data, operating software, or information in that set.

[0043] Figure 2B An example medical device system 204 in a surgical system 200 is shown. In some embodiments, the medical device system 204 is used for image-guided medical procedures. For example, the medical device system 204 may be used for non-telescopic exploratory procedures or procedures involving conventionally manually operated medical devices such as endoscopes.

[0044] The medical device system 204 includes an elongated flexible device 220, such as a flexible catheter or endoscope (e.g., a gastroscope, bronchoscope), coupled to a drive unit 222. The elongated flexible device 220 includes a flexible body 224 having a proximal end 226 and a distal or terminal portion 228. In some embodiments, the flexible body 224 has an outer diameter of approximately 14 mm to 20 mm. Other flexible bodies may have larger or smaller outer diameters. The flexible body 224 is of suitable length to reach portions of anatomical structures, such as the lungs, sinuses, throat, or upper or lower gastrointestinal region, when inserted into a patient's mouth or nasal cavity.

[0045] The medical device system 204 includes a tracking system 230 for determining the position, orientation, rate, velocity, pose, and / or shape of the distal end 228 and / or one or more segments 232 along the flexible body 224 using one or more sensors and / or imaging devices. The entire length of the flexible body 224 between the distal end 228 and the proximal end 226 is effectively divided into segments 232. The tracking system 230 is implemented as hardware, firmware, software, or a combination thereof, and interacts with or is otherwise executed by one or more computer processors, which may include the processor 214 of the control system 212.

[0046] Tracking system 230 uses shape sensor 234 to track one or more of distal end 228 and / or segment 232. In some embodiments, tracking system 230 uses position sensor system 236, such as an electromagnetic (EM) sensor system, to track distal end 228. In some examples, position sensor system 236 measures six degrees of freedom (e.g., three position coordinates x, y, and z, and three orientation angles indicating pitch, yaw, and roll of the base point) or five degrees of freedom (e.g., three position coordinates x, y, and z, and two orientation angles indicating pitch and yaw of the base point).

[0047] The flexible body 224 includes one or more channels 238, which are sized and configured to receive one or more medical devices 240. In some embodiments, the flexible body 224 includes two channels 238 for a single device 240; however, a different number of channels 238 may be provided. Figure 2C It shows Figure 2B Example section of medical device system 204. (e.g.) Figure 2CAs shown, medical device 240 extends through flexible body 224. In some embodiments, medical device 240 can be used for procedures and aspects of procedures, such as surgery, biopsy, ablation, mapping, imaging, illumination, irrigation, or aspiration. Medical device 240 unfolds through channel 238 of flexible body 224 and is used at a targeted location within an anatomical structure. Medical device 240 includes, for example, image capturing devices, biopsy instruments, ablation instruments, catheters, laser ablation fibers, and / or other surgical, diagnostic, or therapeutic tools. Medical tools include end effectors with a single working component, such as scalpels, blunt blades, lenses, optical fibers, electrodes, etc. Other end effectors include, for example, forceps, graspers, balloons, needles, scissors, clamps, etc. Other end effectors also include electrically activated end effectors, such as electrosurgical electrodes, transducers, sensors, imaging devices, etc. Medical device 240 is advanced from the opening of channel 238 to perform the procedure and then retracts into the channel when the procedure is complete. The medical device 240 is removed from the proximal end 226 of the flexible body 224 or along the flexible body 224 from another optional device port (not shown). The medical device 240 can be used with an image capturing device (e.g., an endoscopic camera device) also located within the elongated flexible device 220. Alternatively, the medical device 240 itself can be an image capturing device.

[0048] The medical device 240 additionally accommodates a cable, link, or other actuation control (not shown) extending between the proximal and distal ends to controllably bend the distal end of the medical device 240. The flexible body 224 also accommodates a cable, link, or other steering control (not shown) extending between the drive unit 222 and the distal end 228 to controllably bend the distal end 228, for example as shown by the dashed line 242 depicting the distal end 228. In some examples, at least four cables are used to provide independent “up-down” steering for controlling the pitch motion of the distal end 228 and “left-right” steering for controlling the yaw motion of the distal end 228. In embodiments where the medical device system 204 is actuated by a robot-assisted component, the drive unit 222 may include a drive input removably coupled to and receiving power from a drive element, such as an actuator, of the teleoperation component. In some embodiments, the medical device system 204 includes a gripping feature, a manual actuator, or other components for manually controlling the movement of the medical device system 204. Information from tracking system 230 can be sent to navigation system 244, where it is combined with information from visualization system 246 and / or preoperatively acquired models to provide real-time location information to doctors or other operators.

[0049] Figures 3 to 7 It shows the medical system (e.g., Figure 1A Surgical system 100 or Figure 2AThe example operation performed by the computer system in the surgical system 200. Typically, the computer system (which may be implemented in the user control device 104 and / or auxiliary device 106 of the surgical system 100 using a processing device 132, and / or in the control system 212 of the surgical system 200 using a processor 214 and a memory 216) detects misalignment between the camera devices of the endoscope and adjusts the parameters of the endoscope to resolve or compensate for the misalignment.

[0050] A computer system can be described as performing certain actions that may involve other components such as endoscopes, brackets, etc. (e.g., stopping the endoscope or bracket, capturing images, etc.). In these cases, it should be understood that the controller performs these actions by sending signals to other components that cause these components to perform these actions.

[0051] Figure 3 An example operation 300 for performing a measurement is shown. The computer system executes operation 300. (Example...) Figure 3 As shown, operation 300 involves an endoscope 302 (which can capture images). Endoscope 302 can be a stereoscopic endoscope including a left camera device 304 and a right camera device 306. These camera devices 304 and 306 can be offset from each other and can capture images of the object 308. For example, endoscope 302 can be positioned at the surgical site, and camera devices 304 and 306 can capture images of the anatomical object at the surgical site. Figure 3 In the example, the left camera device 304 captures the left image 310 of the object 308, and the right camera device 306 captures the right image 312 of the object 308.

[0052] Although the left camera device 304 and the right camera device 306 are positioned close to each other on the endoscope 302, they are in different physical locations and have different orientations. Therefore, the left camera device 304 and the right camera device 306 capture different images of the object 308. For example, the left image 310 and the right image 312 may show different perspectives of the object 308. A portion of the object 308 may appear in a certain set of pixels in the left image 310, and the same portion of the object 308 may appear in a slightly different set of pixels in the right image 312.

[0053] The computer system can use the left image 310, the right image 312, and parameters 314 to determine the measurement result 316 at the surgical site. For example, the computer system can use the left image 310, the right image 312, and parameters 314 to measure depth and / or distance. Parameters 314 may include external parameters (e.g., external camera calibration parameters) and internal parameters (e.g., internal camera calibration parameters) of the endoscope 302. External parameters may indicate the pose of the left camera 304 and the right camera 306. For example, external parameters may include rotation matrices and / or translation vectors indicating the position and / or orientation of the left camera 304 and / or the right camera 306. Internal parameters may indicate how the left camera 304 and the right camera 306 capture the left image 310 and the right image 312. For example, internal parameters may include optical axis, focal length, principal point, skew coefficient, etc.

[0054] The computer system uses parameter 314 to determine measurement result 316 based on left image 310 and right image 312. For example, the computer system can use intrinsic and extrinsic parameters to convert pixels in a two-dimensional (2D) plane of left image 310 and / or right image 312 into three-dimensional (3D) coordinates in the world. The computer system can then use the 3D coordinates to determine measurement result 316, such as depth and distance.

[0055] As previously described, during operations at different surgical sites, the endoscope 302 may be subjected to various temperatures and / or pressures. These temperatures and pressures may cause displacement or other physical distortions in the left camera unit 304 and / or the right camera unit 306. Due to these distortions, the left image 310 and / or the right image 312 generated by the left camera unit 304 and / or the right camera unit 306 are also distorted. For example, an object 308 in the left image 310 and / or the right image 312 may occupy different sets of pixels in the left image 310 and / or the right image 312, which may lead to inaccurate measurement results 316. When the computer system uses the left image 310 and the right image 312 to calculate the measurement results 316, the displacement and / or distortion in the left image 310 and / or the right image 312 may cause the computer system to calculate a depth and / or distance that is greater or less than the actual depth or distance. When a user moves surgical instruments at the surgical site based on inaccurate depth and / or distance, this can lead to damage or injury at the surgical site.

[0056] Figures 4A to 4DAn example operation for adjusting the endoscopic imaging device is shown. Typically, the computer system performs these operations to determine distortions in the endoscopic imaging device before the endoscope reaches the surgical site. The computer system can then adjust the endoscope's parameters to account for these distortions. When the computer system subsequently uses the adjusted parameters to calculate measurements at the surgical site (e.g., distance, depth, etc.) based on the images captured by the endoscope, it produces more accurate measurements, which can improve patient health.

[0057] Figure 4A An example operation 400, performed by a computer system, to move an endoscope 302 through a cannula 402. Typically, the computer system navigates the endoscope 302 toward the surgical site by moving it through the cannula 402, which may resemble a tube. The operator can position the endoscope 302 on a bracket 404 located within the cannula 402. The computer system can then move the bracket 404 through the cannula 402 to move the endoscope 302 through the cannula 402. When the endoscope 302 reaches the end of the cannula 402, the endoscope 302 is exposed from the cannula 402 and enters the surgical site.

[0058] Figure 4B An example operation 420, performed by a computer system to calibrate endoscope 302, is shown. Cannula 402 has markings printed on its inner wall. These markings may be encountered by endoscope 302 as it moves across cannula 402 on bracket 404. Figure 4B In the example, endoscope 302 encounters a first mark 422 printed on the inner wall of cannula 402. Additionally, a second mark 424 (e.g., a square or rectangle, an arrangement of dots, an April label, etc.) used as a reference is printed on the inner wall of cannula 402 at a predetermined or preset distance from the first mark 422. Typically, the second mark 424 can be any visual label that can provide subpixel coordinates in image space. When endoscope 302 encounters the first mark 422, the computer system can stop the carriage 404 and / or endoscope 302 to prevent further downward movement of the endoscope 302 along cannula 402. For example, when the computer system detects the first mark 422 at a specific location in a video or image captured by endoscope 302, the computer system can stop the carriage 404 and / or endoscope 302. In this way, the computer system stops the endoscope 302 at a predetermined or preset distance from the second mark 424.

[0059] In some embodiments, a plurality of second marks 424 are printed on the inner wall of the sleeve 402. For example, the second marks 424 (e.g., a plurality of April labels) may be printed in a ring on the inner wall of the sleeve 402. Additionally, any type of ink may be used to print the second marks 424. For example, ultraviolet marking ink may be used to print the second marks 424.

[0060] The computer system then uses endoscope 302 to capture an image of the second marker 424 from a predetermined or preset distance. Figure 4B In the example, the computer system uses the camera device of endoscope 302 to capture a left image 426 and a right image 428 of the second marker 424. The left image 426 can be captured by the left camera device of endoscope 302, and the right image 428 can be captured by the right camera device of endoscope 302.

[0061] The computer system then adjusts the left image 426 by 430 and the right image 428 by 432. Adjustments 430 and 432 can dedistort the left and right images 426 and 428. For example, the second mark 424 shown in the left and right images 426 and 428 may include distortion due to the shape of the lenses on the left and right camera devices. Adjustments 430 and 432 can remove some of this curvature, which straightens the lines and produces a more accurate depiction of the second mark 424. In some embodiments, adjustments 430 and 432 can also shift or move the second mark 424 in the left and / or right images 426 and 428 to account for different positions and / or orientations of the left and right camera devices. In this way, the computer system places the left and right images 426 and 428 in the same image plane.

[0062] As an example, the computer system can detect the edges and / or corners of the second mark 424 in the left image 426 and the right image 428. The computer system then straightens the edges and / or corners of the second mark 424. In some cases, the computer system may not distort or straighten other parts of the second mark 424.

[0063] In embodiments where multiple second marks 424 are printed on the inner wall of the cannula, the left image 426 and right image 428 may show the multiple second marks 424. The computer system can detect the edges and / or corners of these second marks 424 in the left image 426 and right image 428. These edges and / or corners can provide the computer system with sufficient information to calibrate the endoscope.

[0064] Figure 4CAn example operation 440, performed by a computer system to adjust parameters of an endoscope, is illustrated. After adjusting the left image 426 and the right image 428, the computer system begins with the left image 426 and the right image 428 containing the second marker 424. The computer system compares the left image 426 and the right image 428 to determine a misalignment 442 between the left image 426 and the right image 428. For example, the computer system may determine the pixels 444 occupied by the second marker 424 in the left image 426 and the right image 428. The computer system may compare these pixels 444 to determine the differences in the pixels 444 occupied by the second marker 424 between the left image 426 and the right image 428. The misalignment 442 may indicate this difference. For example, the misalignment 442 may indicate horizontal differences and vertical differences in the pixels 444. Therefore, the misalignment 442 may indicate the number of pixels 444 by which the second marker 424 in the left image 426 is offset relative to the second marker 424 in the right image 428.

[0065] In some implementations, the computer system converts misalignment 442 into translational misalignment and rotational misalignment. Translational misalignment can be a translational component of misalignment 442, and rotational misalignment can be a rotational component of misalignment 442. For example, the translational component can indicate the magnitude (e.g., measured in pixels) of misalignment 442 along a direction axis in image space (e.g., along the horizontal or vertical axis of image space). The rotational component can indicate the angular component of misalignment 442 (e.g., the angular offset relative to the horizontal or vertical axis of image space).

[0066] The computer system also provides a dimension 446 for the second mark 424. For example, dimension 446 may be a parameter or input provided to the computer system during sleeve operation. Dimension 446 may indicate the physical dimensions of the second mark 424 printed in the sleeve. For example, dimension 446 may indicate the physical dimensions (e.g., length and width) of the second mark 424. The computer system uses dimension 446 and pixels 444 to calculate the per-pixel dimension 448. For example, pixels 444 may indicate a plurality of pixels in the horizontal direction and a plurality of pixels in the vertical direction occupied by the second mark 424 in the left image 426 and / or the right image 428. The computer system may divide dimension 446 by the number of pixels 444 to determine the per-pixel dimension 448. For example, the computer system may divide the horizontal dimension indicated by dimension 446 by the number of pixels in the horizontal direction, and the computer system may divide the vertical dimension indicated by dimension 446 by the number of pixels in the vertical direction.

[0067] The computer system then uses misalignment 442 and per-pixel size 448 to determine misalignment distance 450. The computer system can multiply misalignment 442 (which is pixel misalignment) by per-pixel size 448 to produce misalignment distance 450. As an example, the computer system can multiply the number of horizontally misaligned pixels indicated by misalignment 442 by the horizontal per-pixel size indicated by per-pixel size 448 to produce horizontal misalignment distance, and the computer system can multiply the number of vertically misaligned pixels indicated by misalignment 442 by the vertical per-pixel size indicated by per-pixel size 448 to produce vertical misalignment distance. Therefore, misalignment distance 450 is the physical distance represented by pixel misalignment between the second mark 424 in the left image 426 and the second mark 424 in the right image 428. Per-pixel size 448 effectively converts misalignment 442 in image or pixel space into misalignment distance 450 in world or global space.

[0068] In an implementation where the computer system determines translation misalignment and rotation misalignment, the computer system can determine the misalignment distance 450 by multiplying the translation misalignment by the pixel size 448 and the rotation misalignment by the cosine of the rotation misalignment.

[0069] The computer system determines adjustments 452 to the endoscope's parameters 314 to correct for misalignment 450. Adjustment 452 may include adjustments to external parameters of the endoscope. By adjusting these external parameters, the computer system calibrates how 2D pixel coordinates in the image captured by the endoscope are converted to 3D global coordinates to account for or correct for misalignment 450. In this way, when the computer system uses parameters 314 to measure distance or depth relative to the image captured by the endoscope, the measured distance or depth is accurate, and the misalignment is corrected.

[0070] In some implementations, the computer system compares the misalignment distance 450 to one or more thresholds 454 (e.g., a horizontal threshold and a vertical threshold) to determine whether the computer system should adjust parameter 314. If the misalignment distance 450 decreases below the threshold 454, the computer system can determine that the misalignment distance 450 is within tolerance and maintain parameter 314. If the misalignment distance 450 exceeds the threshold 454, the computer system can adjust parameter 314 452 to correct the misalignment distance 450.

[0071] In this way, the computer system adjusts the endoscope's parameters 314 (e.g., external parameters) to correct for physical distortions experienced by the endoscope's imaging device. By making these adjustments, the computer system uses the images captured by the endoscope to produce more accurate measurements (e.g., distance measurements and / or depth measurements). More accurate measurements can reduce the chance of injury or damage at the surgical site during the procedure.

[0072] In some implementations, the computer system implements a threshold 454 indicating whether an endoscope should be used to measure distance or depth. For example, if the misalignment distance 450 exceeds the threshold 454, the computer system can determine that the endoscope cannot be calibrated to correct the misalignment and that a different endoscope should be used. If the endoscope continues to be used, the computer system can prevent measurement applications (e.g., digital ruler applications or fluorescence imaging applications) from being loaded or used.

[0073] Figure 4D An example operation 460 for calibrating endoscope 302, performed by a computer system, is shown. Typically, the computer system can perform operation 460 when the markings on the inner wall of the cannula are unavailable.

[0074] like Figure 4D As shown, the bracket 404 can move the endoscope 302 through the cannula 402. The bracket 404 can stop the endoscope 302 at position 462 in the cannula 402. Position 462 can be a predetermined position at a predetermined or preset distance from the end of the cannula 402. In some cases, position 462 can be specified by software in a computer system (e.g., software stop), and the software can stop the bracket 404 when the bracket reaches position 462. An opening 464 (e.g., a circular opening) is provided at the end of the cannula 402, which serves as a reference. When the bracket 404 stops at position 462, the endoscope 302 can be at a predetermined or preset distance from the opening 464.

[0075] The computer system then uses endoscope 302 to capture an image of opening 464 from a predetermined or preset distance. Figure 4D In the example, the computer system uses the camera device of endoscope 302 to capture a left image 466 and a right image 468 of opening 464. The left image 466 can be captured by the left camera device of endoscope 302, and the right image 468 can be captured by the right camera device of endoscope 302.

[0076] The computer system then adjusts the left image 466 by 470 and the right image 468 by 472. Adjustments 470 and 472 can dedistort the left and right images 466 and 468. For example, the opening 464 shown in the left and right images 466 and 468 may include distortion due to the shape of the lenses on the left and right camera devices. Adjustments 470 and 472 can remove some of this curvature, which straightens or smooths the boundary and produces a more accurate depiction of the opening 464. In some embodiments, adjustments 470 and 472 can also shift or move the opening 464 in the left and / or right images 466 and 468 to account for different positions and / or orientations of the left and right camera devices. In this way, the computer system places the left and right images 466 and 468 in the same image plane.

[0077] As an example, the computer system can detect the boundary of opening 464 in the left image 466 and the right image 468. The computer system then smooths the boundary of opening 464. In some cases, the computer system may not distort or smooth other parts of opening 464.

[0078] The computer system can then use left image 466 and right image 468 (e.g., instead of left image 426 and right image 428) to perform. Figure 4C The operation 440 shown is used to adjust the endoscope 302. For example, the computer system can determine a misalignment between the left image 466 and the right image 468. The computer system can then determine an adjustment to the parameters of the endoscope 302 that will reduce or correct the misalignment. The computer system can determine the adjustment of the parameters using a predetermined or current distance between the endoscope 302 and the end of the cannula 402, as well as the size of the opening 464. In this way, the computer system can correct misalignment in the endoscope 302 even when there are no markings on the inner wall of the cannula 402. Alternatively, the computer system can use the opening 464 at the end of the cannula 402 as a substitute for markings.

[0079] In some implementations, the computer system can stop the bracket 404 and endoscope 302 at multiple locations within the cannula 402. Each of these locations can be at a different distance from the opening 464. For example, the computer system can stop the endoscope 302 at location 462 and at the end of the cannula 402 itself. The endoscope 302 can then capture an image of the opening 464 at the end of the cannula 402. The computer system can use multiple sets of images of the opening 464 to calibrate the endoscope 302.

[0080] Figure 5An example operation 500 for adjusting an endoscopic camera device is illustrated. Typically, the computer system performs operation 500 to adjust the brightness of the endoscope. The computer system begins with a left image 426 from the left camera device of the endoscope and a right image 428 from the right camera device of the endoscope. The computer system analyzes the left image 426 and the right image 428 to determine the contrast 502 and / or intensity 504 of the left image 426 and the right image 428. Contrast 502 is a measure indicating the difference in brightness or color that makes objects in the left image 426 and the right image 428 visible against a background of different brightness or color. Intensity 504 is a measure indicating the amount of light reflected by objects in the left image 426 and the right image 428. Both measures can indicate the ease with which objects in the left image 426 and the right image 428 can be distinguished, as well as the ease with which portions of the left image 426 and the right image 428 (e.g., the boundaries between the left image 426 and the right image 428) can be distinguished.

[0081] The computer system determines an adjustment 506 to the brightness 508 of the endoscope based on contrast 502 and intensity 504. The computer system can compare contrast 502 and / or intensity 504 to one or more thresholds that indicate whether brightness 508 should be increased or decreased. For example, if contrast 502 and / or intensity 504 decreases below certain thresholds, the computer system can determine an adjustment 506 to increase brightness 508. If contrast 502 and / or intensity 504 exceeds a specific threshold, the computer system can determine an adjustment 506 to decrease brightness 508. The magnitude of the adjustment 506 can depend on the difference between contrast 502 and / or intensity 504 and their respective thresholds. The greater the difference, the greater the magnitude of the adjustment 506, and vice versa.

[0082] Adjusting the endoscope's brightness 508 adjusts the amount of light emitted by a lamp (e.g., a light-emitting diode) positioned on the endoscope. Increasing the brightness 508 increases the amount of emitted light, and decreasing the brightness 508 reduces the amount of emitted light. By emitting more or less light, the computer system can increase or decrease the contrast 502 and / or intensity 504 in the image captured by the endoscope. In this way, the computer system can make it easier to distinguish objects appearing in the image and to differentiate different parts of the image from one another.

[0083] Figures 6A to 6C Example procedures for determining if a camera device is misaligned are shown. Typically, computer systems perform these procedures after the endoscope has passed through the cannula to the surgical site to determine whether further calibration of the endoscope is necessary.

[0084] Figure 6A Example operation 600 performed by a computer system is shown. For example... Figure 6AAs shown, endoscope 302 has passed through the cannula to reach surgical site 602. Subject 604 (e.g., anatomical object) is located at surgical site 602. Endoscope 302 is guided toward subject 604. The imaging device of endoscope 302 captures left image 606 and right image 608 of subject 604.

[0085] Figure 6B Example left image 606 or right image 608 are shown. (e.g.) Figure 6B As shown, images 606 / 608 illustrate object 604. Additionally, images 606 / 608 include a boundary 610 near the periphery of the image. Depending on the characteristics of the lens in the endoscope's imaging device, the boundary 610 may be circular, elliptical, or annular. The portion of image 606 / 608 between the boundary 610 and the periphery of image 606 / 608 may be black.

[0086] Figure 6C An example operation 620, performed by a computer system to determine whether further adjustments are needed to calibrate the endoscope, is illustrated. The computer system begins by analyzing boundaries 610 in an image from the endoscope (e.g., left image 606). The computer system compares this image to a reference image 622. Specifically, the computer system compares the boundaries 610 in the image to a reference boundary 624 in the reference image 622. The reference image 622 may have already been captured by the endoscope when the endoscope is confirmed to be calibrated. The computer system then stores the reference image 622 for future use.

[0087] The computer system compares boundary 610 with reference boundary 624 to determine misalignment 626. For example, the computer system may determine whether boundary 610 occupies the same number of pixels in the image as reference boundary 624 in reference image 622. Misalignment 626 may indicate the number of pixels in the image that boundary 610 differs from reference boundary 624 in reference image 622 (e.g., the number of pixels in the horizontal direction and / or the number of pixels in the vertical direction).

[0088] The computer system then compares the misalignment 626 to one or more thresholds 628. For example, the computer system may compare the number of pixels in the horizontal direction indicated by the misalignment 626 to a horizontal threshold, and the computer system may compare the number of pixels in the vertical direction indicated by the misalignment 626 to a vertical threshold. If the misalignment 626 decreases below the threshold 628, the computer system may determine that no further calibration of the endoscope is required.

[0089] If the misalignment 626 exceeds the threshold 628, the computer system determines that further calibration should be performed. The computer system generates an alert 630, which may include a message indicating that further calibration of the endoscope should be performed. The computer system sends the alert 630 to the user to remind them that calibration is necessary. The user can then use... Figures 4A to 4C The procedure shown involves retracting the endoscope into the cannula to recalibrate it in response.

[0090] Figure 7 This is a flowchart of an example method 700 for adjusting an endoscopic camera device. In some embodiments, a computer system (which can use...) Figures 1A to 1C The processing device 132 shown is implemented in the user control device 104 and / or auxiliary device 106 of the surgical system 100, and / or can be used Figures 2A to 2C The processor 214 and memory 216 shown are implemented in the control system 212 of the surgical system 200 to execute method 700. By executing method 700, the computer system calibrates the endoscope.

[0091] In box 702, the computer system moves the endoscope toward the surgical site through a cannula (which may resemble a tube). The cannula may have different markings printed on its inner wall. As the endoscope travels through the cannula, it may encounter these markings. The computer system can determine when the endoscope encounters a marking by detecting the markings in images or videos generated by the endoscope. The computer system can control a bracket positioned on the endoscope to move or stop the endoscope within the cannula.

[0092] In box 704, the computer system stops the endoscope. For example, the computer system stops the endoscope based on a first mark. The first mark is printed on the inner wall of the cannula and can indicate the stopping point of the endoscope. For example, the first mark can be a line or a box. When the computer system detects that the endoscope is positioned near or at the first mark based on images or video from the endoscope, the computer system can stop the endoscope (e.g., stop the holder). As another example, the computer system can stop the endoscope based on software stop. The computer system can detect when the endoscope is at a predetermined or preset distance from the end of the cannula and stop the endoscope at the predetermined or preset distance.

[0093] In box 706, the computer system uses an endoscope to capture a reference image. For example, when the endoscope is stopped, the reference may be a second mark printed on the inner wall of the cannula. The second mark may be an April label, and the computer system may know its dimensions (e.g., the physical dimensions of the second mark). Additionally, the second mark may be printed at a predetermined or preset distance from the first mark. As another example, the reference may be an opening (e.g., a circular opening) at the end of the cannula. The endoscope may include stereoscopic imaging devices (e.g., a left camera and a right camera) that generate multiple images (e.g., a left image and a right image) of the reference.

[0094] In box 708, the computer system adjusts endoscope parameters based on images from reference images received from the endoscope. For example, the computer system can compare reference images (e.g., left and right images) to determine pixel misalignment between references in the images. Because the computer system knows the physical dimensions of the references, it can use the dimensions of the references to convert the pixel misalignment into a physical misalignment distance. The computer system then adjusts the endoscope parameters to correct the misalignment distance. After adjusting the parameters, the computer system can then use those parameters to convert the 2D coordinates of pixels in the images from the endoscope into 3D global coordinates. The computer then uses the 3D global coordinates to determine the measurement outcome (e.g., the measured distance and / or depth).

[0095] In summary, medical systems (e.g., surgical systems) detect misalignment between the imaging devices of an endoscope and adjust the endoscope's parameters to correct the misalignment. Typically, as the endoscope is advanced toward the surgical site through a cannula (e.g., a tube), the system stops the endoscope within the cannula. The endoscope uses its imaging devices to capture images of references (e.g., markings on the inner wall of the cannula, openings at the end of the cannula, etc.). The system analyzes the images to determine pixel misalignment between the images. Because the system knows the dimensions of the reference, it can calculate distance misalignment based on the pixel misalignment. The system then adjusts the endoscope's parameters (e.g., external parameters) to correct or compensate for the distance misalignment.

[0096] The aspects, embodiments, or modules illustrated in this specification and the accompanying drawings should not be considered limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this specification and the claims. In some cases, well-known circuits, structures, or techniques have not been shown or described in detail so as not to obscure other features. Similar reference numerals in two or more figures denote the same or similar elements.

[0097] In this specification, specific details are set forth in relation to some embodiments consistent with this disclosure. Numerous specific details are set forth to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are intended to be illustrative and not restrictive. Other elements within the scope and spirit of this disclosure may be implemented by those skilled in the art, although not specifically described herein. Furthermore, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments unless otherwise specifically described or if one or more features would render the embodiment inoperable.

[0098] Furthermore, the terminology used in this specification is not intended to be limiting. For example, spatially related terms such as “below,” “under,” “lower,” “above,” “upper,” “near,” “far”, etc., may be used to describe the relationship of one element or feature to another element or feature as shown in the figures. In addition to the positions and orientations shown in the figures, these spatially related terms are also intended to include different positions (i.e., positioning) and orientations (i.e., rotational placement) of elements or their operations. For example, if one of the contents in the figures is flipped, the element described as being “below” or “under” other elements or features would then be “above” or “on” other elements or features. Thus, the exemplary term “below” can include both above and below positions and orientations. Devices may be oriented in other ways (rotated 90 degrees or in other orientations), and the spatially related descriptors used herein are interpreted accordingly. Similarly, descriptions of movement along and about various axes include various specific element positions and orientations. Additionally, unless the context otherwise indicates, the singular forms “a,” “an,” and “the” are also intended to include the plural forms. Furthermore, the terms "comprises," "comprising," and "includes" specify the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups. Components described as coupled may be electrically or mechanically directly coupled, or they may be indirectly coupled via one or more intermediate components.

[0099] Where feasible, elements described in detail with reference to one embodiment or module may be included in other embodiments or modules in which such elements are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment but not with reference to a second embodiment, the element may still be claimed as included in the second embodiment. Therefore, to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment or application may be incorporated into other embodiments or aspects unless otherwise specifically described, unless one or more elements would render one or more embodiments inoperable, or unless two or more of the elements provide conflicting functionality.

[0100] In some cases, well-known methods, processes, components, and circuits are not described in detail to avoid unnecessarily obscuring aspects of the implementation.

[0101] This disclosure describes the various devices, elements, and portions of computer-aided devices and elements in terms of their state in three-dimensional space. As used herein, the term "position" refers to the location of an element or portion of an element in three-dimensional space (e.g., three translational degrees of freedom along Cartesian x, y, and z coordinates). As used herein, the term "orientation" refers to the rotational placement of an element or portion of an element (three rotational degrees of freedom—e.g., roll, pitch, and yaw). As used herein, the term "shape" refers to a set of positions or orientations measured along the element. As used herein, and for devices with repositionable arms, the term "proximal" refers to a direction along its kinematic chain toward the base of the computer-aided device, and "distal" refers to a direction along the kinematic chain away from the base.

[0102] Various aspects of this disclosure are described with reference to computer-aided systems and apparatuses, which may include teleoperated, remotely controlled, autonomous, semi-autonomous, robotic, and other such systems and apparatuses. Furthermore, various aspects of this disclosure are described according to embodiments using medical systems, such as the da Vinci Surgical System or ion systems commercially available from Intuitive Surgical Inc. of Sunnyvale, California. However, those skilled in the art will understand that the aspects disclosed herein can be implemented and carried out in various ways, including robotic and non-robotic implementations (where applicable). The techniques described with reference to surgical instruments and methods can be used in other contexts. Therefore, the instruments, systems, and methods described herein can be used for humans, animals, parts of human or animal anatomy, industrial systems, general-purpose robotics, or teleoperation systems. As another example, the instruments, systems, and methods described herein can be used for non-medical purposes, including industrial applications, general-purpose robotic applications, sensing or manipulating non-tissue artifacts, cosmetic enhancements, imaging of human or animal anatomy, collecting data from human or animal anatomy, installing or disassembling systems, training medical or non-medical personnel, etc. Additional example applications include procedures for removing tissue from human or animal anatomy (with or without returning it to the anatomy) and procedures for human or animal cadavers. Furthermore, these techniques can also be used in medical treatments or diagnostic procedures, with or without surgical involvement.

[0103] Although illustrative embodiments have been shown and described, a wide range of modifications, alterations, and substitutions are contemplated in the foregoing disclosure, and in some cases, certain features of the embodiments may be employed without the need for corresponding use of other features. Many variations, substitutions, and modifications will be recognized by those skilled in the art. Therefore, the scope of this disclosure should be limited only by the appended claims, and it is appropriate that the claims be interpreted broadly and in a manner consistent with the scope of the embodiments disclosed herein.

[0104] As can be seen from the above description, the embodiments of the present invention also disclose the following technical solutions, including but not limited to:

[0105] Note 1. A system for adjusting an endoscope, the system comprising:

[0106] Memory; and

[0107] A controller, communicatively coupled to the memory, wherein the controller:

[0108] The endoscope is moved through the cannula;

[0109] The endoscope is stopped at the position in the cannula according to the first mark on the inner wall of the cannula;

[0110] The endoscope at the stated location is used to capture an image of the second mark on the inner wall of the cannula; and

[0111] The parameters of the endoscope are adjusted based on the second marker in the image.

[0112] Note 2. The system according to Note 1, wherein stopping the endoscope includes stopping the bracket on which the endoscope is positioned.

[0113] Note 3. The system according to Note 1, wherein the controller further prevents the endoscope from moving further through the cannula until the endoscope has captured the image.

[0114] Appendix 4. According to the system described in Appendix 1, adjusting the parameters includes:

[0115] The image is dedistorted to produce a dedistorted image;

[0116] The dedistorted images are compared to determine misalignment in the endoscope;

[0117] Adjustment is determined based on the size of the second mark and the misalignment in the endoscope; and

[0118] The parameters are adjusted as described.

[0119] Note 5. The system according to Note 4, wherein the parameters are external parameters of the endoscope.

[0120] Note 6. The system according to Note 1, wherein the controller further:

[0121] Determine at least one of the contrast or intensity of the second marker in the image; and

[0122] The brightness of the endoscope is adjusted based on at least one of the contrast or the intensity.

[0123] Note 7. The system according to Note 1, wherein the second mark includes a plurality of April tags.

[0124] Note 8. The system according to Note 1, wherein the first marker is positioned at a predetermined distance from the second marker.

[0125] Note 9. The system according to Note 1, wherein the controller further:

[0126] After the endoscope is positioned at the surgical site outside the cannula, the boundaries of the image captured by the endoscope are determined;

[0127] Determine the difference between the boundary of the image and the reference boundary; and

[0128] An alarm is generated based on the difference exceeding a threshold, indicating that the endoscope should be calibrated.

[0129] Appendix 10. The system according to Appendix 1, wherein the endoscope includes a first camera device and a second camera device, wherein the image includes a first image captured by the first camera device and a second image captured by the second camera device, and wherein adjusting the parameters includes:

[0130] It was determined that the pixels between the first image and the second image were misaligned;

[0131] Based on the size of the second marker, the pixel misalignment is converted into translational misalignment and rotational misalignment; and

[0132] The parameters are adjusted based on the translation misalignment and the rotation misalignment.

[0133] Note 11. The system according to Note 1, wherein the controller further measures distance based on adjusted parameters.

[0134] Appendix 12. A method for adjusting an endoscope, the method comprising:

[0135] The endoscope is moved through the cannula;

[0136] The endoscope is stopped at the position in the cannula according to the first mark on the inner wall of the cannula;

[0137] The endoscope at the stated location is used to capture an image of the second mark on the inner wall of the cannula; and

[0138] The parameters of the endoscope are adjusted based on the second marker in the image.

[0139] Note 13. The method according to Note 12, wherein stopping the endoscope includes stopping the bracket on which the endoscope is positioned.

[0140] Note 14. The method according to Note 12 further includes: preventing the endoscope from moving further through the cannula until the endoscope has captured the image.

[0141] Note 15. According to the method described in Note 12, adjusting the parameter includes:

[0142] The image is dedistorted to produce a dedistorted image;

[0143] The dedistorted images are compared to determine misalignment in the endoscope;

[0144] Adjustment is determined based on the size of the second mark and the misalignment in the endoscope; and

[0145] The parameters are adjusted as described.

[0146] Note 16. The method according to Note 15, wherein the parameter is an external parameter of the endoscope.

[0147] Note 17. The method described according to Note 12 further includes:

[0148] Determine at least one of the contrast or intensity of the second marker in the image; and

[0149] The brightness of the endoscope is adjusted based on at least one of the contrast or the intensity.

[0150] Note 18. The method according to Note 12, wherein the second tag includes a plurality of April tags.

[0151] Note 19. The method according to Note 12, wherein the first mark is positioned at a predetermined distance from the second mark.

[0152] Note 20. The method described according to Note 12 further includes:

[0153] After the endoscope is positioned at the surgical site outside the cannula, the boundaries of the image captured by the endoscope are determined;

[0154] Determine the difference between the boundary of the image and the reference boundary; and

[0155] An alarm is generated based on the difference exceeding a threshold, indicating that the endoscope should be calibrated.

[0156] Appendix 21. The method according to Appendix 12, wherein the endoscope includes a first camera device and a second camera device, wherein the image includes a first image captured by the first camera device and a second image captured by the second camera device, and wherein adjusting the parameters includes:

[0157] It was determined that the pixels between the first image and the second image were misaligned;

[0158] Based on the size of the second marker, the pixel misalignment is converted into translational misalignment and rotational misalignment; and

[0159] The parameters are adjusted based on the translation misalignment and the rotation misalignment.

[0160] Note 22. The method described according to Note 12 also includes measuring distance based on the adjusted parameters.

[0161] Appendix 23. A non-transitory machine-readable medium storing instructions for adjusting an endoscope, the instructions causing the processor, when executed by a processor, to:

[0162] Perform the method according to any one of Appendix 12 to 22.

[0163] This technology can also be implemented as follows:

[0164] Option 1. A system for adjusting an endoscope, the system comprising:

[0165] Memory; and

[0166] A controller, communicatively coupled to the memory, wherein the controller:

[0167] The endoscope is moved through the cannula;

[0168] Stop the endoscope at the position in the cannula;

[0169] The endoscope at the stated location is used to capture a reference image corresponding to the cannula; and

[0170] The parameters of the endoscope are adjusted based on the reference in the image.

[0171] Option 2. The system according to Option 1, wherein stopping the endoscope includes stopping the bracket on which the endoscope is positioned.

[0172] Option 3. The system according to Option 1, wherein the controller further prevents the endoscope from moving further through the cannula until the endoscope has captured the image.

[0173] Option 4. The system according to Option 1, wherein adjusting the parameters includes:

[0174] The image is dedistorted to produce a dedistorted image;

[0175] The dedistorted images are compared to determine misalignment in the endoscope;

[0176] Adjustments are determined based on the reference dimensions and the misalignment within the endoscope; and

[0177] The parameters are adjusted as described.

[0178] Option 5. The system according to Option 4, wherein the parameter is an external parameter of the endoscope.

[0179] Option 6. The system according to Option 1, wherein the controller further:

[0180] Determine at least one of the contrast or intensity of the reference in the image; and

[0181] The brightness of the endoscope is adjusted based on at least one of the contrast or the intensity.

[0182] Option 7. The system according to Option 1, wherein the reference includes a mark on the inner wall of the sleeve.

[0183] Option 8. The system according to Option 1, wherein the reference includes an opening at the end of the sleeve.

[0184] Option 9. The system according to Option 1, wherein stopping the endoscope at the position in the cannula is based on a mark on the inner wall of the cannula and at a predetermined distance from the reference.

[0185] Option 10. The system according to Option 1, wherein the controller further:

[0186] After the endoscope is positioned at the surgical site outside the cannula, the boundaries of the image captured by the endoscope are determined;

[0187] Determine the difference between the boundary of the image and the reference boundary; and

[0188] An alarm is generated based on the difference exceeding a threshold, indicating that the endoscope should be calibrated.

[0189] Option 11. The system according to Option 1, wherein the endoscope includes a first camera device and a second camera device, wherein the image includes a first image captured by the first camera device and a second image captured by the second camera device, and wherein adjusting the parameters includes:

[0190] It was determined that the pixels between the first image and the second image were misaligned;

[0191] Based on the reference size, the pixel misalignment is converted into translation misalignment and rotation misalignment; and

[0192] The parameters are adjusted based on the translation misalignment and the rotation misalignment.

[0193] Option 12. The system according to Option 1, wherein the controller further measures the distance based on the adjusted parameters.

[0194] Option 13. A method for adjusting an endoscope, the method comprising:

[0195] The endoscope is moved through the cannula;

[0196] Stop the endoscope at the position in the cannula;

[0197] The endoscope at the stated location is used to capture a reference image corresponding to the cannula; and

[0198] The parameters of the endoscope are adjusted based on the reference in the image.

[0199] Option 14. The method according to Option 13, wherein stopping the endoscope includes stopping the bracket on which the endoscope is positioned.

[0200] Option 15. The method according to Option 13 further includes: preventing the endoscope from moving further through the cannula until the endoscope has captured the image.

[0201] Option 16. The method according to Option 13, wherein adjusting the parameter includes:

[0202] The image is dedistorted to produce a dedistorted image;

[0203] The dedistorted images are compared to determine misalignment in the endoscope;

[0204] Adjustments are determined based on the reference dimensions and the misalignment within the endoscope; and

[0205] The parameters are adjusted as described.

[0206] Scheme 17 is based on the method described in Scheme 16, wherein the parameter is an external parameter of the endoscope.

[0207] Option 18. The method according to Option 13 further includes:

[0208] Determine at least one of the contrast or intensity of the reference in the image; and

[0209] The brightness of the endoscope is adjusted based on at least one of the contrast or the intensity.

[0210] Option 19. The method according to Option 13, wherein the reference includes a mark on the inner wall of the sleeve.

[0211] Option 20. A non-transitory machine-readable medium storing instructions for adjusting an endoscope, the instructions causing the processor, when executed by a processor, to:

[0212] The endoscope is moved through the cannula;

[0213] Stop the endoscope at the position in the cannula;

[0214] The endoscope at the stated location is used to capture a reference image corresponding to the cannula; and

[0215] The parameters of the endoscope are adjusted based on the reference in the image.

Claims

1. A system for adjusting an endoscope, the system comprising: a memory; and a controller communicatively coupled to the memory, wherein the controller: moves the endoscope through a cannula; stops the endoscope at a location in the cannula; captures an image of a reference corresponding to the cannula using the endoscope at the location; and adjusts a parameter of the endoscope based on the reference in the image.

2. The system of claim 1, wherein, Stopping the endoscope includes stopping a carriage on which the endoscope is positioned.

3. The system of claim 1, wherein, The controller also prevents the endoscope from moving further through the cannula until after the endoscope captures the image.

4. The system of claim 1, wherein, Adjusting the parameter includes: de-warping the image to produce a de-warp image; comparing the de-warp image to determine misalignment in the endoscope; determining an adjustment based on a size of the reference and the misalignment in the endoscope; and making the adjustment to the parameter.

5. The system of claim 1, wherein, The controller also: determines at least one of a contrast or an intensity of the reference in the image; and adjusts a brightness of the endoscope based on at least one of the contrast or the intensity.

6. The system of claim 1, wherein, The reference includes a mark on an inner wall of the cannula.

7. The system of claim 1, wherein, The reference includes an opening at an end of the cannula.

8. The system of claim 1, wherein, Stopping the endoscope at the location in the cannula is according to a mark on an inner wall of the cannula and at a predetermined distance from the reference.

9. The system of claim 1, wherein, The controller also: determines a boundary of an image captured by the endoscope after the endoscope is positioned at a surgical site outside the cannula; determines a difference between the boundary of the image and a reference boundary; and generates an alert indicating that the endoscope should be calibrated based on the difference exceeding a threshold.

10. The system of claim 1, wherein, The endoscope includes a first camera and a second camera, wherein the image includes a first image captured by the first camera and a second image captured by the second camera, and wherein adjusting the parameter includes: determining a pixel misalignment between the first image and the second image; converting the pixel misalignment to a translational misalignment and a rotational misalignment based on a size of the reference; and adjusting the parameter based on the translational misalignment and the rotational misalignment.

Images