Off-camera calibration parameters for image capture devices

Abstract

The surgical system includes a network-accessible database configured to store calibration data for the endoscopic image capture device. The surgical system also includes an image processor configured to be communicatively coupled to the endoscopic image capture device to receive image data of a surgical site during a surgical or diagnostic procedure. The image processor is configured to retrieve the calibration data from the network-accessible database via a network connection. The image processor is further configured to process the received images based on the retrieved calibration data. The surgical system also includes a user control system coupled to the image processor and configured to receive the processed images from the image processor. The user control system has a display configured to display the processed images. The calibration data is updated on the database periodically or dynamically to account for changes or new calibration methods for the surgical system.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Patent Application No. 62 / 722,314, filed Aug. 24, 2018, the disclosure of which is expressly incorporated herein by reference. [Background technology]

[0002] Image capture devices are typically calibrated at the factory before use in the field. During the calibration process, one or more images of a test pattern are captured and used as a reference to generate calibration data for the image capture device. For example, one or more transformations are used to adjust the captured image from the image capture device to match a known image of the test pattern. The calibration process can calibrate the captured image to correct for color, optical, alignment, or other image sensor induced or optically induced errors or variations in the image capture device. The calibration process generates a set of calibration data that is stored at the image capture device and applied to the corrected images captured by the image capture device during use. Summary of the Invention

[0003] A first aspect of the present disclosure includes a system having a network-accessible database configured to store calibration data for an image capture device. The system also has a controller system having an image processor configured to receive images captured by the image capture device. The image processor is further configured to retrieve the calibration data from the network-accessible database via a network connection. The image processor is further configured to process the received images based on the retrieved calibration data.

[0004] In some implementations of the first aspect of the present disclosure, the system further includes a tool having an image capture device and electrically coupled to the controller system, the tool having a unique identifier.

[0005] In some implementations of the first aspect of the present disclosure, the calibration data is indexed with the image capture device in a network-accessible database based on a unique identifier.

[0006] In some implementations of the first aspect of the present disclosure, the unique identifier is maintained by the tool, and the image processor is further configured to obtain the unique identifier from the tool.

[0007] In some implementations of the first aspect of the present disclosure, the unique identifier is maintained by the tool on a memory device of the tool or on a visual indicator on the surface of the tool.

[0008] In some implementations of the first aspect of the present disclosure, the image processor further includes an optical scanner configured to read a visual indicator from a surface of the tool to obtain a unique identifier from the tool.

[0009] In some implementations of the first aspect of the present disclosure, the optical scanner is a camera or a barcode reader.

[0010] In some implementations of the first aspect of the disclosure, the visual indicator is printed, glued, labeled, embossed, or stamped onto a surface of the tool.

[0011] In some implementations of the first aspect of the present disclosure, the visual indicator is an alphanumeric serial number, a linear barcode that encodes a unique identifier, a two-dimensional barcode that encodes a unique identifier, or an image encoded with a unique identifier.

[0012] In some implementations of the first aspect of the present disclosure, the tool includes an endoscope and the image capture device includes a pair of image sensors configured to capture stereoscopic image data of the surgical site during the surgical procedure.

[0013] In some implementations of the first aspect of the present disclosure, the calibration data includes data for processing the received image to correct the received image based on one or more of a color of light emitted by an illumination device configured to illuminate a scene captured by the received image, a calibration image captured by the image capture device, optical distortion introduced by an optical system of the image capture device, stereoscopic image calibration, or color space mapping.

[0014] In some implementations of the first aspect of the disclosure, the system further includes a calibration processor in communication with the network-accessible database, the calibration processor configured to receive calibration images captured by the image capture device and generate calibration data based on applying a calibration algorithm to one or more of the calibration images.

[0015] In some implementations of the first aspect of the present disclosure, the calibration images are stored in a network-accessible database along with the calibration data.

[0016] In some implementations of the first aspect of the disclosure, the calibration processor is configured to generate the calibration data upon receiving a request for the calibration data from the controller system.

[0017] In some implementations of the first aspect of the present disclosure, the calibration data is distinct from a pre-set of calibration data for the image capture device, and the pre-set of calibration data is generated by a calibration processor based on applying a different calibration algorithm to one or more of the calibration images.

[0018] In some implementations of the first aspect of the present disclosure, the calibration algorithm is selected based on the type of procedure in which the image capture device is to be used or based on the configuration of the controller system.

[0019] In some implementations of the first aspect of the present disclosure, calibration data is generated to enhance image features important to the type of procedure for which the image capture device is to be used.

[0020] In some implementations of the first aspect of the present disclosure, the image characteristic is one or more of focus level, sharpness, color fidelity, or color accuracy.

[0021] In some implementations of the first aspect of the present disclosure, the calibration data further adversely affects the second image characteristic.

[0022] In some implementations of the first aspect of the present disclosure, the configuration of the controller system includes one or more of the illumination source provided by the controller system, the type of image processor, or the software release version of the controller system.

[0023] In some implementations of the first aspect of the present disclosure, the calibration processor is further configured to retrieve operational images captured by the image capture device during the procedure from the controller system and adjust the calibration data based on the operational images.

[0024] In some implementations of the first aspect of the present disclosure, the brightness value of the surgical image is compared to an expected brightness value determined based on a known distance to an object captured in the surgical image and a known power output to an illumination source that illuminates the object captured in the surgical image.

[0025] In some implementations of the first aspect of the present disclosure, the color space of the surgical image is compared to an expected color space of the image capture device to determine whether the center of the color space is offset from the center of the expected color space.

[0026] In some implementations of the first aspect of the present disclosure, stereoscopic alignment of the surgical image is evaluated to identify misalignment with the corresponding surgical image.

[0027] In some implementations of the first aspect of the disclosure, the calibration processor is further configured to receive a user feedback survey that evaluates the calibration data in comparison to a previous set of calibration data.

[0028] In some implementations of the first aspect of the present disclosure, the controller system further includes a calibration cache memory configured to store the calibration data retrieved from the network-accessible database.

[0029] In some implementations of the first aspect of the present disclosure, the controller system is further configured to store calibration data for image capture devices known to have been dispatched to locations associated with the location of the controller system.

[0030] In some implementations of the first aspect of the present disclosure, the controller system is further configured to store calibration data for a predetermined number of previously used image capture devices.

[0031] In some implementations of the first aspect of the present disclosure, the controller system is further configured to store default calibration data for a certain type of image capture device.

[0032] In some implementations of the first aspect of the present disclosure, the controller system is further configured to compare the cached set of calibration data for the image capture device with the calibration data stored in the network-accessible database before retrieving the calibration data from the network-accessible database.

[0033] A second aspect of the disclosure includes a method that includes storing calibration data for an image capture device in a network-accessible database. The method also includes retrieving, by a controller system, the calibration data from the network-accessible database via a network connection. The method also includes receiving, by the controller system, an image captured by the image capture device. The method also includes processing, by an image processor of the controller system, the received image based on the retrieved calibration data.

[0034] A third aspect of the present disclosure includes a surgical system having a network-accessible database configured to store calibration data for an endoscopic image capture device. The surgical system also has an image processor configured to be communicatively coupled to the endoscopic image capture device to receive image data of a surgical site during a surgical procedure. The image processor is configured to retrieve the calibration data from the network-accessible database via a network connection. The image processor is further configured to process the received images based on the retrieved calibration data. The surgical system further has a user control system coupled to the image processor and configured to receive the processed images from the image processor. The user control system has a display configured to display the processed images.

[0035] In some implementations of the third aspect of the present disclosure, the surgical system further includes a manipulator system having a manipulator arm configured to attach to the endoscopic image capture device and configured to physically change the orientation of the endoscopic image capture device during the surgical procedure.

[0036] In some implementations of the third aspect of the present disclosure, the user control system further includes a manipulation user interface, where upon actuation of the manipulation user interface, the user control system is configured to issue a command to the manipulator system to change the orientation of the endoscopic image capture device.

[0037] A fourth aspect of the present disclosure includes a method of processing images of a surgical procedure captured by an endoscopic image capture device. The method includes receiving, by an image processor, images captured by the endoscopic image capture device. The method also includes retrieving, by the image processor, calibration data for the endoscopic image capture device from a network-accessible database. The method also includes processing, by the image processor, the received images from the endoscopic image capture device based on the retrieved calibration data. The method also includes transmitting, by the image processor, the processed images to a user control system. The method also includes displaying, on the user control system, the processed images on a display.

[0038] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims. [Brief description of the drawings]

[0039] For a more complete understanding of the present disclosure, reference is now made to the following brief description, taken in conjunction with the accompanying drawings and detailed description, in which like reference numerals represent like parts.

[0040] [Figure 1] FIG. 1 is a plan view of a minimally invasive teleoperated surgical system. [Diagram 2] FIG. 2 is a perspective view of a user control system. [Diagram 3] FIG. 2 is a perspective view of an electronics cart. [Figure 4] FIG. 1 is a diagram of a teleoperated surgical system. [Diagram 5] FIG. 1 is a perspective view of an endoscopic image capture device. [Figure 6] FIG. 1 is a schematic diagram of an image capture device calibration tool. [Figure 7] 4 is a flowchart illustrating an example process for obtaining calibration data for an image capture device. [Figure 8] 5 is a flowchart illustrating an exemplary process for transmitting calibration data in response to a request. [Figure 9] 5 is a flow chart illustrating an example process for determining altered calibration parameters or maintenance conditions based on a field image captured by an image capture device. [Figure 10A] FIG. 4 is a chromaticity diagram generated from the processed field images showing a shift in the center of the color space of the image capture device. [Figure 10B] FIG. 4 is a chromaticity diagram generated from the processed field images showing a shift in the center of the color space of the image capture device. [Figure 10C] FIG. 4 is a chromaticity diagram generated from the processed field images showing a shift in the center of the color space of the image capture device. [Figure 10D] FIG. 4 is a chromaticity diagram generated from the processed field images showing a shift in the center of the color space of the image capture device. [Figure 11] 1 is a flowchart illustrating an exemplary process for performing a calibration parameter lookup. [Figure 12] 1 illustrates an exemplary computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0041] First, an exemplary implementation of one or more embodiments is shown below, but it should be understood that the disclosed system and method can be implemented using any number of technologies, whether currently known or in existence. The present disclosure should not be limited to the exemplary implementations, drawings, and technologies shown below, but can be modified within the scope of the appended claims, together with their full scope of equivalents. The use of the term "and / or" indicates that any one or any combination of a list of options can be used. For example, "A, B, and / or C" means "A", or "B", or "C", or "A and B", or "A and C", or "B and C", or "A and B and C".

[0042] Elements described in detail with reference to one embodiment, implementation, or application may, whenever practical, be included in other embodiments, implementations, or applications in which they are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment and not with reference to a second embodiment, the element may still be claimed as being included in the second embodiment. Thus, to avoid unnecessary repetition in the following description, one or more elements shown and described in connection with one embodiment, implementation, or application may be incorporated in other embodiments, implementations, or aspects, unless specifically described otherwise, unless one or more elements render the embodiment or implementation non-functional or unless two or more elements provide conflicting functionality.

[0043] Aspects of the invention are described primarily with respect to implementations using the da Vinci® Surgical System (specifically, model IS4000, marketed as the da Vinci® Xi™ HD™ Surgical System), as commercialized by Intuitive Surgical, Inc., Sunnyvale, Calif. However, those skilled in the art will appreciate that aspects of the invention disclosed herein may be practiced and implemented in a variety of ways, including robotic and, where applicable, non-robotic embodiments and implementations. Implementations with the da Vinci® Surgical System (e.g., model IS4000 da Vinci® Xi™ Surgical System, model IS3000 da Vinci Si® Surgical System) are merely exemplary and should not be considered as limiting the scope of aspects of the invention disclosed herein.

[0044] According to various aspects, the present disclosure describes systems and methods for off-camera calibration of an image capture device. Calibration data is typically maintained in a read-only memory of the image capture device and cannot be changed or updated after the image capture device leaves the factory. Calibration data for an image capture device typically does not take into account wear on the device over time. Thus, calibration data stored on the image capture device may become less accurate over time, resulting in processed images from the image capture device having more error.

[0045] For example, an endoscopic image capture device includes a camera housing having a flexible cable with a fiber optic bundle. The cable includes a connector configured to couple the fiber optic bundle to a light source. The camera housing also includes a read-only memory that stores calibration data for the endoscopic image capture device. The camera housing also includes a rigid camera shaft having a camera tip at a distal end. The camera tip includes one or more image sensors and an associated optical system. The camera shaft also includes a second fiber optic bundle configured to convey light received from the fiber optic bundle in the flexible cable to the camera tip to illuminate a scene being imaged by the one or more image sensors, such as a diagnostic or surgical procedure. Images captured by the one or more image sensors in the camera tip are conveyed to the camera housing via a wired or wireless electrical connection and then conveyed to a connector via a wired or wireless electrical connection in the flexible cable.

[0046] The controller system has a socket configured to receive the connector. The controller system has a light source coupled to the socket and configured to provide light to the fiber optic bundle in the flexible cable. The controller system also has an image processor coupled to the socket and configured to receive images transmitted through the electrical connection in the flexible cable. The image processor is configured to read calibration data from a read-only memory in the camera housing when the connector is inserted into the socket. The image processor processes the received images based on the read calibration data to generate one or more processed images. When using an endoscopic image capture device, the optical transmission characteristics of the fiber optic bundle in the flexible cable may change over time. Also, the camera chip may wear over time due to use in a medical procedure. Thus, at least the optical characteristics of the endoscopic image capture device may change over time, resulting in calibration data that is not optimized for a current state of the endoscopic image capture device.

[0047] Additionally, an endoscopic image capture device may be coupled to different controller systems at different times, one or more of which may have different configurations. Different controller systems may have different system configurations, such as different image processors, vision systems, or other hardware variations, different software release versions, and / or different light sources having different spectral responses. Thus, the calibration data may not be optimized for the particular controller system to which the endoscopic image capture device is currently coupled.

[0048] Furthermore, advances in the transformations used to generate calibration data from captured test patterns cannot be retroactively applied to existing surgical endoscopic image capture devices, and thus the calibration data stored in the surgical endoscopic image capture devices may not be optimized according to current knowledge and technology.

[0049] In various implementations of the pending disclosure, the calibration data is moved off-camera and retrieved by a controller system from a central calibration server. When an image capture device, such as an endoscopic image capture device, is connected to the controller system, the controller system retrieves a unique identifier for the image capture device. Although various examples provided herein are described with respect to an endoscopic image capture device, the pending disclosure is not so limited and is intended to encompass any device coupled to a controller system having corresponding calibration data to control the operation of the device or the processing of data received from the device. Similarly, the pending disclosure is intended to encompass any image capture device coupled to a controller system to process images captured by the image capture device. For example, the pending disclosure may be equally applicable to a borescope or other such inspection camera.

[0050] The unique identifier may be maintained in a memory of the image capture device. Alternatively, a reader coupled to the controller system reads the unique identifier from the image capture device. For example, the reader may be a radio frequency identifier (RFID), Bluetooth® low energy (BLE), Wi-Fi®, or other wireless tag reader, and the unique identifier may be stored in a wireless tag such as an RFID, BLE, Wi-Fi®, or other wireless tag.

[0051] In another example, the unique identifier is provided as an optically readable label on the housing of the image capture device. The label may include an alphanumeric string, symbol, or code in encrypted or unencrypted form. The label may be embossed, glued, attached, printed, or otherwise attached to the housing of the image capture device. The unique identifier may be read from a camera, barcode reader, or other optical scanner coupled to or otherwise in communication with the controller system.

[0052] Using the retrieved unique identifier of the image capture device, the controller system retrieves calibration data of the image capture device from a central calibration server. The controller system then processes images received from the image capture device based on the retrieved calibration data. In some implementations, the unique identifier is a universal unique identifier (UUID) for the medical device.

[0053] By offloading the calibration data from the image capture device, the memory requirements of the image capture device can be reduced, resulting in a less expensive image capture device. For example, newly manufactured image capture devices may omit the read-only calibration memory, or may reduce the size of the read-only memory to store only a unique identifier for the image capture device.

[0054] The calibration data may be generated periodically or dynamically at a central calibration server. Thus, the method of calculating one or more of the calibration parameters may be changed based on the type of procedure to be performed, based on the current device configuration, and / or based on improvements in the calibration methodology. Additionally, obtaining the calibration data from a central calibration server extends the life and / or functionality of existing devices in the field that may otherwise store outdated calibration data or calibration data that is not optimized for a particular procedure.

[0055] Because the controller system is in regular communication with the central calibration server, additional functionality may be enabled. For example, images captured from the image capture devices may be periodically communicated to the central calibration server. These field images may be processed to identify improvements to the calibration data of a particular image capture device over time or to flag the image capture device for maintenance. For example, the field capture images may reveal spectral attenuation of the captured image over time due to wear of the fiber optic bundles in the flexible cable. The spectral attenuation may be corrected by adjustments to the calibration data or to the light source provided to the image capture device. Similarly, color space shifts of one or more image sensors over time may be corrected by adjustments to the calibration data or to the light source provided to the image capture device. Additionally, alignment issues in the stereoscopic images may be identified and flagged for maintenance.

[0056] As another example, preference studies may be conducted with end users of image capture devices having different sets of calibration data. For example, different color mapping or distortion correction may be used with different sets of calibration data, and feedback from end users may be obtained to inform an optimized set of calibration data. Additional controls may be added to prevent use of the controller system with non-registered image capture devices, facilitate recall control of image capture devices, and limit the total number of uses of image capture devices before requiring refurbishing.

[0057] Referring now to the drawings, where like reference numbers represent like parts throughout the several views, FIG. 1 is a plan view of a minimally invasive teleoperated surgical system 10 typically used to perform a minimally invasive diagnostic or surgical procedure on a patient 12 lying on a mobile operating table 14. The system includes a user control system 16, such as a mobile surgeon's console, for use by a surgeon 18 during the procedure. One or more assistants 20 may also participate in the procedure. The minimally invasive teleoperated surgical system 10 further includes a manipulation system 22, such as a mobile patient side cart, and a mobile electronics cart 24. In some embodiments, the user control system 16, manipulation system 22, and electronics cart 24 are mounted on wheels to provide mobility.

[0058] The manipulation system 22, or other such manipulation systems, includes a plurality of segmented mechanical support arms 72, each having one end rotatably mounted to a vertical support structure 74 and another end for mounting a removably coupled surgical instrument 26. In some embodiments, each mechanical support arm 72 includes a first segment 72-1, a second segment 72-2, and a third segment 72-3. During set-up for a procedure, the multiple segments of at least one support arm 72 are moved to position the surgical instrument for insertion into a minimally invasive incision in the body of the patient 12.

[0059] During a procedure, the surgeon 18 views the surgical site through the user control system 16 while instruments are inserted into the patient's body cavity. Images of the surgical site may be acquired by an endoscope 28, such as a stereoscopic endoscope, which may be manipulated by a manipulation system 22 to orient the endoscope 28. Computer processor(s) located on the electronics cart 24 may be used to process the images of the surgical site for subsequent presentation to the surgeon 18 via the user control system 16. The computer processor(s) may alternatively be referred to herein as an image processor or a video processor.

[0060] One or more illumination sources or illuminators may also be provided on the electronics cart 24 to provide light for use by the endoscope 28 to illuminate a surgical site. The illuminators may include white light sources, colored light sources (e.g., red, green, blue, cyan, magenta, yellow, etc.), infrared light sources, laser light sources, or any other type of light source or combinations thereof. Different illuminators may be used at different times in a surgical or diagnostic procedure. For example, the electronics cart 24 may be configured, such as through selection on the user control system 16, to provide light from one or more of a first set of illuminators at a first time and provide light from one or more of a second set of illuminators at a second time. Can be controlled.

[0061] The number of surgical instruments 26 used at one time generally depends on, among other things, the diagnostic or surgical procedure and the space constraints within the operating room. If it becomes necessary to change one or more of the surgical instruments 26 being used during surgery, the assistant 20 can remove the surgical instrument 26 from the manipulation system 22 and replace it with another surgical instrument 26 from a tray 30 in the operating room.

[0062] 2 is a perspective view of user control system 16. User control system 16 includes a display area 31 having a left-eye display 32 and a right-eye display 34 for presenting surgeon 18 with a coordinated stereoscopic view of the surgical site that enables depth perception.

[0063] The console 16 further includes one or more control inputs 36. One or more surgical instruments installed for use with the manipulation system 22 (shown in FIG. 1 ) move in response to the surgeon 18's manipulation of the one or more control inputs 36. The control inputs 36 may be provided with the same mechanical degrees of freedom as the associated surgical instruments 26 (shown in FIG. 1 ) to provide the surgeon 18 with a telepresence, or a sense that the control inputs 36 are one with the instruments 26, so that the surgeon has a strong sense of direct control over the instruments 26. To this end, position, force, and tactile feedback sensors (not shown) may be used to communicate position, force, and tactile sensations from the surgical instruments 26 through the control inputs 36 and back to the surgeon's hands. The height of the control inputs 36 may be adjusted with a height adjustment lever 38.

[0064] The user control system 16 is typically located in the same room as the patient so that the surgeon can directly monitor the procedure and, if necessary, be physically present and speak directly to the patient assistant rather than by telephone or other communication medium, however, the surgeon may be located in a different room, an entirely different building, or other location remote from the patient to enable remote surgical procedures.

[0065] 3 is a perspective view of electronics cart 24. Electronics cart 24 is coupled to endoscope 28 and may include a computer processor for processing captured images for later display, such as to the surgeon on user control system 16 or on another suitable display located locally and / or remotely. For example, if a stereoscopic endoscope is used, the computer processor on electronics cart 24 may process the captured images to present the surgeon with coordinated stereoscopic images of the surgical site. Such coordination may include alignment between opposing images and may include adjusting the stereoscopic working distance of the stereoscopic endoscope.

[0066] As another example, image processing may include the use of previously or dynamically determined camera calibration parameters to compensate for imaging errors of the image capture device, such as optical aberrations. Calibration data may be obtained by the electronics cart 24 from a central calibration server, as described in more detail below.

[0067] Because the calibration data is obtained from a central calibration server, rather than from the image capture device itself, new calibration data may be used with existing image capture devices that may store the current calibration data on the image capture device's local memory. Thus, the useful life or functionality of existing image capture devices may be extended by remotely obtaining calibration data for the existing image capture devices. Similarly, memory requirements of new image capture devices may be reduced, resulting in less expensive image capture devices. For example, a newly manufactured image capture device may omit memory for storing calibration data, or may reduce the size of the memory to store only a unique identifier for the image capture device. Additionally, the method of calculating one or more of the calibration parameters may be changed based on the type of procedure to be performed, based on the current device configuration, and / or based on improvements to the calibration method.

[0068] The electronics cart 24 may also maintain the acquired calibration data in a local cache. In addition to maintaining the acquired calibration data in a local cache, the electronics cart 24 may also maintain calibration data for one or more image capture devices expected to be used with the electronics cart 24. For example, when a new image capture device is ordered, delivered, or received at a location with the electronics cart 24, the electronics cart 24 may acquire or receive calibration data for the new image capture device. In some implementations, a central calibration server may push the calibration data for the new image capture device to be maintained on the local cache of the electronics cart 24. The local cache of the electronics cart 24 may also maintain a default set of calibration data for use when there is limited or no connectivity with the central calibration server.

[0069] Optionally, the equipment in the electronics cart 24 may be integrated into the user control system 16 or operating system 22, or may be distributed to various other locations within the operating room. More generally, the electronics cart 24 or the user control system 16 with the equipment integrated from the electronics cart 24 may be referred to herein as a controller system for obtaining calibration data and processing images from the image capture devices based on the retrieved calibration data.

[0070] FIG. 4 diagrammatically illustrates a teleoperated surgical system 50 (such as the minimally invasive teleoperated surgical system 10 of FIG. 1). A user control system 52 (such as the user control system 16 of FIG. 1) can be used by a surgeon to control a manipulation system 54 (such as the manipulation system 22 of FIG. 1) during a minimally invasive procedure. The manipulation system 54 can use an image capture device, such as a stereoscopic endoscope, to capture images of the surgical site and output the captured images to a computer processor located in an electronics cart 56 (such as the electronics cart 24 of FIG. 1). The computer processor typically includes one or more data processing boards intended to execute computer readable code stored in a non-volatile memory device of the computer processor. Similar to the electronics cart 24, the electronics cart 56 also includes one or more illumination sources for providing light to the image capture device. The electronics cart 56 can also include a calibration cache for maintaining a local cache of calibration data similar to the electronics cart 24.

[0071] In one aspect, the computer processor can process the captured images in various ways prior to subsequent display. For example, the computer processor can use previously determined or dynamically determined camera calibration parameters to compensate for imaging errors of the image capture device before displaying the processed images to the surgeon via the user control system 52. Additionally or alternatively, the captured images can undergo image processing by a computer processor located outside the electronics cart 56. In one aspect, the teleoperated surgical system 50 includes an optional computer processor 58 (shown in dashed lines) similar to the computer processor located in the electronics cart 56, and the manipulation system 54 outputs the captured images to the computer processor 58 for image processing before displaying on the user control system 52. In another aspect, the captured images first undergo image processing by the computer processor of the electronics cart 56 and then undergo additional image processing by the computer processor 58 before displaying on the user control system 52. In some implementations, the electronics cart 56 and / or the computer processor 58 are collectively referred to as a controller system for obtaining calibration data and processing images from the image capture device based on the retrieved calibration data. In some implementations, the controller system also provides light from one or more illumination sources to the image capture device.

[0072] Teleoperated surgical system 50 may include an optional display 60, as indicated by the dashed lines. Display 60 is coupled to computer processors located on electronics cart 56 and computer processor 58, such that captured images processed by these computer processors may be displayed on display 60 in addition to being displayed on the display of user control system 52. In various implementations, display 60 may be located on electronics cart 56, such as display 25 on electronics cart 24. In some implementations, display 60 may be separate from user control system 52 and electronics cart 58.

[0073] 5 is a perspective view of an endoscopic image capture device 500. The endoscopic image capture device 500 includes a connector 502, a flexible cable 506, a camera housing 508, a rigid camera shaft 512, and a camera tip 514. The connector 502 has an optical port 504 and an electrical port (not shown). The connector 502 is sized and shaped to be inserted into a socket of the electronics cart 56, such as socket 27 of the electronics cart 24 or other mating receptacle. The optical port 504 is configured to receive light provided by the electronics cart 24 to the socket 27.

[0074] A flexible cable 506 is coupled between the connector 502 and the camera housing 508. The flexible cable 506 includes a fiber optic bundle configured to transmit light received from the optical port 504 of the connector 502 to the camera housing 508. The flexible cable 506 also includes an electrical connection configured to provide electrical communication between the electrical port of the connector 502 and the camera housing 508. The electrical connection can be a wired or wireless connection. In some implementations, the wired connection is a wire, ladder wire, twisted wire pair, Universal Serial Bus (USB) cable, Ethernet cable, or other wired communication line.

[0075] The camera housing 508 receives the distal end of the fiber optic bundle from the flexible cable 506. The camera housing 508 also receives the proximal end of the rigid camera shaft 512. The distal end of the rigid camera shaft 512 includes a camera chip 514 having one or more image sensors and associated optical systems. For example, the camera chip 514 may include two image sensors with respective optical components for capturing stereoscopic images of a scene, such as a surgical or diagnostic procedure. The rigid camera shaft 512 may also include a second fiber optic bundle configured to convey light received at the camera housing 508 from the fiber optic bundle in the flexible cable to the camera chip 514 to illuminate the scene being imaged by the one or more image sensors.

[0076] The rigid camera shaft 512 may also include a second electrical connection configured to provide electrical communication between the one or more image sensors of the camera chip 514 and the camera housing 508. Images captured by the one or more image sensors in the camera chip 514 are communicated to the camera housing 508 via an electrical connection in the rigid camera shaft 512. The electrical connection may be a wired or wireless connection. In some implementations, the wired connection is a wire, a ladder wire, a twisted wire pair, a Universal Serial Bus (USB) cable, an Ethernet cable, or other wired communication line.

[0077] The camera housing 508 may also have one or more camera control units (not shown) configured to provide power and control signals for capturing images from one or more image sensors of the camera chip 514. For example, if the camera chip 514 has two image sensors for capturing stereoscopic images, the camera housing 508 may have a respective camera control unit for controlling each of the two image sensors. The one or more camera control units are also configured to communicate captured images to an electrical port of the connector 502 for processing by the electronics cart 56 and / or computer processor 58.

[0078] The camera housing 508 may also include a display 510 for displaying one or more operational controls of the endoscopic image capture device 500.

[0079] The camera housing 508 may also have a read-only memory (not shown) that stores a unique identifier for the endoscopic image capture device 500. In some implementations, the unique identifier is a universal unique identifier (UUID) for the medical device. The read-only memory does not store calibration data for the endoscopic image capture device 500. The read-only memory is smaller and less expensive than read-only memory used in conventional endoscopic image capture devices that additionally store calibration data.

[0080] In some implementations, the read-only memory in the camera housing 508 may be omitted entirely. Instead, a wireless identification tag located on or in the camera housing 508, or an optically readable label on the exterior surface of the camera housing 508, may provide a unique identifier for the endoscopic image capture device 500. For example, the wireless identification tag may be an RFID, BLE, Wi-Fi, or other wireless tag. The label may include an alphanumeric string, symbol, or code in encrypted or unencrypted form. The label may be embossed, glued, adhered, printed, or otherwise attached to the camera housing 508.

[0081] If an optically readable label is used to provide the unique identifier on the camera housing 508, an optical scanner coupled to or in communication with the controller system is used to read the label to obtain the unique identifier. The optical scanner may be a camera, a barcode reader, or other optical scanner. For example, any of the operating system 54, the electronics cart 56, and / or the computer processor 58 may have an optical scanner configured to read the label on the camera housing 508 to obtain the unique identifier of the endoscopic image capture device 500.

[0082] In one embodiment, manipulation system 54 has a camera (not shown) configured to capture images of the camera housing 508 of endoscopic image capture device 500 when installed thereon. Images captured by the camera on manipulation system 54 are processed by electronics cart 56 and / or computer processor 58 to identify a unique identifier in the captured image.

[0083] In another implementation, the socket 27 or other mating receptacle of the electronics cart 24 includes an optical scanner. A surface of the connector 502 or optical port 504 includes an optically readable label and is configured to be in the field of view or face the optical scanner when the connector 502 is inserted into the socket 27. The optical scanner of the electronics cart 24 is configured to read the label upon insertion of the connector 502 into the socket 27 to obtain the unique identifier of the endoscopic image capture device 500. Thus, the user does not need to perform any additional action to scan the unique identifier beyond inserting the connector 502 into the socket 27. Some users may attempt to "spoof" the electronics cart 24, where a photograph or other copy of the optically readable label is scanned for unauthorized tools or endoscopic image capture devices 500. By providing an optical scanner in socket 27, it becomes more difficult to spoof electronics cart 24 since it becomes more difficult to have both an optically readable copy of the label and connector 502 in socket 27 at the same time.

[0084] In further implementations, the electronics cart 24 has an optical scanner located at a location other than within the socket 27 or other mating receptacle. For example, the optical scanner may be located on a surface of the electronics cart 24 adjacent to the socket 27 or within a handheld scanner device coupled to the electronics cart 24. The optical scanner may also be located on other devices other than the electronics cart 24, such as the operating system 54 and / or the computer processor 58, as described above. By locating the optical scanner at a location outside the socket 27, a corresponding set of calibration data for the endoscopic image capture device 500 or other tool may be retrieved prior to use or insertion into the socket 27. In this manner, the corresponding set of calibration data is retrieved without counting as use of the endoscopic image capture device 500 or other tool when the optical scanner scans an optically readable label. Also, additional endoscopic image capture devices 500 or other tools may be scanned to retrieve a corresponding set of calibration data while the endoscopic image capture device 500 or other tool is being used or is already inserted into the socket 27.

[0085] 6 is a schematic diagram of an image capture device calibration tool 600. The image capture device calibration tool 600 includes a first teleoperated surgical system 604 and a second teleoperated surgical system 606 in data communication with a central calibration server 602. The teleoperated surgical systems 604, 606 are similar to the teleoperated surgical system 50 shown in FIG. 4 and described above. In some implementations, the teleoperated surgical systems 604, 606 have the same system configuration. In some implementations, the teleoperated surgical systems 604, 606 have different system configurations. For example, the teleoperated surgical systems 604, 606 may have differences in one or more of an image processor, a vision system, or other hardware system, a software release version, and / or a light source having a different spectral response. Other differences that affect the calibration of the image capture devices are contemplated by the present disclosure.

[0086] 6, only two teleoperated surgical systems 604, 606 are shown, however, there may be more or fewer systems that may be in data communication with the central calibration server 602. In some implementations, the teleoperated surgical systems 604, 606 may be located at the same or different healthcare provider locations. For example, a first teleoperated surgical system 604 may be located at a first healthcare provider location and a second teleoperated surgical system 606 may be located at a second healthcare provider location.

[0087] The central calibration server 602 has or is in communication with one or more databases that store calibration records. Each calibration record is indexed to a unique identifier of the corresponding image capture device. The calibration records include one or more sets of calibration data and a set of test pattern images. Different sets of calibration data may be optimized for different procedures or controller system configurations. Each set of calibration data includes calibration parameters for adjusting the captured images based on one or more of optical distortion, color mapping correction, color of light illuminating the scene, and / or stereo image alignment. Other calibration parameters may be used. For example, the calibration parameters may include image sensor characteristics such as dead or stuck pixels, cropping area, analog gain, digital gain, or other such image sensor characteristics.

[0088] In some implementations, a different set of calibration parameters may increase performance in a region of interest during a particular procedure while potentially decreasing performance of functionality in other regions. For example, if a particular procedure has a need for very fine close-up images, the calibration parameters of a set of calibration data for the procedure may be optimized for sharpness of images processed using the calibration data. Such optimization for image sharpness may come at the expense of color accuracy or fidelity. Similarly, the functionality of a lower performance image capture device may be extended through a set of calibration data that improves the performance of the lower performance image capture device in a particular region. The use of a set of calibration data with a particular performance improvement may be based on the requirements of a particular procedure, end user (e.g., surgeon) preferences, or other criteria.

[0089] The set of test pattern images includes images captured by the corresponding image capture device in a calibration environment, such as a factory calibration of the corresponding image capture device. Each image in the set of test pattern images may capture a different test pattern and / or a different lighting condition. In some implementations, a subset of the test pattern images may be captured to calibrate the image capture device for a particular procedure, a particular lighting condition, or a particular controller system configuration.

[0090] The central calibration server 602 also includes or is in communication with a calibration processor configured to generate a set of calibration data from one or more images in the set of test pattern images. The calibration processor may dynamically generate the set of calibration data in response to a request for calibration data from a controller system, such as a request from the electronics cart 56 or processor 58 of one of the teleoperated surgical systems 604, 606.

[0091] Alternatively or additionally, the calibration processor may generate the set of calibration data in response to a change in the calibration conditions, for example, the change in the calibration conditions may be based on a change in the system configuration of one of the teleoperated surgical systems 604, 606 or a change in the transformation used to generate the calibration data.

[0092] A change in system configuration may include a change in one or more of the image processor, vision system, or other hardware components, software release version, and / or light source in the controller system. A change in system configuration may be caused by an upgrade or maintenance of one of the teleoperated surgical systems 604, 606. A change in system configuration may also occur by connecting an image capture device to a different controller system.

[0093] For example, an image capture device may be initially connected to a first electronics cart 56 having a first system configuration, and a first set of calibration data is provided for the first system configuration. Later, the image capture device may be connected to a second electronics cart 56 having a second system configuration that is different from the first system configuration. The second system configuration may have a different image processor, vision system, or other hardware components, a different software release version, and / or a different light source than the first system configuration. Thus, the calibration processor may generate a second set of calibration data for the image capture device that is optimized for the second system configuration.

[0094] Changes in the system configuration of the controller system may be identified by the calibration processor as part of processing a request for calibration data from the controller system. For example, the request from the controller system may include information regarding the current system configuration of the controller system. Alternatively, upon implementation of a system configuration change of the controller system, a notification may be sent from the controller system to the central calibration server 602.

[0095] Other calibration events may also trigger the calibration processor to generate a set of calibration data. For example, a calibration event is received by the central calibration server 602 when a new image capture device is ordered, delivered, or received at a healthcare provider location of one of the teleoperated surgical systems 604, 606. For example, when a new image capture device is ordered, delivered, or received at a first healthcare provider location of the teleoperated surgical system 604, the central calibration server receives a calibration notification for the new image capture device. The calibration processor generates a set of calibration data for the new image capture device for use with the electronics cart 56 and / or processor 58. The central calibration server 602 may send or push the newly generated set of calibration data to the electronics cart 56 and / or processor 58 of the first teleoperated surgical system 604.

[0096] 7 is a flow chart illustrating an example process 700 for obtaining calibration data for an image capture device. Process 700 is performed by a controller system, such as electronics cart 56 and / or processor 58. Other components or systems of a teleoperated surgical system may also perform process 700. For simplicity, process 700 is described below as being performed by electronics cart 56, but is not intended to be limiting to the spirit or scope of the present disclosure.

[0097] At 702, the electronics cart 56 obtains the unique identifier of the image capture device. For example, when the connector 502 of the endoscopic image capture device 500 is inserted into the socket 27 of the electronics cart 56, the electronics cart 56 obtains the unique identifier of the endoscopic image capture device 500. As described above, the unique identifier may be obtained by reading a memory, wireless tag, or optical label by the electronics cart 56 or an external reader in communication with the electronics cart 56.

[0098] At 704, the electronics cart 56 checks its local calibration cache against the obtained unique identifier for calibration data for the image capture device. At 706, if a local copy of the calibration data for the image capture device is maintained in cache, the electronics cart 56 sends a request to the central calibration server 602 to verify that the cached version of the calibration data is up to date.

[0099] For example, the electronics cart 56 may request a hash of the current version of the calibration data from the central calibration server 602. The electronics cart 56 may also determine or read a hash of a local copy of the calibration data to compare with the hash received from the central calibration server 602. Alternatively, the electronics cart 56 may send a hash of a local copy of the calibration data to the central calibration server 602 and receive the results of the comparison performed by the central calibration server 602.

[0100] At 708, the electronics cart 56 determines whether the local copy of the calibration data is up to date, for example, based on the results of a comparison of a hash of the local copy against a hash received from the central calibration server 602, or based on receipt of a comparison performed by the central calibration server 602.

[0101] If, at 710, the electronics cart 56 determines that the local copy of the calibration data is up to date, the local copy of the calibration data is used to process the image received from the image capture device.

[0102] Otherwise, at 712, the electronics cart 56 sends a request for calibration data to the central calibration server 602 based on the obtained unique identifier of the image capture device. Similarly, if it is determined at 704 that a local copy of the calibration data does not exist in cache, the electronics cart 56 sends a request for calibration data to the central calibration server 602 based on the obtained unique identifier of the image capture device.

[0103] The electronics cart 56 receives the requested calibration data at 714. The electronics cart 56 uses the received calibration data to process images received from the image capture device at 716. At 718, the received calibration data is stored in a local calibration cache and indexed to the unique identifier of the image capture device.

[0104] One or more features of process 700 may be optional or may be performed in a different order. For example, if the electronics cart 56 does not have a local calibration cache, the features performed at 704-710 may be omitted. As another example, the feature performed at 718 may be performed prior to the feature performed at 716. Other additions or variations of process 700 are contemplated by this disclosure.

[0105] 8 is a flow chart illustrating an example process 800 for transmitting calibration data in response to a request. Process 800 is performed by central calibration server 602 and / or one or more processors in communication with central calibration server 602. For simplicity, process 800 is described below as being performed by central calibration server 602, but is not intended to be limiting to the spirit or scope of the disclosure.

[0106] At 802, the central calibration server 602 receives a request for calibration data from a controller system, such as the electronics cart 56 and / or the processor 58. The request includes a unique identifier for the image capture device. The request may additionally include information about the controller system, the configuration of the controller system, or an action to be performed on the controller system.

[0107] At 804, using the received unique identifier, the central calibration server 602 retrieves a set of calibration data for the image capture device from a database on or in communication with the central calibration server 602. For example, the central calibration server 602 performs a lookup operation for one or more tables of calibration data indexed to the unique identifier. Additional data received from the request or obtained by the central calibration server 602 may be used to retrieve a particular set of calibration data from among multiple sets of calibration data. For example, upon receiving an indication that the image capture device will be used for a particular procedure, the central calibration server 602 may retrieve a set of calibration data optimized for performance of the particular procedure.

[0108] At 806, the central calibration server 602 determines whether an error condition exists associated with the image capture device. For example, the central calibration server 602 may implement a policy that prevents use of the controller system with unregistered image capture devices, may limit use of image capture devices before maintenance or refurbishment is required, or may lock out image capture devices that are subject to a recall.

[0109] At 808, if an error condition exists, the central calibration server 602 responds to the received request for calibration data with an error message. For example, the error message may indicate that the image capture device is not compatible with the controller system, that the image capture device requires maintenance or refurbishment, or that the image capture device is subject to a recall. Other error conditions and error messages are contemplated by this disclosure.

[0110] At 810, if the central calibration server 602 determines that no error condition exists for the image capture device, a response is sent to the controller system along with the retrieved set of calibration data.

[0111] One or more features of process 800 may be optional or may be performed in a different order. Other additions or variations of process 800 are contemplated by this disclosure.

[0112] 9 is a flow chart illustrating an example process 900 for determining altered calibration parameters or maintenance conditions based on field images captured by the image capture devices. In use, a controller system, such as the electronics cart 56 and / or the processor 58, may periodically provide field images captured by the image capture devices during a procedure to the central calibration server 602. For example, the controller system may provide at least one image captured by the image capture devices for each lighting condition used during a procedure. Other images and frequencies for providing images to the central calibration server 602 may also be used.

[0113] At 902, the central calibration server 602 receives a field image from a controller system. The field image may be received along with a unique identifier of the image capture device that captured the field image. Additional information regarding the configuration of the controller system, current lighting conditions, or other contextual information regarding the conditions under which the field image is captured may also be provided.

[0114] At 904, the central calibration server 602 processes the received field images to evaluate one or more calibration parameters for the image capture device. For example, multiple received field images may be processed to identify changes over time. For example, processing of the field images may reveal spectral attenuation in the captured images over time due to wear of the fiber optic bundles in the flexible cable 506. Similarly, processing of the field images may reveal color space shifts over time of one or more image sensors in the image capture device. Additionally, processing of the field images may reveal alignment issues in the stereoscopic images.

[0115] At 906, the central calibration server 602 determines whether one or more calibration parameters of the calibration data should be changed based on the processing of the field image. If so, at 908, the central calibration server 602 determines and transmits updated calibration parameters to the controller system. For example, the updated calibration parameters may adjust the brightness, color space, alignment, or other characteristics of the image processing with the updated calibration data. The central calibration server 602 may also transmit one or more updated operating parameters for the controller system or the image capture device. For example, in response to detecting a spectral attenuation in the field image, the central calibration server may transmit an operating parameter to increase the power of a light source provided by the controller system or to increase the gain of an image sensor of the image capture device. Other variations of the operating parameters may be used.

[0116] At 910, the central calibration server 602 stores the calibration parameters of the updated calibration data in a calibration record associated with the image capture device, e.g., the calibration record associated with the unique identifier on one or more databases maintained by or in communication with the central calibration server 602 is updated.

[0117] If the central calibration server 602 determines at 906 that the calibration parameters are not to be changed, the central calibration server 602 determines whether any maintenance conditions exist at 912. For example, upon detecting a stereoscopic alignment problem in the processed field images, the central calibration server may determine that the image capture device requires maintenance. Other maintenance conditions are contemplated by this disclosure.

[0118] At 914, the central calibration server 602 sends a maintenance message to the controller system. For example, the maintenance message may provide instructions to adjust one or more operating parameters, notify the image capture device of maintenance or modification, or provide notification of required maintenance. Otherwise, if the central calibration server 602 determines that a maintenance condition does not exist, the process 900 ends at 916.

[0119] One or more features of process 900 may be optional or may be performed in a different order. Other additions or variations of process 900 are contemplated by this disclosure.

[0120] 10A-10D are chromaticity diagrams generated from processed field images that show a shift in the center of the color space of the image capture device. Each of the chromaticity diagrams may be generated from a set of field images received and processed by the central calibration server 602.

[0121] For example, as shown in FIG. 10A, a first chromaticity diagram 1002 is generated by the central calibration server 602 when processing a first set of received field images of the image capture devices. The chromaticity diagram 1002 shows a first color space 1004 and a center 1006 of the first color space determined from the first set of received field images. FIG. 10B shows a second chromaticity diagram 1008 generated by the central calibration server 602 when processing a second set of received field images of the image capture devices. The chromaticity diagram 1008 shows a second color space 1010 and a center 1012 of the second color space determined from the second set of received field images. FIG. 10C shows a third chromaticity diagram 1014 generated by the central calibration server 602 when processing a third set of received field images of the image capture devices. Chromaticity diagram 1014 shows a third color space 1016 and a center of the third color space 1018 determined from the received third set of field images. The first, second, and third sets of received field images may be received at different times.

[0122] Figure 10D shows a chromaticity diagram 1020 in which the first, second, and third color spaces 1004, 1010, and 1016 are superimposed on one another to show tracks of the first, second, and third color space centers 1006, 1012, 1018. As shown in Figure 10D, the first, second, and third color space centers 1006, 1012, 1018 indicate a shift in the color space of the image capture device over time. As mentioned above, updated calibration parameters can be generated by the central calibration server 602 based on this detected shift in the color space of the image capture device.

[0123] 11 is a flow chart illustrating an example process 1100 for performing a calibration parameter lookup. At 1102, the central calibration server 602 sends a first set of calibration parameters to a first controller system, such as the electronics cart 56 and / or processor 58 of a first teleoperated surgical system 604. At 1104, the central calibration server 602 receives one or more lookup results from the first controller system based on user feedback provided in response to the first set of calibration parameters.

[0124] At 1106, the central calibration server 602 transmits the second set of calibration parameters to a second controller system, such as the electronics cart 56 and / or processor 58 of the second teleoperated surgical system 606. At 1108, the central calibration server 602 receives one or more survey results from the second controller system based on user feedback provided in response to the second set of calibration parameters.

[0125] At 1110, the central calibration server 602 processes the survey received in response to the first and second sets of calibration parameters and determines an optimized set of calibration parameters based on the received survey. For example, the central calibration server 602 may select the highest rated calibration parameters from among the first and second sets of calibration parameters to determine the optimized set of calibration parameters. Other methods of processing the survey results may also be used to determine the optimized set of calibration parameters.

[0126] One or more features of process 1100 may be optional and / or may be performed in a different order. For example, the features performed at 1106-1108 may be performed prior to the features performed at 1102-1104. Other additions or variations of process 1100 are contemplated by this disclosure.

[0127] It should be understood that the logical operations described herein with respect to the various figures may be implemented (1) as a sequence of computer-implemented operations or program modules (i.e., software) operating on a computing device (e.g., the computing device described in FIG. 12); (2) as interconnected machine logic circuits or circuit modules (i.e., hardware) within a computing device; and / or (3) as a combination of software and hardware in a computing device. Thus, the logical operations described herein are not limited to any particular combination of hardware and software. The implementation is a matter of choice dependent on the performance and other requirements of the computing device. Thus, the logical operations described herein are variously referred to as operations, structural devices, acts, or modules. These operations, structural devices, acts, and modules may be implemented in software, firmware, special purpose digital logic, and any combination thereof. It should also be understood that more or fewer operations may be performed than illustrated in the figures and described herein. These operations may also be performed in different orders than those described herein.

[0128] Referring to FIG. 12, an example of a computing device 1200 on which an embodiment of the present invention may be implemented is shown. For example, each of the computer processors located on the electronics cart 56 or electronics cart 24 described herein, the computer processor 58, and the central calibration server 602 may each be implemented as a computing device such as the computing device 1200. It should be understood that the exemplary computing device 1200 is only one example of a suitable computing environment on which an embodiment of the present invention may be implemented. Optionally, the computing device 1200 may be a well-known computing system, including, but not limited to, a personal computer, a server, a handheld or laptop device, a multiprocessor system, a microprocessor-based system, a network personal computer (PC), a minicomputer, a mainframe computer, an embedded system, and / or a distributed computing environment including a plurality of the above systems or devices. A distributed computing environment enables remote computing devices connected to a communication network or other data transmission medium to perform various tasks. In a distributed computing environment, program modules, applications, and other data are stored on local and / or remote computer storage media.

[0129] In one embodiment, the computing device 1200 may have two or more computers communicating with each other that cooperate to perform a task. For example, but not limited to, an application may be divided in a manner that allows for simultaneous and / or parallel processing of the application's instructions. Alternatively, data processed by an application may be divided in a manner that allows for simultaneous and / or parallel processing of different portions of a data set by two or more computers. In one embodiment, virtualization software may be used by the computing device 1200 to provide the functionality of several servers that are not directly bound to the number of computers in the computing device 1200. For example, the virtualization software may provide 20 virtual servers on four physical computers. In one embodiment, the functionality disclosed above may be provided by running the application and / or multiple applications in a cloud computing environment. Cloud computing may include providing computing services over a network connection using dynamically scalable computing resources. Cloud computing may be supported, at least in part, by virtualization software. The cloud computing environment may be established by an enterprise or may be hired from a third party provider as needed. A cloud computing environment may include cloud computing resources that are owned and operated by an enterprise, or cloud computing resources that are rented or leased from a third-party provider.

[0130] In its most basic configuration, computing device 1200 typically includes at least one processing unit 1220 and a system memory 1230. Depending on the exact configuration and type of computing device, system memory 1230 may be volatile (such as random access memory (RAM)), non-volatile (such as read-only memory (ROM), flash memory, etc.), or a combination of the two. This most basic configuration is illustrated in FIG. 12 by dashed line 1210. Processing unit 1220 may be a standard programmable processor that performs arithmetic and logical operations necessary for the operation of computing device 1200. Although only one processing unit 1220 is shown, multiple processors may be present. Thus, while instructions may be discussed as executed by a processor, instructions may be executed simultaneously, serially, or otherwise by one or multiple processors. Computing device 1200 may also include a bus or other communication mechanism for communicating information between various components of computing device 1200.

[0131] Computing device 1200 may have additional features / functionality. For example, computing device 1200 may include additional storage such as removable storage 1240 and non-removable storage 1250, including, but not limited to, magnetic or optical disks or tape. Computing device 1200 may also include network connection(s) 1280 that allow the device to communicate with other devices, such as via communication paths described herein. Network connection(s) 1280 may take the form of a modem, modem bank, Ethernet card, Universal Serial Bus (USB) interface card, serial interface, token ring card, Fiber Distributed Data Interface (FDDI) card, Wireless Local Area Network (WLAN) card, radio transceiver card such as Code Division Multiple Access (CDMA), Global System for Mobile Communications (GSM), Long Term Evolution (LTE), Worldwide Interoperability for Microwave Access (WiMAX), and / or other air interface protocol radio transceiver cards, and other well-known network devices. Computing device 1200 may also have input device(s) 1270, such as a keyboard, keypad, switches, dials, mouse, trackball, touch screen, voice recognition device, card reader, paper tape reader, or other well-known input devices. Output device(s) 1260, such as a printer, video monitor, liquid crystal display, touch screen display, display, speakers, etc., may also be included. Additional devices may be coupled to the bus to facilitate data communication between components of computing device 1200. All of these devices are well-known in the art and need not be described at length here.

[0132] The processing unit 1220 may be configured to execute program code encoded on a tangible computer-readable medium. A tangible computer-readable medium refers to any medium that can provide data that causes the computing device 1200 (i.e., machine) to operate in a particular manner. A variety of computer-readable media may be utilized to provide instructions to the processing unit 1220 for execution. Examples of tangible computer-readable media include, but are not limited to, volatile, non-volatile, removable, and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. The system memory 1230, the removable storage 1240, and the non-removable storage 1250 are all examples of tangible computer storage media. Examples of tangible computer-readable recording media include, but are not limited to, integrated circuits (e.g., field programmable gate arrays or application specific ICs), hard disks, optical disks, magneto-optical disks, floppy disks, magnetic tapes, holographic storage media, solid state devices, RAM, ROM, electrically erasable program read only memory (EEPROM), flash memory or other memory technologies, CD-ROMs, digital versatile disks (DVDs), or other optical storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices, or other magnetic storage devices.

[0133] It is fundamental to the art of electrical and software engineering that functionality that can be implemented by loading executable software into a computer can be transformed into a hardware implementation by well-known design rules. The decision between implementing a concept in software and hardware typically depends on considerations of design stability and number of units to be produced, rather than any issues involved in translating from the software domain to the hardware domain. Designs that still change frequently are preferably implemented in software, since re-spinning a hardware implementation is generally more expensive than redesigning a software design. Generally, stable designs that are mass-produced are preferably implemented and performed in hardware, e.g., application specific integrated circuits, because in large-scale production, hardware implementations are less expensive than software implementations. Often, designs are developed and tested in software form, and later transformed by well-known design rules into an equivalent hardware implementation in an application specific integrated circuit that wires the software's instructions. Just as a machine controlled by a new ASIC is a specific machine or device, similarly a computer programmed and / or loaded with executable instructions can be viewed as a specific machine or device.

[0134] In the illustrated implementation, the processing unit 1220 may execute program code stored in the system memory 1230. For example, a bus may carry data to the system memory 1230, from which the processing unit 1220 receives and executes the instructions. Data received by the system memory 1230 may optionally be stored on a removable storage device 1240 or a non-removable storage device 1250 before or after execution by the processing unit 1220.

[0135] It should be understood that the various techniques described herein may be implemented in connection with hardware or software, or, where appropriate, with a combination thereof. Thus, the methods and apparatus of the presently disclosed subject matter, or certain aspects or portions thereof, may take the form of program code (i.e., instructions) embodied in a tangible medium, such as a floppy diskette, CD-ROM, hard drive, or any other machine-readable storage medium, where the program code, when loaded and executed on a machine, such as a computing device, makes the machine an apparatus for implementing the presently disclosed subject matter. In the case of program code execution on a programmable computer, the computing device generally includes a processor, a processor-readable storage medium (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. One or more programs may implement or utilize the processes described in connection with the presently disclosed subject matter, for example, through the use of application programming interfaces (APIs), reusable controls, or the like. Such programs may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, the program(s) can be implemented in assembly or machine language, if desired. In either case, the language can be a compiled or interpreted language, and combined with hardware implementations.

[0136] Embodiments of the methods and systems may be described herein with reference to block diagrams and flowchart illustrations of methods, systems, apparatuses, and computer program products. It will be understood that each block of the block diagrams and flowchart illustrations, and combinations of blocks in the block diagrams and flowchart illustrations, respectively, can be implemented by computer program instructions. These computer program instructions may be loaded into a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine such that the instructions, which execute on the computer or other programmable data processing apparatus, create means for implementing the functionality specified in the flowchart block or block.

[0137] These computer program instructions may also be stored in a computer readable memory that can cause a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory create an article of manufacture that includes computer readable instructions for implementing the functions specified in the flowchart block or blocks. The computer program instructions may also be loaded into a computer or other programmable data processing apparatus and cause the computer or other programmable apparatus to perform a series of operational steps, such that the instructions executing on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block, to create a computer implemented process.

[0138] Thus, the blocks in the block diagrams and flowchart diagrams support combinations of means for performing the specified functions, combinations of steps for performing the specified functions, and program instruction means for performing the specified functions. It will also be understood that each block of the block diagrams and flowchart diagrams, and combinations of blocks in the block diagrams and flow diagrams, can be implemented by a special purpose hardware-based computer system that performs the specified functions or steps, or a combination of special purpose hardware and computer instructions.

[0139] Although several embodiments have been provided in this disclosure, it should be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit or scope of the disclosure. The examples should be considered as illustrative and not limiting, and the intention is not to be limited to the details given herein. For example, various elements or components may be combined or integrated into another system, or certain features may be omitted or not implemented.

[0140] Additionally, the techniques, systems, subsystems, and methods described and illustrated as discrete or separate in various embodiments may be combined or integrated with other systems, modules, techniques, or methods without departing from the scope of the disclosure. Other items shown or described as being directly coupled or in communication with one another may be indirectly coupled or in communication through some interface, device, or intermediate component, whether electrical, mechanical, or otherwise. Other examples of changes, substitutions, and alterations will be ascertainable by one of ordinary skill in the art and may be made without departing from the spirit and scope disclosed herein.

Claims

1. a network-accessible database configured to store calibration data for the image capture devices; a controller system having an image processor configured to receive an image captured by the image capture device, the image processor further configured to retrieve the calibration data from the network-accessible database via a network connection, the image processor further configured to process the received image based on the retrieved calibration data; and a calibration processor in communication with the network-accessible database, the calibration processor configured to: (i) receive calibration images captured by the image capture device; and (ii) generate the calibration data in response to a change in a configuration of the controller system, the calibration data being generated based on applying a calibration algorithm to one or more of the calibration images, the calibration algorithm being selected based on one or more of: a changed type of procedure for which the image capture device will be used, a changed configuration of the controller system, or an improved calibration method for calculating one or more parameters of the calibration data. system.

2. a tool having the image capture device and electrically coupled to the controller system, the tool having a unique identifier. The system of claim 1 .

3. the calibration data is indexed with the image capture device in the network-accessible database based on the unique identifier; The system of claim 2.

4. the unique identifier is maintained by the tool, and the image processor is further configured to obtain the unique identifier from the tool. The system of claim 3.

5. the unique identifier is maintained by the tool on a memory device of the tool, on a radio frequency tag of the tool, or on a visual indicator on a surface of the tool; The system of claim 4.

6. the image processor further comprising an optical scanner configured to read the visual indicator from the surface of the tool to obtain the unique identifier from the tool. The system of claim 5.

7. the optical scanner is disposed in a mating receptacle configured to couple the tool to the controller system; The system of claim 6.

8. The optical scanner is a camera or a barcode reader. The system of claim 6.

9. the visual indicator is printed, glued, labeled, embossed, or stamped onto the surface of the tool; A system according to any one of claims 5 to 8.

10. the visual indicator is an alphanumeric serial number, a one-dimensional barcode encoding the unique identifier, a two-dimensional barcode encoding the unique identifier, or an image encoded with the unique identifier; A system according to any one of claims 5 to 8.

11. the tool comprises an endoscope and the image capture device comprises a pair of image sensors configured to capture volumetric image data of a surgical site during a surgical procedure. A system according to any one of claims 2 to 8.

12. the calibration data includes data for processing the received image to correct the received image based on one or more of: a color of light emitted by an illumination device configured to illuminate a scene captured by the received image, the calibration image captured by the image capture device, optical distortions introduced by an optical system of the image capture device, stereoscopic image calibration, color space mapping, or image sensor characteristics of the image capture device. A system according to any preceding claim.

13. the controller system further comprising a calibration cache memory configured to store the calibration data retrieved from the network-accessible database; the controller system is further configured to store calibration data for the image capture devices known to have been dispatched to locations associated with the location of the controller system; The controller system is further configured to store calibration data for a predetermined number of previously used image capture devices; The controller system is further configured to store default calibration data for a type of the image capture device; the controller system is further configured to compare a cached set of calibration data for the image capture device to the calibration data stored in the network-accessible database prior to retrieving the calibration data from the network-accessible database. A system according to any preceding claim.

14. a calibration processor in communication with a network-accessible database receiving from a controller system for an image capture device a calibration image captured by the image capture device; the calibration processor generating calibration data in response to a change in a configuration of the controller system, the calibration data being generated based on applying a calibration algorithm to one or more of the calibration images, the calibration algorithm being selected based on one or more of: a changed type of procedure for which the image capture device will be used, a changed configuration of the controller system, or an improved calibration method for calculating one or more parameters of the calibration data; the controller system storing the calibration data in the network accessible database; the controller system retrieving the calibration data from the network-accessible database via a network connection; receiving, by the controller system, a second image captured by the image capture device; and an image processor of the controller system processing the received second image based on the retrieved calibration data. method.

15. a network-accessible database configured to store calibration data for the endoscopic image capture device; an image processor configured to be communicatively coupled to the endoscopic image capture device to receive image data of a surgical site during a surgical procedure, the image processor configured to retrieve the calibration data from the network-accessible database via a network connection, the image processor further configured to process the received image data based on the retrieved calibration data; a calibration processor in communication with the network-accessible database, the calibration processor configured to: (i) receive calibration images captured by the endoscopic image capture device; and (ii) generate the calibration data in response to a change in configuration of the image processor, the calibration data being generated based on applying a calibration algorithm to one or more of the calibration images, the calibration algorithm being selected based on one or more of: a changed type of procedure for which the endoscopic image capture device will be used, a changed configuration of the image processor, or an improved calibration method for calculating one or more parameters of the calibration data; a user control system coupled to the image processor and configured to receive the processed image data from the image processor, the user control system having a display configured to display the processed image data; a manipulator system configured to attach to the endoscopic image capture device and having a manipulator arm configured to physically change the orientation of the endoscopic image capture device during the surgical procedure; Surgery system.

Images