Method for a system for processing images using extrinsic camera calibration parameters

Abstract

A surgical system includes a network accessible database configured to store calibration data for an endoscopic image capture device. The surgical system also includes an image processor configured to be communicatively coupled to the endoscopic image capture device to receive image data of a surgical site during a surgical procedure or diagnostic routine. The image processor is configured to retrieve the calibration data from the network accessible database via a network connection. The image processor is also configured to process the received image based on the retrieved calibration data. The surgical system also includes a user control system coupled to the image processor and configured to receive processed images from the image processor. The user control system includes a display configured to display the processed image. The calibration data is updated periodically or dynamically on the database to account for changes to the surgical system or new calibration methods.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Patent Application Serial No. 62 / 722,314, filed on August 24, 2018, the disclosure of which is expressly incorporated herein by reference. Technical Field

[0003] The present disclosure relates to processing images, and more particularly to methods for systems for processing images using calibration parameters. Background Art

[0004] Image capture devices are typically factory calibrated prior to use in the field. During the calibration process, one or more images of a test pattern are captured and used as a reference to generate calibration data for the image capture device. For example, one or more transformations are used to adjust the image captured from the image capture device to align with a known image of the test pattern. The calibration process can calibrate the captured image to correct for color, optical, alignment, or other image sensor-induced or optically-induced errors or variations in the image capture device. The calibration process generates a calibration data set that is stored with the image capture device and applied to correct images captured by the image capture device in use. Summary of the invention

[0005] A first aspect of the present disclosure includes a system comprising a network accessible database configured to store calibration data for an image capture device. The system also includes a controller system comprising an image processor configured to receive an image captured by the image capture device. The image processor is further configured to retrieve the calibration data from the network accessible database via a network connection. The image processor is further configured to process the received image based on the retrieved calibration data.

[0006] In some embodiments of the first aspect of the present disclosure, the system further comprises a tool comprising the image capture device and electrically coupled to the controller system.The tool has a unique identifier.

[0007] In some embodiments of the first aspect of the present disclosure, the calibration data is indexed by the image capture device in a network accessible database based on the unique identifier.

[0008] In some embodiments of the first aspect of the present disclosure, the unique identifier is maintained by the tool, and wherein the image processor is further configured to obtain the unique identifier from the tool.

[0009] In some embodiments of the first aspect of the present disclosure, the unique identifier is maintained by the tool on a memory device of the tool or on a visual indicator on a surface of the tool.

[0010] In some embodiments of the first aspect of the present disclosure, the image processor further comprises an optical scanner configured to read the visual indicator from the surface of the tool to obtain the unique identifier from the tool.

[0011] In some embodiments of the first aspect of the present disclosure, the optical scanner is a camera or a barcode reader.

[0012] In some embodiments of the first aspect of the present disclosure, the visual indicator is printed, adhered, marked, embossed or stamped on a surface of the tool.

[0013] In some embodiments of the first aspect of the disclosure, the visual indicator is an alphanumeric serial number, a linear barcode encoding the unique identifier, a two-dimensional barcode encoding the unique identifier, or an image encoded with the unique identifier.

[0014] In some embodiments of the first aspect of the present disclosure, the tool includes an endoscope, and the image capture device includes a pair of image sensors configured to capture stereoscopic image data of a surgical site during a surgical procedure.

[0015] In some embodiments of the first aspect of the present disclosure, the calibration data includes data for processing a received image to correct the received image based on one or more of the following items: the color of light emitted by an illuminator configured to illuminate a scene captured by the received image, a calibration image captured by an image capture device, optical distortion introduced by an optical system of the image capture device, stereo image calibration, or color space mapping.

[0016] In some embodiments of the first aspect of the present disclosure, the system further comprises a calibration processor in communication with the network accessible database. The calibration processor is configured to receive calibration images captured by the image capture device and generate calibration data based on applying the calibration algorithm to one or more calibration images.

[0017] In some embodiments of the first aspect of the present disclosure, the calibration image is stored together with the calibration data in a network accessible database.

[0018] In some embodiments of the first aspect of the present disclosure, the calibration processor is configured to generate the calibration data after receiving a request for the calibration data from the controller system.

[0019] In some embodiments of the first aspect of the present disclosure, the calibration data is different from a previous calibration data set for the image capture device, wherein the previous calibration data set was generated by the calibration processor based on applying a different calibration algorithm to one or more calibration images.

[0020] In some embodiments of the first aspect of the present disclosure, the calibration algorithm is selected based on the type of routine in which the image capture device is to be used or based on the configuration of the controller system.

[0021] In some implementations of the first aspect of the present disclosure, calibration data is generated to enhance image features that are important for the type of routine in which the image capture device is to be used.

[0022] In some embodiments of the first aspect of the present disclosure, the image characteristic is one or more of focus level, sharpness, color fidelity, or color accuracy.

[0023] In some embodiments of the first aspect of the present disclosure, the calibration data further adversely affects the second image feature.

[0024] In some embodiments of the first aspect of the present disclosure, the configuration of the controller system includes one or more of an illumination source provided by the controller system, a type of image processor, or a software release version of the controller system.

[0025] In some embodiments of the first aspect of the present disclosure, the calibration processor is further configured to retrieve an operation image captured by the image capture device from the controller system during the routine, and adjust the calibration data based on the operation image.

[0026] In some embodiments of the first aspect of the present disclosure, the illuminance value of the operating image is compared with an expected illuminance value determined based on a known distance from an object captured in the operating image and a known power output to an illumination source that illuminates the object captured in the operating image.

[0027] In some embodiments of the first aspect of the present disclosure, a color space of the operating image is compared with an expected color space of the image capture device to determine whether a center of the color space deviates from a center of the expected color space.

[0028] In some implementations of the first aspect of the present disclosure, a stereoscopic alignment of the operational image is evaluated to identify a misalignment with a corresponding operational image.

[0029] In some embodiments of the first aspect of the present disclosure, the calibration processor is further configured to receive a user feedback survey evaluating the calibration data compared to previous calibration data sets.

[0030] In some embodiments of the first aspect of the present disclosure, the controller system further comprises a calibration cache memory configured to store calibration data retrieved from the network accessible database.

[0031] In some embodiments of the first aspect of the present disclosure, the controller system is further configured to store calibration data for image capture devices known to have been transported to a location associated with the location of the controller system.

[0032] In some embodiments of the first aspect of the present disclosure, the controller system is further configured to store calibration data of a predetermined number of previously used image capture devices.

[0033] In some embodiments of the first aspect of the present disclosure, the controller system is further configured to store default calibration data for a type of image capture device.

[0034] In some embodiments of the first aspect of the present disclosure, the controller system is further configured to compare the cached calibration data set for the image capture device with the calibration data stored on the network accessible database before retrieving the calibration data from the network accessible database.

[0035] A second aspect of the present disclosure includes a method comprising storing calibration data for an image capture device in a network accessible database. The method also comprises retrieving, by a controller system, the calibration data from the network accessible database via a network connection. The method also comprises receiving, by the controller system, an image captured by the image capture device. The method also comprises processing, by an image processor of the controller system, the received image based on the retrieved calibration data.

[0036] A third aspect of the present disclosure includes a surgical system comprising a network accessible database configured to store calibration data for an endoscopic image capture device. The surgical system also includes an image processor configured to be communicatively coupled to the endoscopic image capture device to receive image data of a surgical site during a surgical routine. The image processor is configured to retrieve the calibration data from the network accessible database via a network connection. The image processor is also configured to process the received image based on the retrieved calibration data. The surgical system also includes a user control system coupled to the image processor and configured to receive a processed image from the image processor. The user control system includes a display configured to display the processed image.

[0037] In some embodiments of the third aspect of the present disclosure, the surgical system further comprises a manipulator system comprising a manipulator arm configured to be attached to the endoscopic image capture device and configured to physically change a posture of the endoscopic image capture device during a surgical routine.

[0038] In some embodiments of the third aspect of the present disclosure, the user control system further comprises a manipulation user interface. After the manipulation user interface is activated, the user control system is configured to issue a command to the manipulator system to change the posture of the endoscopic image capture device.

[0039] A fourth aspect of the present disclosure includes a method of processing an image of a surgical procedure routine captured by an endoscopic image capture device. The method includes receiving, by an image processor, an image captured by the endoscopic image capture device. The method also includes retrieving, by the image processor, calibration data for the endoscopic image capture device from a network accessible database. The method also includes processing, by the image processor, the image received from the endoscopic image capture device based on the retrieved calibration data. The method also includes transmitting, by the image processor, the processed image to a user control system. The method also includes displaying the processed image on a display on the user control system.

[0040] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] For a more complete understanding of the present disclosure, reference is now made to the following brief description taken in conjunction with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.

[0042] Figure 1 is a plan view of a minimally invasive teleoperated surgical system.

[0043] Figure 2 is the user's perspective on controlling the system.

[0044] Figure 3 is a perspective view of the electronics cart.

[0045] Figure 4 is a schematic diagram of a remotely operated surgical system.

[0046] Figure 5 is a perspective view of an endoscopic image capture device.

[0047] Figure 6 is a schematic diagram of an image capture device calibration tool.

[0048] Figure 7 is a flow chart illustrating an example process for obtaining calibration data for an image capture device.

[0049] Figure 8 is a flow chart illustrating an example process for sending calibration data in response to a request.

[0050] Fig. 9is a flow chart illustrating an example process for determining changed calibration parameters or maintenance conditions based on live images captured by an image capture device.

[0051] FIG. 10A to FIG. 10D is a chromaticity diagram generated from a processed live image that shows the shift in the center of the color space of the image capture device.

[0052] Fig.11 is a flow chart illustrating an example process for conducting a calibration parameter survey.

[0053] Fig.12 An exemplary computer system is shown. DETAILED DESCRIPTION

[0054] First, it should be understood that although illustrative implementations of one or more embodiments are shown below, the disclosed systems and methods can be implemented using any number of technologies, whether or not the technology is currently known or exists. The present disclosure should in no way be limited to the illustrative implementations, drawings, and techniques shown below, but can be modified within the full scope of the appended claims and their equivalents. The use of the phrase "and / or" indicates that any one or any combination of the list of options can be used. For example, "A, B, and / or C" means "A", or "B", or "C", or "A and B", or "A and C", or "B and C", or "A, B, and C".

[0055] An element described in detail with reference to one embodiment, implementation or application may be included in other embodiments, implementations or applications where it is not specifically shown or described, as far as feasible. For example, if an element is described in detail with reference to one embodiment, but not described with reference to a second embodiment, protection may still be requested in the second embodiment for including the element. Therefore, to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment, implementation or application may be incorporated into other embodiments, implementations or aspects unless otherwise specifically stated, unless one or more elements would render an embodiment or implementation nonfunctional, or unless two or more elements provide conflicting functions.

[0056] Based on the use of the da Vinci surgical system Surgical System) (specifically, model IS4000, with da Xi TM HD TMVarious aspects of the present invention are described herein with reference to embodiments of the da Vinci Surgical System (e.g., Model IS4000 da Vinci Surgical System) sold by Intuitive Surgical, Inc. of Sunnyvale, California. However, those skilled in the art will appreciate that the inventive aspects disclosed herein may be embodied and implemented in a variety of ways, including robotic and, if applicable, non-robotic embodiments and implementations. Xi TM HD TM Surgical system / model IS3000da The embodiments of the surgical system are merely exemplary and should not be considered as limiting the scope of the inventive aspects disclosed herein.

[0057] According to various aspects, the present disclosure describes a system and method for off-camera calibration of an image capture device. Calibration data is typically maintained in read-only memory of the image capture device and cannot be changed or updated after the image capture device leaves the factory. Calibration data for an image capture device typically does not take into account wear and tear of the device over time. Therefore, over time, the calibration data stored on the image capture device may become less accurate and result in processed images from the image capture device having more errors.

[0058] For example, an endoscopic image capture device includes a camera housing including a flexible cable having an optical fiber bundle. The cable includes a connector configured to couple the optical fiber bundle to a light source. The camera housing also includes a read-only memory that stores calibration data for the endoscopic image capture device. The camera housing also includes a rigid camera shaft having a camera tip at a distal end. The camera tip includes one or more image sensors and an associated optical system. The camera shaft also includes a second optical fiber bundle configured to deliver light received from the optical fiber bundle in the flexible cable to the camera tip to illuminate a scene imaged by the one or more image sensors (such as a diagnostic or surgical routine). Images captured by the one or more image sensors in the camera tip are delivered to the camera housing via a wired or wireless connection, and in turn delivered to the connector via a wired or wireless connection in the flexible cable.

[0059] A controller system includes a socket configured to receive a connector. The controller system includes a light source coupled to the socket and configured to supply light to an optical fiber bundle in a flexible cable. The controller system also includes an image processor coupled to the socket and configured to receive an image transmitted via an electrical connection in the flexible cable. The image processor is configured to read calibration data from a read-only memory in a camera housing after the connector is inserted into the socket. The image processor processes the received image based on the read calibration data to generate one or more processed images. When an endoscopic image capture device is used, the optical transmission properties of the optical fiber bundle in the flexible cable may change over time. In addition, due to use in medical routines, the camera tip may wear out over time. Therefore, at least the optical properties of the endoscopic image capture device will change over time, resulting in the calibration data not being optimized for the current condition of the endoscopic image capture device.

[0060] Additionally, the endoscopic image capture device may be coupled to different controller systems at different times, wherein one or more of the controller systems may have different configurations. Different controller systems may have different system configurations, such as different image processors, vision systems or other hardware variations, different software releases, and / or different light sources with different spectral responses. Therefore, the calibration data may not be optimized for the particular controller system to which the endoscopic image capture device is currently coupled.

[0061] Furthermore, advances in transforms for generating calibration data from captured test patterns cannot be retroactively applied to existing surgical endoscopic image capture devices. Thus, based on current knowledge and technology, it may not be possible to optimize the calibration data stored on a surgical endoscopic image capture device.

[0062] In various embodiments of the present disclosure, the calibration data is moved outside the camera and obtained by a controller system from a central calibration server. After an image capture device, such as an endoscope image capture device, is connected to the controller system, the controller system obtains a unique identifier of the image capture device. Although various examples are described herein for an endoscope image capture device, the present disclosure is not limited thereto, but is intended to cover any device coupled to a controller system that has corresponding calibration data to control the operation of the device or process data received from the device. Similarly, the present disclosure is intended to cover any image capture device coupled to a controller system to process images captured by the image capture device. For example, the present disclosure may be equally applicable to a borescope or other such inspection camera.

[0063] The unique identifier may be maintained in a memory of the image capture device. Alternatively, a reader coupled to the controller system reads the unique identifier from the image capture device. For example, the reader may be a radio frequency identifier (RFID), a low power Bluetooth ( low energy, BLE), or other wireless tag reader, and the unique identifier can be stored in a wireless tag, such as RFID, BLE, or other wireless logo.

[0064] In another example, the unique identifier is provided as an optically readable label on the housing of the image capture device. The label can include an alphanumeric string, symbol or code in an encrypted or non-encrypted format. The label can be embossed, glued, adhered, printed or otherwise affixed to the housing of the image capture device. The unique identifier can be read from a camera, barcode reader or other optical scanner coupled to the controller system or otherwise communicating therewith.

[0065] Using the obtained unique identifier of the image capture device, the controller system obtains calibration data for the image capture device from the central calibration server. The controller system then processes the image received from the image capture device based on the retrieved calibration data. In some embodiments, the unique identifier is a universally unique identifier (UUID) of the medical device.

[0066] By offloading the calibration data from the image capture device, the memory requirements of the image capture device can be reduced, resulting in a cheaper image capture device. For example, a newly produced image capture device can omit the read-only calibration memory, or reduce the size of the read-only memory to store only the unique identifier of the image capture device.

[0067] Calibration data may be generated periodically or dynamically at a central calibration server. Thus, the method for calculating one or more calibration parameters may be changed based on the type of routine to be performed, based on the current instrument configuration, and / or based on improvements to the calibration method. Furthermore, obtaining calibration data from a central calibration server extends the life and / or functionality of existing instruments in the field, which may otherwise store outdated calibration data or calibration data that is not optimized for a particular routine.

[0068] Because the controller system communicates with the central calibration server on a regular basis, additional functionality can be enabled. For example, images captured from an image capture device can be periodically transmitted to a central calibration server. These live images can be processed to identify improvements to the calibration data for a particular image capture device over time, or to flag the image capture device for maintenance. For example, due to wear of the fiber optic bundle in a flexible cable, live captured images may exhibit spectral attenuation in the captured image over time. The spectral attenuation can be corrected by adjustments to the calibration data or to the light source provided to the image capture device. Similarly, the shift in the color space of one or more image sensors over time can be corrected by adjustments to the calibration data or to the light source provided to the image capture device. In addition, alignment problems in stereo images can be identified and flagged for maintenance.

[0069] As another example, preference studies may be conducted on end users of image capture devices with different calibration data sets. For example, different color mapping or distortion corrections may be used in different calibration data sets, and feedback from end users may be obtained to inform the optimized calibration data set. Additional controls may be added to prevent the controller system from being used with unregistered image capture devices, facilitate recall control of image capture devices, and limit the total number of uses of an image capture device before refurbishment is required.

[0070] Referring now to the drawings, wherein like reference numerals refer to like parts throughout the several views. Figure 1 1 is a plan view of a minimally invasive teleoperated surgical system 10, which is typically used to perform minimally invasive diagnostic or surgical routines on a patient 12 lying on a mobile operating table 14. The system includes a user control system 16, such as a mobile surgeon's console used by a surgeon 18 during the routine. One or more assistants 20 may also participate in the routine. The minimally invasive teleoperated surgical system 10 also includes a manipulation system 22 (such as a mobile patient side cart) and a mobile electronics cart 24. In some embodiments, the table 14, user control system 16, manipulation system 22, and electronics cart 24 are mounted with wheels to provide mobility.

[0071] The manipulation system 22 or other such manipulation system includes a plurality of segmented mechanical support arms 72, each mechanical support arm 72 having one end rotatably mounted to a vertical support structure 74 and having the other end mounted with a removably coupled surgical instrument 26. In some embodiments, each mechanical support arm 72 includes a first segment 72-1, a second segment 72-2, and a third segment 72-3. During routine setup, the plurality of segments of at least one support arm 72 are moved to position the surgical instrument for insertion into a minimally invasive incision in the body of the patient 12.

[0072] During the routine, while inserting instruments into the patient's body cavity, the surgeon 18 views the surgical site through the user control system 16. Images of the surgical site may be obtained through an endoscope 28, such as a stereo endoscope, which may be manipulated by the manipulation system 22 to orient the endoscope 28. The computer processor(s) located on the electronics cart 24 may be used to process the images of the surgical site for subsequent display to the surgeon 18 through the user control system 16. The computer processor(s) may be alternatively referred to herein as an image processor or a video processor.

[0073] One or more illumination sources or illuminators may also be provided on the electronics cart 24 to provide light used by the endoscope 28 to illuminate the surgical site. The illuminators may include white light sources, colored light sources (e.g., red, green, blue, cyan, magenta, yellow, etc.), infrared light sources, laser light sources, or any other type of light source or combination thereof. Different illuminators may be used at different points in a surgical or diagnostic routine. For example, the electronics cart 24 may be controlled, such as by selection on the user control system 16, to provide light from a first set of one or more illuminators at a first time and light from a second set of one or more illuminators at a second time.

[0074] The number of surgical instruments 26 used at one time will generally depend on the diagnostic or surgical routine and the space limitations within the operating room, among other factors. If it is necessary to change one or more surgical instruments 26 being used during a routine, the assistant 20 can remove the surgical instrument 26 from the manipulation system 22 and replace it with another surgical instrument 26 from the tray 30 in the operating room.

[0075] Figure 2 is a perspective view of the user control system 16. The user control system 16 includes a display area 31 having a left eye display 32 and a right eye display 34 for presenting the surgeon 18 with a coordinated stereoscopic view of the surgical site enabling depth perception.

[0076] The console 16 also includes one or more control inputs 36. In response to the surgeon 18 manipulating the one or more control inputs 36, the manipulation system 22 ( Figure 1 The control input 36 may provide a control for the surgical instrument 26 (as shown) to be mounted thereon and used thereon. Figure 1The same mechanical degrees of freedom as those shown in the figure above can be used to provide the surgeon 18 with telepresence or the perception that the control input 36 is integral to the instrument 26, thereby giving the surgeon a strong sense of direct control of the instrument 26. To this end, position, force, and tactile feedback sensors (not shown) can be employed to transmit position, force, and tactile sensations from the surgical instrument 26 back to the surgeon's hand through the control input 36. The height of the control input 36 can be adjusted using a height adjustment lever 38.

[0077] The user control system 16 is typically located in the same room as the patient so that the surgeon can directly monitor the procedure, be physically present if necessary, and speak directly to the patient-side assistant rather than via a telephone or other communication medium. However, the surgeon may be located in a different room from the patient, an entirely different building, or other remote location, thereby allowing a telesurgery procedure to be performed.

[0078] Figure 3 2 is a perspective view of the electronics cart 24. The electronics cart 24 may be coupled to the endoscope 28 and include a computer processor to process the captured images for subsequent display, such as on the control system 16 or on another suitable display located locally and / or remotely to the surgeon. For example, if a stereoscopic endoscope is used, the computer processor on the electronics cart 24 may process the captured images to present the surgeon with a coordinated stereoscopic image of the surgical site. Such coordination may include alignment between the relative images and may include adjusting the stereoscopic working distance of the stereoscopic endoscope.

[0079] As another example, image processing may include using previously or dynamically determined camera calibration parameters to compensate for imaging errors of the image capture device, such as optical aberrations. The calibration data may be obtained by the electronics cart 24 from a central calibration server, as discussed in more detail below.

[0080] Because the calibration data is obtained from a central calibration server rather than from the image capture device itself, the new calibration data can be used with existing image capture devices, which may store the current calibration data on the local memory of the image capture device. Thus, the service life or functionality of the existing image capture device can be extended by remotely obtaining calibration data for the existing image capture device. Likewise, the memory requirements of the new image capture device can be reduced, resulting in a less expensive image capture device. For example, a newly produced image capture device can omit the memory for storing the calibration data, or reduce the size of the memory to store only the unique identifier of the image capture device. In addition, the method for calculating one or more calibration parameters can be changed based on the type of routine to be performed, based on the current instrument configuration and / or based on improvements to the calibration method.

[0081] The electronics cart 24 may also maintain the obtained calibration data in a local cache. In addition to maintaining the obtained calibration data in a local cache, the electronics cart 24 may also maintain calibration data for one or more image capture devices intended for use with the electronics cart 24. For example, after a new image capture device is ordered, delivered, or received at a location with the electronics cart 24, the electronics cart 24 may obtain or receive calibration data for the new image capture device. In some embodiments, the central calibration server may push the calibration data for the new image capture device to be maintained on the local cache of the electronics cart 24. The local cache of the electronics cart 24 may also maintain a default calibration data set for use when there is limited or no connection to the central calibration server.

[0082] Alternatively, the instruments in the electronics cart 24 may be integrated into the user control system 16 or the manipulation system 22, or may be distributed in various other locations in the operating room. More generally, the electronics cart 24 or the user control system 16 with integrated instruments from the electronics cart 24 may be referred to herein as a controller system for obtaining calibration data and processing images from the image capture device based on the retrieved calibration data.

[0083] Figure 4 A remotely operated surgical system 50 (such as Figure 1 During a minimally invasive surgical procedure, the surgeon may use a user control system 52 (such as Figure 1 to control the operating system 54 (such as Figure 1 The manipulation system 54 may use an image capture device (such as a stereo endoscope) to capture images of the surgical site and output the captured images to an electronic cart 56 (such as a Figure 1 The electronic cart 56 includes a computer processor on the electronic cart 24 in the electronic cart. The computer processor typically includes one or more data processing boards for executing computer readable code stored in the non-volatile memory device of the computer processor. Like the electronic cart 24, the electronic cart 56 also includes one or more illumination sources for supplying light to the image capture device. The electronic cart 56 may also include a calibration cache similar to the electronic cart 24 for maintaining a local cache of calibration data.

[0084] In one aspect, the computer processor can process the captured image in various ways before any subsequent display. For example, before displaying the processed image to the surgeon via the user control system 52, the computer processor can use previously or dynamically determined camera calibration parameters to compensate for imaging errors of the image capture device. Additionally or alternatively, the captured image can be image processed by a computer processor located outside the electronic cart 56. In one aspect, the teleoperated surgical system 50 includes an optional computer processor 58 (as shown in dashed lines) similar to the computer processor located on the electronic cart 56, and the manipulation system 54 outputs the captured image to the computer processor 58 for image processing before display on the user control system 52. In another aspect, before display on the user control system 52, the captured image is first image processed by the computer processor on the electronic cart 56, and then additional image processing is performed by the computer processor 58. In some embodiments, the electronic cart 56 and / or the computer processor 58 are collectively referred to as a controller system for obtaining calibration data and processing images from the image capture device based on the retrieved calibration data. In some embodiments, the controller system also supplies light to the image capture device from one or more illumination sources.

[0085] The teleoperated surgical system 50 may include an optional display 60, as shown in phantom. The display 60 is coupled to the computer processor and the computer processor 58 located on the electronics cart 56, and captured images processed by these computer processors may be displayed on the display 60 in addition to being displayed on the display of the user control system 52. In various embodiments, the display 60 may be located on the electronics cart 56, such as the display 25 on the electronics cart 24. In some embodiments, the display 60 may be separate from the user control system 52 and the electronics cart 58.

[0086] Figure 5 5 is a perspective view of an endoscopic image capture device 500. The endoscopic image capture device 500 includes a connector 502, a flexible cable 506, a camera housing 508, a rigid camera shaft 512, and a camera tip 514. The connector 502 includes an optical port 504 and an electrical port (not shown). The connector 502 is sized and shaped to be inserted into a socket on the electronics cart 56, such as the socket 27 of the electronics cart 24 or other mating receptacle. The optical port 504 is configured to receive light supplied to the socket 27 by the electronics cart 24.

[0087] A flexible cable 506 is coupled between the connector 502 and the camera housing 508. The flexible cable 506 includes a fiber optic bundle configured to deliver light received from the optical port 504 of the connector 502 to the camera housing 508. The flexible cable 506 also includes an electrical connection configured to provide electrical communication between the electrical port of the connector 502 and the camera housing 508. The electrical connection can be a wired or wireless connection. In some embodiments, the wired connection is an electrical wire, a ladder wire, a twisted pair, a universal serial bus (USB) cable, an Ethernet cable, or other wired communication line.

[0088] The camera housing 508 receives the distal end of the fiber optic bundle from the flexible cable 506. The camera housing 508 also receives the proximal end of the rigid camera shaft 512. The distal end of the rigid camera shaft 512 includes a camera tip 514 having one or more image sensors and an associated optical system. For example, the camera tip 514 can include two image sensors with corresponding optical components for capturing stereoscopic images of a scene, such as a surgical procedure or a diagnostic routine. The rigid camera shaft 512 can also include a second fiber optic bundle configured to deliver light received at the camera housing 508 from the fiber optic bundle in the flexible cable to the camera tip 514 to illuminate the scene imaged by the one or more image sensors.

[0089] The rigid camera shaft 512 may also include a second electrical connection configured to provide electrical communication between the one or more image sensors of the camera head 514 and the camera housing 508. Images captured by the one or more image sensors in the camera head 514 are transmitted to the camera housing 508 via the electrical connection in the rigid camera shaft 512. The electrical connection may be a wired or wireless connection. In some embodiments, the wired connection is an electrical wire, a ladder wire, a twisted pair, a universal serial bus (USB) cable, an Ethernet cable, or other wired communication line.

[0090] The camera housing 508 may also include one or more camera control units (not shown) configured to provide power and control signals to capture images from one or more image sensors in the camera head 514. For example, when the camera head 514 includes two image sensors for capturing stereo images, the camera housing 508 may have a corresponding camera control unit for controlling each of the two image sensors. The one or more camera control units are also configured to transmit the captured images to the electrical port of the connector 502 for processing by the electronics cart 56 and / or the computer processor 58.

[0091] The camera housing 508 may also include a display 510 for displaying one or more operational controls of the endoscopic image capture device 500 .

[0092] The camera housing 508 may also include a read-only memory (not shown) that stores a unique identifier for the endoscopic image capture device 500. In some embodiments, the unique identifier is a universally unique identifier (UUID) of the medical device. The read-only memory does not store calibration data for the endoscopic image capture device 500. The read-only memory is smaller and less expensive than read-only memories used in previous endoscopic image capture devices that additionally store calibration data.

[0093] In some embodiments, the read-only memory in the camera housing 508 may be omitted entirely. Alternatively, a wireless identification tag mounted on or in the camera housing 508 or an optically readable label on an outer surface of the camera housing 508 may provide a unique identifier for the endoscopic image capture device 500. For example, the wireless identification tag may be an RFID, BLE, or other wireless marker. The tag may include an alphanumeric string, symbol or code in an encrypted or non-encrypted format. The tag may be embossed, glued, pasted, printed or otherwise affixed to the camera housing 508.

[0094] When an optically readable label is used to provide a unique identifier on the camera housing 508, an optical scanner coupled to or otherwise in communication with the controller system is used to read the label to obtain the unique identifier. The optical scanner can be a camera, a barcode reader, or other optical scanner. For example, any of the manipulation system 54, the electronics cart 56, and / or the computer processor 58 can include an optical scanner configured to read the label on the camera housing 508 to obtain the unique identifier of the endoscopic image capture device 500.

[0095] In one embodiment, the manipulator system 54 includes a camera (not shown) configured to capture an image of the camera housing 508 of the endoscopic image capture device 500 when the endoscopic image capture device 500 is mounted on the manipulator system 54. The electronics cart 56 and / or computer processor 58 processes the image captured by the camera on the manipulator system 54 to identify the unique identifier in the captured image.

[0096] In another embodiment, the receptacle 27 or other mating receptacle of the electronics cart 24 includes an optical scanner. A surface of the connector 502 or optical port 504 includes an optically readable label and is configured to be in the field of view of the optical scanner or otherwise face the optical scanner when the connector 502 is inserted into the receptacle 27. The optical scanner of the electronics cart 24 is configured to read the label after the connector 502 is inserted into the receptacle 27 to obtain the unique identifier of the endoscopic image capture device 500. Thus, the user is not required to perform any additional actions to scan the unique identifier other than inserting the connector 502 into the receptacle 27. Some users may attempt to "spoof" the electronics cart 24 by scanning a photo or other copy of the optically readable label for an unauthorized tool or endoscopic image capture device 500. By having an optical scanner in the receptacle 27, it becomes more difficult to spoof the electronics cart 24 because it is difficult to provide both a copy of the optically readable label and the connector 502 to the receptacle 27 at the same time.

[0097] In further embodiments, the electronics cart 24 includes an optical scanner at a location other than in the receptacle 27 or other mating receptacle. For example, the optical scanner may be located on a surface of the electronics cart 24 adjacent to the receptacle 27, or in a handheld scanner device coupled to the electronics cart 24. The optical scanner may also be located on other devices other than the electronics cart 24, such as the manipulation system 54 and / or the computer processor 58 as described above. By positioning the optical scanner at a location other than the receptacle 27, a corresponding calibration data set for the endoscopic image capture device 500 or other tool may be retrieved before the endoscopic image capture device 500 or other tool is used or inserted into the receptacle 27. In this manner, after the optical scanner scans the optically readable label, the corresponding calibration data set is retrieved without counting the use of the endoscopic image capture device 500 or other tool. Furthermore, while the endoscopic image capture device 500 or other tool is in use or has been inserted into the receptacle 27, an additional endoscopic image capture device 500 or other tool may be scanned to retrieve a corresponding calibration data set.

[0098] Figure 6 6 is a schematic diagram of an image capture device calibration tool 600. The image capture device calibration tool 600 includes a first teleoperated surgical system 604 and a second teleoperated surgical system 606 in data communication with a central calibration server 602. The teleoperated surgical systems 604 and 606 are similar to Figure 4The teleoperated surgical system 50 shown and described above. In some embodiments, the teleoperated surgical systems 604, 606 have the same system configuration. In some embodiments, the teleoperated surgical systems 604, 606 have different system configurations. For example, the teleoperated surgical systems 604, 606 may have differences in one or more aspects of an image processor, a visual system or other hardware system, a software release version, and / or a light source with a different spectral response. The present disclosure contemplates other differences that affect the calibration of the image capture device.

[0099] Despite Figure 6 Only two teleoperated surgical systems 604, 606 are shown, but more or fewer systems may be in data communication with the central calibration server 602. In some embodiments, the teleoperated surgical systems 604, 606 may be located at the same or different healthcare provider locations. For example, the first teleoperated surgical system 604 may be located at a first healthcare provider location, and the second teleoperated surgical system 606 may be located at a second healthcare provider location.

[0100] The central calibration server 602 includes or communicates with one or more databases storing calibration records. Each calibration record is indexed for the unique identifier of the corresponding image capture device. The calibration record includes one or more calibration data sets and a test pattern image set. Different calibration data sets can be optimized for different routines or controller system configurations. Each calibration data set includes calibration parameters for adjusting the captured image based on one or more of optical distortion, color mapping correction, color of the light illuminated by the scene, and / or stereo image alignment. Other calibration parameters can be used. For example, the calibration parameters can include image sensor characteristics, such as dead pixels or residual pixels, cropped areas, analog gain, digital gain, or other such image sensor characteristics.

[0101] In some embodiments, different calibration parameter sets can improve the performance of the region of interest during a particular routine while possibly reducing the performance of other functional areas. For example, if a particular routine requires very fine close-up images, the calibration parameters of the calibration data set for the routine can be optimized for the sharpness of the image processed using the calibration data. This optimization for the sharpness of the image may be at the expense of color accuracy or fidelity. Similarly, the functionality of a less capable image capture device can be expanded by a calibration data set that improves the performance of a less capable image capture device in a particular area. The use of a calibration data set with a specific performance enhancement can be based on the requirements for a particular routine, the preferences of an end user (e.g., a surgeon), or any other basis.

[0102] The test pattern image set includes images captured by the corresponding image capture device in a calibration environment (such as at a factory calibration of the corresponding image capture device). Each image in the test pattern image set can capture a different test pattern and / or a different lighting condition. In some embodiments, a subset of the test pattern images can be captured to calibrate the image capture device for a specific routine, specific lighting condition, or specific controller system configuration.

[0103] The central calibration server 602 also includes or is in communication with a calibration processor. The calibration processor is configured to generate a calibration data set from one or more images in the test pattern image set. The calibration processor can dynamically generate the calibration data set in response to a request for calibration data from a controller system, such as a request from the electronics cart 56 or the processor 58 in one of the teleoperated surgical systems 604, 606.

[0104] Alternatively or additionally, the calibration processor may generate the calibration data set in response to a change in calibration conditions. For example, the change in calibration conditions may be based on a change in system configuration of one of the teleoperated surgical systems 604, 606 or a change in a transform used to generate the calibration data.

[0105] A change in system configuration may include a change in one or more of an image processor, a vision system or other hardware component, a software release, and / or a light source in a controller system. A change in system configuration may be caused by an upgrade or maintenance of one of the teleoperated surgical systems 604, 606. A change in system configuration may also be caused by connecting an image capture device to a different controller system.

[0106] For example, an image capture device may initially be connected to a first electronics cart 56 in a first system configuration and provide a first calibration data set for the first system configuration. At a later time, the image capture device may be connected to a second electronics cart 56 in a second system configuration that is different from the first system configuration. The second system configuration may have a different image processor, vision system or other hardware components, a different software release, and / or a different light source than the first system configuration. Thus, the calibration processor may generate a second calibration data set for the image capture device that is optimized for the second system configuration.

[0107] As part of processing a request for calibration data from a controller system, the calibration processor can identify a change in the system configuration of the controller system. For example, the request from the controller system can include information about the current system configuration of the controller system. Alternatively, a notification can be sent from the controller system to the central calibration server 602 after a change in the system configuration of the controller system is implemented.

[0108] Other calibration events may also trigger the calibration processor to generate a calibration data set. For example, the central calibration server 602 may receive a calibration event when a new image capture device is ordered, delivered, or received at a healthcare provider location of one of the teleoperated surgical systems 604, 606. For example, after a new image capture device is ordered, delivered, or received at a first healthcare provider location of the teleoperated surgical system 604, the central calibration server receives a calibration notification for the new image capture device. The calibration processor generates a calibration data set for the new image capture device for use with the electronics cart 56 and / or processor 58. The central calibration server 602 may communicate or otherwise push the newly generated calibration data set to the electronics cart 56 and / or processor 58 in the first teleoperated surgical system 604.

[0109] Figure 7 700 is a flow chart illustrating an example process 700 for obtaining calibration data for an image capture device. The process 700 is performed by a controller system such as the electronics cart 56 and / or the processor 58. Other components or systems of the teleoperated surgical system may perform the process 700. For simplicity, the process 700 is described below as being performed by the electronics cart 56, but it does not limit the spirit or scope of the present disclosure.

[0110] At 702, the electronics cart 56 obtains a unique identifier of the image capture device. For example, after the connector 502 of the endoscopic image capture device 500 is plugged into the receptacle 27 of the electronics cart 56, the electronics cart 56 obtains the unique identifier of the endoscopic image capture device 500. As described above, the unique identifier may be obtained by reading a memory, a wireless tag, or an optical label by the electronics cart 56 or an external reader in communication with the electronics cart 56.

[0111] At 704, the electronics cart 56 checks the local calibration cache for calibration data for the image capture device against the obtained unique identifier. At 706, if a local copy of the calibration data for the image capture device is maintained in the cache, the electronics cart 56 sends a request to the central calibration server 602 to verify that the cached version of the calibration data is up to date.

[0112] For example, the electronics cart 56 may request a hash of a current version of the calibration data from the central calibration server 602. The electronics cart 56 may also determine or read a hash of a local copy of the calibration data to compare with the hash received from the central calibration server 602. Alternatively, the electronics cart 56 may send the hash of the local copy of the calibration data to the central calibration server 602 and receive the results of the comparison performed by the central calibration server 602.

[0113] At 708 , the electronics cart 56 determines whether the local copy of the calibration data is up to date, for example, based on the results of a comparison of the hash of the local copy with the hash received from the central calibration server 602 , or based on the results of a comparison performed by the central calibration server 602 .

[0114] At 710, if the electronics cart 56 determines that the local copy of the calibration data is up to date, the local copy of the calibration data is used to process the image received from the image capture device.

[0115] Otherwise, at 712, the electronics cart 56 sends a request for calibration data to the central calibration server 602 based on the obtained unique identifier of the image capture device. Likewise, if it is determined at 704 that no local copy of the calibration data exists in the cache, the electronics cart 56 sends a request for calibration data to the central calibration server 602 based on the obtained unique identifier of the image capture device.

[0116] At 714, the electronics cart 56 receives the requested calibration data. At 716, the electronics cart 56 processes the image received from the image capture device using the received calibration data. At 718, the received calibration data is stored in a local calibration cache and indexed for the unique identifier of the image capture device.

[0117] One or more features of process 700 may be optional or performed in a different order. For example, if the electronics cart 56 does not have a local calibration cache, the features performed at 704-710 may be omitted. As another example, the features performed at 718 may be performed before the features performed at 716. The present disclosure contemplates other additions or variations of process 700.

[0118] Figure 8 8 is a flow chart illustrating an example process 800 for sending calibration data in response to a request. The process 800 is performed by the central calibration server 602 and / or one or more processors in communication with the central calibration server 602. For simplicity, the process 800 is described below as being performed by the central calibration server 602, but it does not limit the spirit or scope of the present disclosure.

[0119] At 802, the central calibration server 602 receives a request for calibration data from a controller system, such as the electronics cart 56 and / or the processor 58. The request includes a unique identifier of the image capture device. The request may additionally include information about the controller system, a configuration of the controller system, or a routine to be performed with the controller system.

[0120] At 804, using the received unique identifier, the central calibration server 602 retrieves a calibration data set for the image capture device from a database on the central calibration server 602 or in communication with the central calibration server 602. For example, the central calibration server 602 performs a lookup operation on one or more calibration data tables indexed for the unique identifier. Additional data received from the request or otherwise obtained by the central calibration server 602 can be used to retrieve a particular calibration data set from the plurality of calibration data sets. For example, upon receiving an indication that the image capture device is to be used for a particular routine, the central calibration server 602 can retrieve a calibration data set optimized for execution of the particular routine.

[0121] At 806, the central calibration server 602 determines whether there are any error conditions associated with the image capture device. For example, the central calibration server 602 can impose policies to prevent the use of the controller system with unregistered image capture devices, limit the number of times an image capture device can be used before maintenance or refurbishment is required, or lock out image capture devices that are subject to recall.

[0122] At 808, if an error condition exists, the central calibration server 602 replies to the received request for calibration data with an error message. For example, the error message may indicate that the image capture device is incompatible with the controller system, that the image capture device needs to be serviced or refurbished, or that the image capture device is subject to a recall. Other error conditions and error messages are contemplated by this disclosure.

[0123] At 810 , if the central calibration server 602 determines that no error condition exists for the image capture device, a reply is sent to the controller system with the retrieved calibration data set.

[0124] One or more features of process 800 may be optional or performed in a different order. Other additions or variations of process 800 are contemplated by this disclosure.

[0125] Fig. 9 9 is a flow chart illustrating an example process 900 for determining a changed calibration parameter or maintenance condition based on a live image captured by an image capture device. In use, a controller system (such as an electronic device cart 56 and / or processor 58) may periodically provide live images captured by an image capture device to a central calibration server 602 during a routine. For example, the controller system may provide at least one image captured by an image capture device for each lighting condition used during a routine. Other numbers of images and frequencies of providing images to the central calibration server 602 may be used.

[0126] At 902, the central calibration server 602 receives a live image from the controller system. The live image may be received along with a unique identifier of the image capture device that captured the live image. Additional information about the configuration of the controller system, current lighting conditions, or other contextual information for conditions used to capture the live image may also be provided.

[0127] At 904, the central calibration server 602 processes the received live images to evaluate one or more calibration parameters of the image capture device. For example, multiple received live images can be processed to identify changes over time. For example, due to wear of the fiber bundle in the flexible cable 506, the processing of the live images can reveal the spectral attenuation of the captured image over time. Similarly, the processing of the live images can reveal the shift in the color space of one or more image sensors in the image capture device over time. In addition, the processing of the live images can reveal alignment problems in the stereoscopic image.

[0128] At 906, the central calibration server 602 determines whether one or more calibration parameters of the calibration data should be changed based on the processing of the live image. If so, at 908, the central calibration server 602 determines the updated calibration parameters and sends them to the controller system. For example, the updated calibration parameters can adjust the brightness, color space, alignment or other characteristics of the image processing through the updated calibration data. The central calibration server 602 may also send one or more updated operating parameters for the controller system or the image capture device. For example, in response to detecting spectral attenuation in the live image, the central calibration server may send operating parameters to increase the power of the light source provided by the controller system or increase the gain of the image sensor of the image capture device. Other changes in operating parameters may be used.

[0129] At 910, the central calibration server 602 stores the updated calibration parameters of the calibration data in a calibration record associated with the image capture device. For example, a calibration record associated with the unique identifier on one or more databases maintained by the central calibration server 602 or in communication with the central calibration server 602 is updated.

[0130] If the central calibration server 602 determines at 906 that no calibration parameters are to be changed, then the central calibration server 602 determines if there are any maintenance conditions at 912. For example, upon detecting a stereo alignment problem in the processed live images, the central calibration server may determine that the image capture device requires maintenance. Other maintenance conditions are contemplated by the present disclosure.

[0131] At 914, the central calibration server 602 sends a maintenance message to the controller system. For example, the maintenance message may provide instructions for adjusting one or more operating parameters, sending the image capture device for maintenance or refurbishment, or otherwise providing notification that maintenance is required. Otherwise, if the central calibration server 602 determines that there is no maintenance condition, the process 900 ends at 916.

[0132] One or more features of process 900 may be optional or performed in a different order. Other additions or variations of process 900 are contemplated by this disclosure.

[0133] FIG. 10A to FIG. 10D is a chromaticity diagram generated from a processed live image, showing the shift in the center of the color space of the image capture device. Each chromaticity diagram can be generated from a set of live images received and processed by the central calibration server 602.

[0134] For example, Fig. 10A As shown, after processing the first received live image set of the image capture device, the central calibration server 602 generates a first chromaticity diagram 1002. The chromaticity diagram 1002 shows a first color space 1004 determined from the first received live image set and a center 1006 of the first color space. Fig. 10B A second chromaticity diagram 1008 is shown, which the central calibration server 602 generates after processing a second received live image set of the image capture device. The chromaticity diagram 1008 shows a second color space 1010 determined from the second received live image set and a center 1012 of the second color space. Fig. 10C A third chromaticity diagram 1014 is shown, which the central calibration server 602 generates after processing the third received live image set of the image capture device. The chromaticity diagram 1014 shows a third color space 1016 determined from the third received live image set and a center 1018 of the third color space. The first received image set, the second received image set, and the third received image set may be received at different times.

[0135] Fig. 10D A chromaticity diagram 1020 is shown in which the first color space 1004, the second color space 1010, and the third color space 1016 overlap each other, and the loci of the center 1006 of the first color space, the center 1012 of the second color space, and the center 1018 of the third color space are shown. Fig. 10DAs shown, the center 1006 of the first color space, the center 1012 of the second color space, and the center 1018 of the third color space illustrate the shift of the color space of the image capture device over time. As described above, updated calibration parameters can be generated by the central calibration server 602 based on this detected shift in the color space of the image capture device.

[0136] Fig.11 1 is a flow chart illustrating an example process 1100 for conducting a calibration parameter survey. At 1102, the central calibration server 602 sends a first calibration parameter set to a first controller system, such as the electronics cart 56 and / or processor 58 of a first teleoperated surgical system 604. At 1104, the central calibration server 602 receives one or more survey results from the first controller system based on user feedback provided in response to the first calibration parameter set.

[0137] At 1106, the central calibration server 602 sends the second set of calibration parameters to a second controller system, such as the electronics cart 56 and / or processor 58 of a second teleoperated surgical system 606. At 1108, the central calibration server 602 receives one or more findings from the second controller system based on user feedback provided in response to the second set of calibration parameters.

[0138] At 1110, the central calibration server 602 processes the survey received in response to the first calibration parameter set and the second calibration parameter set, and determines the optimized calibration parameter set based on the received survey. For example, the central calibration server 602 can select the calibration parameters with the highest evaluation from the first calibration parameter set and the second calibration parameter set to determine the optimized calibration parameter set. Other methods of processing survey results to determine the optimized calibration parameter set can be used.

[0139] One or more features of process 1100 may be optional or performed in a different order. For example, features performed at 1106-1108 may be performed before features performed at 1102-1104. Other additions or variations of process 1100 are contemplated by this disclosure.

[0140] It should be understood that the logical operations described herein with respect to the various figures may be implemented as (1) on a computing device (e.g., Fig.12The logical operations discussed herein are referred to as a sequence of computer-implemented actions or program modules (i.e., software) running on a computing device (i.e., a computing device described in the drawings), (2) interconnected machine logic circuits or circuit modules (i.e., hardware) within a computing device, and / or (3) a combination of software and hardware of a computing device. Therefore, the logical operations discussed herein are not limited to any particular combination of hardware and software. The implementation is a matter of choice depending on the performance and other requirements of the computing device. Therefore, the logical operations described herein are variously referred to as operations, structural devices, actions, or modules. These operations, structural devices, actions, and modules may be implemented in software, firmware, dedicated digital logic, and any combination thereof. It should also be understood that more or fewer operations than shown in the figures and described herein may be performed. These operations may also be performed in an order different from the order described herein.

[0141] refer to Fig.12 , an example computing device 1200 is shown on which embodiments of the present invention may be implemented. For example, each computer processor, computer processor 58, and central calibration server 602 described herein as being located on the electronics cart 56 or the electronics cart 24 may each be implemented as a computing device, such as computing device 1200. It should be understood that the example computing device 1200 is only one example of a suitable computing environment on which embodiments of the present invention may be implemented. Alternatively, computing device 1200 may be a well-known computing system, including but not limited to a personal computer, a server, a handheld or laptop device, a multiprocessor system, a microprocessor-based system, a network personal computer (PC), a minicomputer, a mainframe computer, an embedded system, and / or a distributed computing environment (which includes any multiple of the above systems or devices). A distributed computing environment enables remote computing devices connected to a communication network or other data transmission medium to perform various tasks. In a distributed computing environment, program modules, application programs, and other data may be stored on local and / or remote computer storage media.

[0142] In an embodiment, the computing device 1200 may include two or more computers that communicate with each other to perform tasks collaboratively. For example, but not by way of limitation, the application may be partitioned in a manner that allows concurrent and / or parallel processing of instructions of the application. Alternatively, the data processed by the application may be partitioned in a manner that allows concurrent and / or parallel processing of different parts of the data set by two or more computers. In an embodiment, the computing device 1200 may employ virtualization software to provide the functionality of multiple servers that are not directly bound to multiple computers in the computing device 1200. For example, the virtualization software may provide twenty virtual servers on four physical computers. In an embodiment, the functionality disclosed above may be provided by executing an application and / or multiple applications in a cloud computing environment. Cloud computing may include providing computing services via a network connection using dynamically scalable computing resources. Cloud computing may be supported at least in part by virtualization software. A cloud computing environment may be established by an enterprise and / or may be hired from a third-party provider as needed. Some cloud computing environments may include cloud computing resources owned and operated by an enterprise, as well as cloud computing resources hired and / or rented from a third-party provider.

[0143] In its most basic configuration, computing device 1200 typically includes at least one processing unit 1220 and system memory 1230. Depending on the exact configuration and type of computing device, system memory 1230 may be volatile (such as random access memory (RAM)), non-volatile (such as read-only memory (ROM), flash memory, etc.), or some combination of the two. Fig.12 12 is shown by dashed line 1210. Processing unit 1220 may be a standard programmable processor that performs arithmetic and logic operations necessary for the operation of computing device 1200. Although only one processing unit 1220 is shown, multiple processors may be present. Thus, although instructions may be discussed as being executed by a processor, instructions may be executed simultaneously, serially, or otherwise by one or more processors. Computing device 1200 may also include a bus or other communication mechanism for transmitting information between the various components of computing device 1200.

[0144] The computing device 1200 may have additional features / functionality. For example, the computing device 1200 may include additional storage devices, such as removable storage devices 1240 and non-removable storage devices 1250, including but not limited to disks, or optical disks, or tapes. The computing device 1200 may also include (one or more) network connections 1280, which allow the device to communicate with other devices, such as communicating through the communication paths described herein. (One or more) network connections 1280 can take the form of a modem, a modem bank, an Ethernet card, a universal serial bus (USB) interface card, a serial interface, a token ring card, a fiber distributed data interface (FDDI) card, a wireless local area network (WLAN) card, a radio transceiver card (such as code division multiple access (CDMA)), a global system for mobile communications (GSM), a long term evolution (LTE), a world wide microwave access interoperability (WiMAX) and / or other air interface protocol radio transceiver cards and other well-known network devices. The computing device 1200 may also have (one or more) input devices 1270, such as a keyboard, keypad, switch, dial, mouse, trackball, touch screen, voice recognizer, card reader, paper tape reader or other well-known input devices. It may also include (one or more) output devices 1260, such as a printer, video monitor, liquid crystal display (LCD), touch screen display, display, speaker, etc. Additional devices can be connected to the bus to facilitate data communication between the components of the computing device 1200. All of these devices are well known in the art and need not be discussed in detail here.

[0145] Processing unit 1220 may be configured to execute program code encoded in a tangible computer-readable medium. A tangible computer-readable medium refers to any medium that can provide data that causes computing device 1200 (i.e., a machine) to operate in a particular manner. Various computer-readable media may be utilized to provide instructions to processing unit 1220 for execution. Example tangible computer-readable media may include, but are not limited to, volatile media, non-volatile media, removable media, and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). System memory 1230, removable storage 1240, and non-removable storage 1250 are all examples of tangible computer storage media. Example tangible computer-readable recording media include, but are not limited to, integrated circuits (e.g., field programmable gate arrays or application specific ICs), hard disks, optical disks, magneto-optical disks, floppy disks, magnetic tapes, holographic storage media, solid-state devices, RAM, ROM, electrically erasable program read-only memory (EEPRQM), flash memory or other memory technology, CD-ROMs, digital versatile disks (DVDs) or other optical storage devices, cassettes, magnetic tapes, disk storage devices, or other magnetic storage devices.

[0146] It is of vital importance to the fields of electrical engineering and software engineering that functions that can be implemented by loading executable software into a computer can be turned into a hardware implementation through well-known design rules. The decision between implementing a concept in software or in hardware usually depends on the stability of the design and the number of units to be produced, rather than on any issues involved in converting from the software domain to the hardware domain. Generally, designs that are still frequently changed may be preferably implemented in software, because it is more expensive to remake the hardware implementation than to remake the software design. Generally, stable designs that are produced in large quantities may be preferably implemented in hardware, such as in an application-specific integrated circuit (ASIC), because for large-volume production runs, hardware implementations can be cheaper than software implementations. Generally, a design can be developed and tested in software form and then turned into an equivalent hardware implementation in an application-specific integrated circuit through well-known design rules, which hardwires the instructions of the software. In the same way that a machine controlled by a new ASIC is a specific machine or device, a computer that has been programmed and / or loaded with executable instructions can also be considered a specific machine or device.

[0147] In an example embodiment, processing unit 1220 may execute program code stored in system memory 1230. For example, a bus may carry data to system memory 1230, from which processing unit 1220 receives and executes instructions. Data received by system memory 1230 may optionally be stored on removable storage device 1240 or non-removable storage device 1250 before or after execution by processing unit 1220.

[0148] It should be understood that the various techniques described herein can be implemented in combination with hardware or software or in combination with their combination in appropriate circumstances. Therefore, the method and apparatus of the currently disclosed subject matter, or some aspects or parts thereof, can take the form of program code (i.e., instruction) embodied in tangible media (such as floppy disk, CD-ROM, hard disk drive or hard disk drive or any other machine-readable storage medium), wherein when the program code is loaded into a machine such as a computing device and executed by it, the machine becomes a device for practicing the disclosed subject matter. In the case of executing program code on a programmable computer, the computing device generally includes a processor, a storage medium (including volatile and non-volatile memory and / or storage element) readable by the processor, at least one input device, and at least one output device. One or more programs can be implemented or utilized in combination with the process described in the currently disclosed subject matter, for example, by using an application programming interface (API), reusable controls, etc. Such a program can be implemented in a high-level program or an object-oriented programming language to communicate with a computer system. However, (one or more) programs can be implemented in an assembly or machine language as desired. In any case, the language can be a compiled or interpreted language, and can be combined with a hardware implementation.

[0149] Embodiments of methods and systems may be described herein with reference to block diagrams and flow charts of methods, systems, devices, and computer program products. It should be understood that each frame of the block diagram and flow chart and the combination of frames in the block diagram and flow chart can be implemented by computer program instructions, respectively. These computer program instructions can be loaded onto a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed on the computer or other programmable data processing device create a component for implementing the functions specified in one or more flow chart frames.

[0150] These computer program instructions may also be stored in a computer-readable memory, which may direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including computer-readable instructions for implementing the functions specified in one or more flowchart blocks. The computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions are executed on the computer or other programmable device to provide steps for implementing the functions specified in one or more flowchart blocks.

[0151] Therefore, the blocks of the block diagrams and flow charts support the combination of components for performing the specified functions, the combination of steps for performing the specified functions, and the program instruction components for performing the specified functions. It should also be understood that each block of the block diagrams and flow charts and the combination of the individual blocks of the block diagrams and flow charts can be implemented by a computer system based on dedicated hardware or a combination of dedicated hardware and computer instructions that performs the specified functions or steps.

[0152] Although several embodiments have been provided in the present disclosure, it should be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples are considered to be illustrative rather than restrictive, and the invention is not intended to be limited to the details given herein. For example, various elements or components may be combined or integrated in another system, or certain features may be omitted or not implemented.

[0153] In addition, without departing from the scope of the present disclosure, the techniques, systems, subsystems and methods described and illustrated in a discrete or separate manner in various embodiments may be combined or integrated with other systems, modules, techniques or methods. Other items shown or discussed as being directly coupled or communicating with each other may be indirectly coupled or communicated electrically, mechanically or otherwise through some interface, device or intermediate component. Other examples of changes, substitutions and alterations may be determined by those skilled in the art and may be made without departing from the spirit and scope disclosed herein.

Claims

1. A system for processing an image, comprising: a network accessible database configured to store calibration data for the image capture device; a controller system comprising: an image processor configured to receive an image captured by the image capture device, the image processor further configured to retrieve the calibration data from the network accessible database via a network connection, and the image processor further configured to process the received image based on the retrieved calibration data; as well as a calibration processor in communication with the network accessible database, the calibration processor being configured to: (i) receive calibration images captured by the image capture device, and (ii) generate the calibration data in response to a change in the configuration of the controller system, wherein the calibration data is generated based on applying a calibration algorithm to one or more of the calibration images, wherein the calibration algorithm is selected based on: a type of medical routine in which the image capture device is to be used, the configuration of the controller system, and different calibration methods for calculating one or more calibration parameters for the calibration data, wherein the one or more calibration parameters include calibration parameters for adjusting the captured calibration images based on at least one of: optical distortion, color mapping correction, color of light illuminating a scene, or stereo image alignment.

2. The system according to claim 1, further comprising: A tool includes the image capture device and is electrically coupled to the controller system, wherein the tool has a unique identifier.

3. The system of claim 2, wherein the calibration data is indexed with the image capture device in the network accessible database based on the unique identifier. 4 . The system of claim 3 , wherein the unique identifier is maintained by the tool, and wherein the image processor is further configured to obtain the unique identifier from the tool.

5. The system of claim 4, wherein the unique identifier is maintained by the tool on a memory device of the tool, on a wireless tag of the tool, or on a visual indicator on a surface of the tool.

6. The system of claim 5, wherein the image processor further comprises an optical scanner configured to read the visual indicator from the surface of the tool to obtain the unique identifier from the tool.

7. The system of claim 6, wherein the optical scanner is located in a mating receptacle configured to couple the tool to the controller system.

8. The system of claim 6, wherein the optical scanner is a camera or a barcode reader.

9. The system of any one of claims 5 to 8, wherein the visual indicator is printed, adhered, marked, embossed or stamped on the surface of the tool.

10. The system of any one of claims 5 to 8, wherein the visual indicator is an alphanumeric serial number, a linear barcode encoding the unique identifier, a two-dimensional barcode encoding the unique identifier, or an image encoded with the unique identifier.

11. The system of any one of claims 2 to 8, wherein the tool comprises an endoscope and the image capture device comprises a pair of image sensors configured to capture stereoscopic image data of a surgical site during a surgical routine.

12. A system according to any one of claims 1 to 8, wherein the calibration data includes data for processing the received image to correct the received image based on one or more of the following items: the color of light emitted by an illuminator, the illuminator being configured to illuminate the scene captured by the received image; a calibration image captured by the image capture device; optical distortion introduced by the optical system of the image capture device; stereo image calibration; color space mapping; or image sensor characteristics of the image capture device.

13. The system of claim 1, wherein the calibration image is stored with the calibration data in the network accessible database.

14. The system of claim 1, wherein the calibration processor is configured to generate the calibration data after receiving a request for the calibration data from the controller system associated with the change in the configuration.

15. The system of claim 1, wherein the calibration data is different from a previous calibration data set for the image capture device, wherein the previous calibration data set was generated by the calibration processor based on applying a different calibration algorithm to one or more of the calibration images.

16. The system of claim 1, wherein the calibration data is generated to enhance image features that are important for a type of medical routine in which the image capture device is to be used.

17. The system of claim 16, wherein the image characteristic is one or more of focus level, sharpness, color fidelity, or color accuracy.

18. The system of claim 16, wherein the calibration data further adversely affects a second image characteristic.

19. The system of claim 1, wherein the configuration of the controller system comprises one or more of an illumination source provided by the controller system, a type of the image processor, or a software release version of the controller system.

20. The system of claim 1, wherein the calibration processor is further configured to retrieve an operation image captured by the image capture device from the controller system during the medical routine and adjust the calibration data based on the operation image.

21. The system of claim 20, wherein the illuminance value of the operational image is compared to an expected illuminance value determined based on a known distance to an object captured in the operational image and a known power output to an illumination source, wherein the illumination source is illuminating the object captured in the operational image.

22. The system of claim 20, wherein a color space of the operational image is compared to an expected color space of the image capture device to determine whether a center of the color space deviates from a center of the expected color space.

23. The system of claim 20, wherein the stereoscopic alignment of the operational images is evaluated to identify misalignment with corresponding operational images.

24. The system of claim 15, wherein the calibration processor is further configured to receive a user feedback survey evaluating the calibration data in comparison to the previous calibration data set.

25. The system of any one of claims 1 to 8, wherein the controller system further comprises a calibration cache memory configured to store the calibration data retrieved from the network accessible database.

26. The system of claim 25, wherein the controller system is further configured to store calibration data for image capture devices known to have been transported to a location associated with the location of the controller system.

27. The system of claim 25, wherein the controller system is further configured to store calibration data for a predetermined number of previously used image capture devices.

28. The system of claim 25, wherein the controller system is further configured to store default calibration data for a type of the image capture device.

29. The system of claim 25, wherein the controller system is further configured to compare a cached calibration data set for the image capture device with the calibration data stored on the network accessible database prior to retrieving the calibration data from the network accessible database.

30. A method for processing an image, comprising: storing calibration data for the image capture device in a network accessible database; retrieving, by a controller system via a network connection, the calibration data from the network accessible database; receiving, by a calibration processor in communication with the network accessible database, a calibration image captured by the image capture device; generating, by the calibration processor in response to a change in the configuration of the controller system, the calibration data, wherein the calibration data is generated based on applying a calibration algorithm to one or more of the calibration images, wherein the calibration algorithm is selected based on: a type of medical routine in which the image capture device is to be used, the configuration of the controller system, and different calibration methods for calculating one or more calibration parameters for the calibration data, wherein the one or more calibration parameters include calibration parameters for adjusting a captured calibration image based on at least one of: optical distortion, color mapping correction, color of light illuminating a scene, or stereo image alignment; receiving, by the controller system, an image captured by the image capture device; as well as The received image is processed by an image processor of the controller system based on the retrieved calibration data.

31. A surgical system comprising: a network accessible database configured to store calibration data for an endoscopic image capture device; an image processor configured to be communicatively coupled to the endoscopic image capture device to receive image data of a surgical site during a surgical routine, wherein the image processor is configured to retrieve the calibration data from the network accessible database via a network connection, and the image processor is further configured to process the received images based on the retrieved calibration data; a calibration processor in communication with the network accessible database, the calibration processor being configured to: (i) receive calibration images captured by the endoscopic image capture device, and (ii) generate the calibration data in response to a change in a configuration of a controller system, wherein the calibration data is generated based on applying a calibration algorithm to one or more of the calibration images, wherein the calibration algorithm is selected based on one or more of: a type of the surgical routine, the configuration of the controller system including the image processor, or different calibration methods for calculating one or more calibration parameters for the calibration data, wherein the one or more calibration parameters include calibration parameters for adjusting the captured calibration images based on at least one of: optical distortion, color mapping correction, color of light illuminating a scene, or stereoscopic image alignment; as well as A user control system is coupled to the image processor and configured to receive the processed image from the image processor, the user control system comprising a display configured to display the processed image.

32. The surgical system of claim 31 , further comprising: A manipulator system includes a manipulator arm configured to be attached to the endoscopic image capture device and configured to physically alter a posture of the endoscopic image capture device during the surgical routine.

33. A surgical system according to claim 32, wherein the user control system further comprises a manipulation user interface, wherein upon initiating the manipulation user interface, the user control system is configured to issue a command to the manipulator system to change the posture of the endoscopic image capture device.

34. A method of processing images of a surgical routine captured by an endoscopic image capture device, the method comprising: receiving, by an image processor, the image captured by the endoscopic image capture device; retrieving, by the image processor, calibration data for the endoscopic image capture device from a network accessible database; receiving, by a calibration processor in communication with the network accessible database, a calibration image captured by the endoscopic image capture device; generating, by the calibration processor in response to a change in the configuration of the controller system, the calibration data, wherein the calibration data is generated based on applying a calibration algorithm to one or more of the calibration images, wherein the calibration algorithm is selected based on: a type of medical routine in which the endoscopic image capture device is to be used, the configuration of the controller system including the image processor, and different calibration methods for calculating one or more calibration parameters for the calibration data, wherein the one or more calibration parameters include calibration parameters for adjusting a captured calibration image based on at least one of: optical distortion, color mapping correction, color of light illuminating a scene, or stereo image alignment; processing, by the image processor, images received from the endoscopic image capture device based on the retrieved calibration data; as well as transmitting the processed image to a user control system by the image processor; as well as On the user controlled system, the processed image is displayed on a display.

Images