Video content search in a medical setting

Abstract

This application relates to video content search in a medical setting. A method for searching for a specific topic of interest within video content enhances traditional image data analysis methods by analyzing simultaneously collected non-image data. One application involves video recordings of medical procedures performed using medical devices. When a user wants to locate a portion of the video recording that displays a medical event of interest, one or more medical device system events that may correspond to the occurrence of the medical event of interest are identified from one or more surgical event logs. Timestamps associated with these system events are used to identify candidate video segments from the surgical video recording. Image data analysis is performed on the candidate video segments to determine whether each candidate video segment contains the medical event of interest.

Description

[0001] This application is a divisional application of Chinese patent application 201680043166.9 entitled "Video Content Search in a Medical Environment", filed on June 8, 2016.

[0002] Copyright Notice This patent document contains a portion of copyrighted material. When this patent document or patent disclosure appears in the Patent and Trademark Office's patent files or records, the copyright holder does not object to any reproduction or copying of this patent document or patent disclosure, but otherwise retains all copyright rights.

[0003] Cross-reference to related applications This patent application claims priority and benefit to U.S. Provisional Patent Application 62 / 173,187, filed June 9, 2015, entitled “COMBINED VIDEO METADATA ANDIMAGE ANALYSIS IN A SURGICAL CONTEXT,” the entire contents of which are incorporated herein by reference. Background Technology 1. Technical Field The inventive aspect generally relates to video content search. Specifically, the inventive aspect relates to searching medical surgical video recordings for certain topics of interest, and retrieving video clips that display topics of interest.

[0005] 2. Technology The goal is to be able to locate a specific topic of interest within a video recording without knowing the exact location of the video recording showing the searched topic. Therefore, it is essential to have video search capabilities that allow users to perform keyword or key phrase topic searches within the video recording.

[0006] Subject searches in video recordings typically involve algorithmic approaches that rely solely on the analysis of image data. See, for example, U.S. Patent 7,623,709, entitled "Method and System for Segmenting Image Data," granted to Gering. Generally, the analysis of image data attempts to identify certain patterns present in the individual pixels of an image (e.g., frames of a video) that are determined to indicate various subjects of interest. Image characteristics used to perform these analyses may include pixel colors, pixel intensities, patterns of lines or contours formed by adjacent pixels in the image, etc. Determining the subject of an image through the analysis of image data is challenging and computationally intensive, a finding consistent with U.S. Patent 7,623,709. Furthermore, the results returned are often inaccurate.

[0007] Given the limitations of using only image data analysis to determine the subject matter of an image, an improved system and method for performing subject searches on video recordings is desirable. On one hand, video recordings are associated with various systemic events occurring in medical systems. Information related to these systemic events provides additional datasets that can indicate certain events occurring in the video recordings. Analysis of this additional dataset can be used to enhance the image data analysis of the video recordings when searching for a specific subject of interest within the recorded video. Such a search method would be preferred if it minimizes the computational intensity of the search and improves search accuracy compared to search methods that rely solely on image data analysis. Summary of the Invention

[0008] The following overview introduces certain aspects of the inventive subject matter to provide a basic understanding. This overview is not a comprehensive overview of the inventive subject matter, and it is not intended to identify key or decisive elements or define the scope of the inventive subject matter. Although this overview contains information relating to various aspects and embodiments of the inventive subject matter, its sole purpose is to present some aspects and embodiments in general form as a prelude to the more detailed description that follows.

[0009] A method for searching recorded videos is disclosed. The method involves receiving a user command to locate one or more video segments from one or more surgical video recordings that display a medical event of interest. Next, one or more system events that may correspond to the occurrence of the medical event of interest are identified from one or more surgical event logs. Using one or more timestamps corresponding to each of the one or more system events, one or more candidate video segments are identified from the one or more surgical video recordings. By performing analysis of image data, a determination is made as to whether each of the one or more candidate video segments is an identified video segment containing the medical event of interest. At least one identified video segment is presented to the user.

[0010] A computing device for performing video search is disclosed. The computing device includes a user interface and a processing unit, the processing unit including one or more processors. The processing unit is operatively coupled to the user interface and configured to receive user commands for locating one or more video segments from one or more surgical video recordings that display a medical event of interest. The processing unit is further configured to identify one or more system events from one or more surgical event logs that may correspond to the occurrence of the medical event of interest. Using one or more timestamps corresponding to each of the one or more system events, the processing unit is configured to identify one or more candidate video segments from the one or more surgical video recordings. The processing unit performs analysis of image data to determine whether each of the one or more candidate video segments is an identified video segment containing the medical event of interest. The processing unit presents at least one identified video segment to a user (e.g., on a display). Attached Figure Description

[0011] Figure 1 This is a floor plan of a minimally invasive remote-controlled surgical system.

[0012] Figure 2 This is a perspective view of the surgeon's console.

[0013] Figure 3 This is a perspective view of an electronics cart.

[0014] Figure 4 This is an illustrated explanation of a remote-controlled surgical system.

[0015] Figure 5 This is a perspective view of the patient's side trolley.

[0016] Figure 6 It is an illustrated explanation of the medical system.

[0017] Figure 7 This is a flowchart of an algorithm used in a medical system.

[0018] Figure 8 This is a diagram illustrating the data structure of the event log in a medical system.

[0019] Figure 9 This is a flowchart of a database maintenance algorithm.

[0020] Figure 10 This is a flowchart of an algorithm used to search video content. Detailed Implementation

[0021] This specification and the accompanying drawings illustrating inventive aspects, embodiments, implementations, or applications should not be considered limiting—the claims define the protected invention. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit of this specification and the claims. In some cases, well-known circuits, structures, or techniques have not been shown or described in detail so as not to obscure the invention. Similar numbers in two or more figures denote the same or similar elements.

[0022] Furthermore, the terminology used in this specification is not intended to limit the invention. For example, spatially related terms such as “below,” “under,” “lower,” “above,” “upper,” “proximal,” and “farthest” can be used to describe the relationship between one element or feature and another element or feature as shown in the figures. In addition to the positions and orientations shown in the figures, these spatially related terms are used to encompass different orientations (i.e., positions) and orientations (i.e., rotational placement) of the device in use or operation. For example, if the device in the figures is flipped, an element described as being “below” or “under” other elements or features will be located “above” or “above” other elements or features. Thus, the exemplary term “below” can encompass both the orientation and orientation of above and below. The device may be oriented in other ways (rotated 90 degrees or in other orientations), and the spatially related descriptive terms used herein can be interpreted accordingly. Similarly, descriptions of movement along or about various axes include various specific device orientations and orientations. Additionally, unless the context otherwise indicates, the singular forms “a,” “an,” and “the” are intended to also include the plural forms. Furthermore, the terms "comprises," "comprising," and "includes" specify the presence of the stated features, steps, operations, elements, and / or components without excluding the presence or addition of one or more other features, steps, operations, elements, components, and / or groups. Components described as coupled may be directly electrically or mechanically coupled, or said components may be indirectly coupled via one or more intermediate components.

[0023] Where practicable, an element described in detail with reference to one embodiment, implementation, or application may be included in other embodiments, implementations, or applications where the element is not specifically shown or described. For example, if an element is described in detail with reference to one embodiment but not with reference to a second embodiment, it may still be claimed that the element is included in the second embodiment. Therefore, to avoid unnecessary repetition in the following description, one or more elements shown and described with respect to one embodiment, implementation, or application may be incorporated into other embodiments, implementations, or aspects unless otherwise specifically described, unless the one or more elements would render the embodiment or implementation inoperable, or unless two or more of the elements provide conflicting functionality.

[0024] The aspects of the invention are described primarily using implementations of the da Vinci® surgical system (specifically, the IS4000 model of the da Vinci® Xi™ HD™ surgical system) commercialized by Intuitive Surgical, Inc. of Sunnyvale, California. However, those skilled in the art will understand that the inventive aspects disclosed herein can be embodied and implemented in various ways, including robotic and (if applicable) non-robotic embodiments and implementations. Implementations on the da Vinci® surgical system (e.g., the IS4000 model da Vinci® Xi™ surgical system and the IS3000 model da Vinci® Si™ surgical system) are merely exemplary and are not intended to limit the scope of the inventive aspects disclosed herein.

[0025] This disclosure describes a medical system including an image capture device for recording medical procedures performed using the medical system. Additionally, the medical system includes a computer processor and a memory device for storing surgical event logs. The surgical event log records the occurrence of specific events on the medical system. The surgical event log further records the time of these events. The medical system can synchronize system events in the surgical event log with corresponding medical surgical video recordings.

[0026] On one hand, a medical user attempts to view a specific segment of a recorded medical video that displays a particular part of a medical procedure. The user inputs one or more keywords via a user interface, which describe the subject shown in the video segment being searched. This input can be made using one of a variety of known input methods, such as via an external keyboard coupled to the medical system, via a touchscreen text input interface of the medical system, or via a voice control interface of the medical system. A computer processor communicating with the medical system uses information stored in the surgical event log to identify one or more candidate video segments (or “video clips”) in the recorded medical video that may contain the subject of interest. This identification is possible because (1) certain types of entries in the surgical event log are known to have a correlation with (2) certain medical events of interest (captured in the video recording of the medical procedure). Next, a computer algorithm is applied to the candidate video segments (but not to any remaining video recordings) to analyze the image data contained in the frames of each identified video clip in order to determine whether any frame displays the subject of interest being searched. If the image data analysis algorithm definitively identifies one or more identified video clips as containing the subject of interest, these video clips are presented to the user.

[0027] Minimally Invasive Remote-Controlled Surgical System Now refer to the accompanying drawings (in which, similar reference numerals denote similar parts throughout multiple views). Figure 1This is a plan view of a minimally invasive remote-controlled surgical system 10, typically used to perform minimally invasive diagnostic or medical procedures on a patient 12 lying on an operating table 14. The system includes a surgeon's console 16 for use by a surgeon 18 during the procedure. One or more assistants 20 may also participate in the procedure. The minimally invasive remote-controlled surgical system 10 further includes a patient-side trolley 22 and an electronics trolley 24. While the surgeon 18 views the surgical site through the surgeon's console 16, the patient-side trolley 22 can manipulate at least one removably coupled surgical instrument 26 through a minimally invasive incision within the patient 12's body. Images of the surgical site can be obtained via an endoscope 28 (such as a stereoscopic endoscope), which can be manipulated by the patient-side trolley 22 to orient the endoscope 28. A computer processor positioned on the electronics trolley 24 can be used to process the images of the surgical site for subsequent display to the surgeon 18 via the surgeon's console 16. The number of surgical instruments 26 used at one time will typically depend on the diagnostic or medical procedure, space constraints within the operating room, and other factors. If it is necessary to change one or more of the surgical instruments 26 used during the operation, the assistant 20 may remove the surgical instrument 26 from the patient-side trolley 22 and replace it with another surgical instrument 26 from the tray 30 in the operating room.

[0028] Figure 2 This is a perspective view of the surgeon's console 16. The surgeon's console 16 includes a left-eye display 32 and a right-eye display 34 for presenting a coordinated stereoscopic view of the surgical site to the surgeon 18, the coordinated stereoscopic view enabling depth perception. The console 16 further includes one or more control input terminals 36. It is mounted on the patient-side trolley 22 (in... Figure 1 (As shown in the diagram) One or more surgical instruments for use move in response to manipulation by the surgeon 18 of the one or more control inputs 36. The control inputs 36 can provide information about the associated surgical instruments 26 (in...) Figure 1 (shown in the figure) The same mechanical degrees of freedom are used to provide the surgeon 18 with a sense of remote presentation, or to provide a sense that the control input 36 is integrated with the housing instrument 26, thereby giving the surgeon a strong sense of direct control over the instrument 26. For this purpose, orientation, force, and tactile feedback sensors (not shown) can be used to transmit orientation, force, and tactile sensations from the surgical instrument 26 back to the surgeon's hand via the control input 36.

[0029] The surgeon's console 16 is typically located in the same room as the patient, allowing the surgeon to directly monitor the surgery, be physically present at the surgical site (if necessary), and speak directly to the patient's assistant rather than via telephone or other communication medium. However, the surgeon may be located in a different room, a completely different building, or in another remote location away from the patient where telemedicine is permitted.

[0030] Figure 3 This is a perspective view of the electronics cart 24. The electronics cart 24 may be coupled to the endoscope 28 and includes a computer processor for processing captured images for subsequent display to the surgeon, such as on a surgeon's console or another suitable display located locally and / or remotely. For example, if a stereoscopic endoscope is used, the computer processor on the electronics cart 24 may process the captured images to present the surgeon with a coordinated stereoscopic image of the surgical site. This coordination may include alignment between relative images and may include adjusting the stereoscopic working distance of the stereoscopic endoscope. As another example, image processing may include compensating for imaging errors, such as optical aberrations, of the image capture device using predetermined camera calibration parameters. The captured images may also be stored on a memory device for later viewing. Optionally, the devices in the electronics cart may be integrated into a surgeon's console or a patient-side cart, or the devices may be distributed in various other locations within the operating room.

[0031] Figure 4 This is an illustrated description of one embodiment of a remote-controlled surgical system 10. The surgeon's console 52 (as shown in the image) Figure 1 The surgeon's console 16 can be used by the surgeon to control the patient-side trolley 54 (e.g., during minimally invasive surgery) Figure 1 The patient-side trolley 22). The patient-side trolley 54 can use imaging devices such as stereoscopic endoscopes to capture images of surgical sites and move them to the electronic device trolley 56 (e.g., ...). Figure 1 The computer processor on the electronic device trolley 24 outputs the captured images. The computer processor typically includes one or more data algorithming boards intended to execute computer-readable code stored in a non-volatile memory device of the computer processor. The computer processor can process the captured images in various ways before any subsequent display. For example, the computer processor can overlay the captured images using a virtual control interface before displaying the combined images to the surgeon via the surgeon's console 52.

[0032] Alternatively or additionally, the captured images may undergo image processing by a computer processor positioned outside the electronics cart 56. In one embodiment, the remote surgical system 50 includes an optional computer processor 58 (as indicated by the dashed line), similar to the computer processor positioned on the electronics cart 56; and the patient-side cart 54 outputs the captured images to the computer processor 58 for image processing, after which they are displayed on the surgeon's console 52. In another embodiment, the captured images first undergo image processing by the computer processor on the electronics cart 56, and then additionally undergo image processing by the computer processor 58, before being displayed on the surgeon's console 52. The remote surgical system 50 may include an optional display 60, as indicated by the dashed line. The display 60 is coupled to both the computer processor positioned on the electronics cart 56 and the computer processor 58, and the captured images processed by these computer processors may be displayed on the display 60 and also on the display of the surgeon's console 52.

[0033] Figure 5 This is a perspective view of a patient-side trolley 500 of a minimally invasive remote-controlled surgical system according to an embodiment of the present invention. The patient-side trolley 500 includes one or more support assemblies 510. Surgical instrument manipulators 512 are mounted at the end of each support assembly 510. Additionally, each support assembly 510 may optionally include one or more non-powered lockable mounting joints for positioning the attached surgical instrument manipulators 512 with reference to the surgical patient. As depicted, the patient-side trolley 500 rests on the floor. In other embodiments, the operable portion of the patient-side trolley may be mounted to a wall, to a ceiling, to an operating table 526 that also supports the patient's body 522, or to other operating room equipment. Further, although the patient-side trolley 500 is shown as including four surgical instrument manipulators 512, more or fewer surgical instrument manipulators 512 may be used.

[0034] Functional minimally invasive remote-controlled surgical systems typically include a vision system component that allows the user of the remote-controlled surgical system to view the surgical site from outside the patient's body 522. The vision system typically includes a camera instrument 528 for capturing video images and one or more video displays for displaying the captured video images. The camera instrument 528 may be similar to... Figure 1Endoscope 28 is shown. In some surgical system configurations, camera instrument 528 includes optics for transferring images from the distal end of camera instrument 528 to outside the patient's body 522 via one or more imaging sensors (e.g., CCD or CMOS sensors). Alternatively, the imaging sensors(s) ...(s))(s)(s)(s)(s)(s))(s)(s)(s)(s)(s)(s))(s)(s)(s)(s)(s)(s))(s)(s)(s)(s)(s)(s)(s))(s)(s)(s)(s

[0035] Reference Figure 5 Surgical instruments 520, used to operate at a surgical site within the patient's body 522, are mounted on each surgical instrument manipulator 512. Each surgical instrument manipulator 512 may be provided in various forms that allow the associated surgical instrument to move with one or more mechanical degrees of freedom (e.g., all six Cartesian degrees of freedom, five or fewer Cartesian degrees of freedom, etc.). Typically, mechanical or control constraints restrict the movement of each manipulator 512 of its associated surgical instrument around a center of motion on the instrument that remains stationary relative to the patient, and this center of motion is typically located at the orientation of the instrument entering the body.

[0036] Surgical instrument 520 is controlled via computer-aided remote control. The functional minimally invasive remote surgical system includes a control input that receives input from a user of the remote surgical system (e.g., a surgeon or other medical personnel). The control input communicates with one or more computer-controlled remote actuators to which the surgical instrument 520 is coupled. In this manner, the surgical instrument 520 moves in response to movement of the control input by the medical personnel. In one embodiment, one or more control inputs are included in a surgeon's console (e.g., in...) Figure 2 (Surgical console 16 shown). The surgeon can manipulate the control input 36 of the surgical console 16 to operate the remote actuator of the patient-side trolley 500. The force generated by the remote actuator is transferred via a transmission system mechanism that transmits the force from the remote actuator to the surgical instrument 520.

[0037] Reference Figure 5Surgical instrument 520 and cannula 524 are removably coupled to manipulator 512, wherein surgical instrument 520 is inserted through cannula 524. One or more remote actuators of manipulator 512 move surgical instrument 512 as a whole. Manipulator 512 further includes instrument holder 530. Surgical instrument 520 is detachably connected to instrument holder 530. Instrument holder 530 accommodates one or more remote actuators that provide multiple controller movements, which surgical instrument 520 translates into various movements of an end effector on surgical instrument 520. Thus, the remote actuators in instrument holder 530 move only one or more components of surgical instrument 520, rather than moving the instrument as a whole. Inputs that control the instrument as a whole or control components of the instrument are such that inputs provided by a surgeon or other medical personnel to a control input (“master” command) are translated by the surgical instrument into corresponding actions (e.g., “slave” responses).

[0038] Figure 6 A schematic diagram of a medical system 600 is shown. The medical system 600 includes a remote surgical system 10. As indicated by dashed lines, the medical system 600 may optionally include a computer 620 communicating with the remote surgical system 10. In one embodiment, the computer 620 is a personal computer. In an alternative embodiment, the computer 620 is a server or mainframe. Data communication between the computer 620 and the remote surgical system 10 can be performed via various methods known in the art, such as a local area network (LAN) connection, an internet connection (wired or wireless), or similar data communication means. Additionally or alternatively, such data communication can occur by means of data transfer from the remote surgical system 10 to the computer 620 using a portable hard disk drive or other similar data storage device. The computer 620 can be used to access information stored on a memory device of the remote surgical system 10, and the remote surgical system 10 can be used to access information stored on a memory device of the computer 620. A keyboard of the computer 620 can provide a user interface for text input. Text entered using a keyboard can be processed by the computer processor of computer 620 and the computer processor of remote surgical system 10.

[0039] The medical system 600 includes a memory device on which one or more surgical event logs are maintained. Figure 8A selected portion of an exemplary surgical event log is shown. In one embodiment, the storage device is a storage device 59 positioned on an electronics cart 56. In an alternative embodiment, the surgical event log is maintained on a storage device positioned on a surgeon's console 52. In yet another embodiment, the surgical event log is maintained on a storage device positioned on a computer 620.

[0040] Entries in the surgical event log may include records of various interactions between the remote surgical system 10 and the medical personnel using the remote surgical system 10. In one embodiment, the surgical event log is automatically generated and automatically maintained in parallel with other operations performed by the remote surgical system 10 (e.g., computer-assisted remote control, video recording, video display, etc.). Input by medical personnel is not required for the generation and maintenance of the surgical event log of the medical system 600.

[0041] Figure 7 Here is a flowchart of the algorithm, which: (1) uses surgical event log data to identify one or more candidate video segments (or “video clips”) in the captured video that may have occurred during the captured video period for certain medical events of interest sought by the device user; and (2) applies an algorithm that examines the image data contained in the frames of each candidate video clip to the candidate video clip (but not to any remaining video recording) in order to determine whether any frame shows the medical event of interest sought by the device user.

[0042] In one embodiment, algorithm 703 is generated by medical system 600 (in... Figure 6 The processor 58 (shown at the point of illustration) implements the algorithm. In an alternative embodiment, one or more algorithms in algorithm 703 may be implemented by a processor other than processor 58.

[0043] Capture, process, display, and store medical videos The flow of an image capture and recording algorithm for a medical system is disclosed. In one embodiment, the medical system includes a remotely controlled surgical system. Video images are captured by an image capture device. In one embodiment, the image capture device is an endoscope including an array of active pixel sensors (APS) (e.g., Figure 1The endoscope 28 is located at the endoscope. The APS array consists of red, green, and blue pixel sensors arranged in a periodic pattern. Each pixel sensor in the APS array includes a photodetector and a filter that limits the light frequencies exposed to the photodetector. The filter of each red pixel sensor allows only light in the red visible portion of the electromagnetic (EM) radiation spectrum to pass through. Similarly, the filter of each green pixel allows only light in the green visible portion of the EM spectrum to pass through, and the filter of each blue pixel allows only light in the blue visible portion of the EM spectrum to pass through. Accordingly, each pixel sensor in the APS array will respond only to light within the EM spectral range of its respective filter. All other light frequencies reaching the pixel sensor are filtered out by its filter and do not reach its photodetector. Accordingly, each pixel sensor in the APS array will generally not respond to light outside the EM spectral range of its filter. The red pixel sensor will not respond to non-red light because the filter of the red pixel sensor prevents all non-red light from reaching the photodetector of the red pixel sensor. The green pixel sensor will not respond to non-green light because its filter prevents all non-green light from reaching its photodetector. Similarly, the blue pixel sensor will not respond to non-blue light because its filter prevents all non-blue light from reaching its photodetector.

[0044] In one embodiment, the APS array described above is configured to generate a raw Bayer pattern image. Typically, each pixel of the raw Bayer pattern image directly corresponds one-to-one with an individual pixel sensor in the APS array. As discussed previously, the APS array consists of red, green, and blue pixel sensors arranged in a periodic pattern. When the periodic arrangement of the various color pixel sensors in the APS array is known, based on this color information, it can be determined what color each pixel of the raw Bayer pattern image should be. If a pixel of the raw Bayer pattern image corresponds to a red pixel sensor in the APS array, the pixel should be red. If a pixel of the raw Bayer pattern image corresponds to a green pixel sensor in the APS array, the pixel should be green. If a pixel of the raw Bayer pattern image corresponds to a blue pixel sensor in the APS array, the pixel should be blue.

[0045] Pixel intensity is another piece of information required to generate the original Bayer pattern image. This information is provided by a signal generated by the photosensitive sensor of each pixel sensor in the APS array. Consider, for example, a pixel of the original Bayer pattern image corresponding to a red pixel sensor in the APS array. As discussed earlier, the ability to determine the association of a pixel with a red pixel sensor is based on prior knowledge of the periodic arrangement of the pixel sensors of various colors in the APS array. The question then becomes whether this pixel in the original Bayer pattern image should be: (1) pale red; or (2) saturated deep red. Whether this pixel is pale red or saturated deep red depends on the amount of light reaching the photosensitive sensor of the corresponding red pixel sensor in the APS array. As discussed earlier, the filter of the red pixel sensor blocks all non-red light and only allows red light to pass through to the photosensitive sensor. Therefore, whether this pixel is pale red or saturated deep red depends on the amount of red light exposed to the red pixel sensor (or more precisely, its photosensitive sensor). The more light reaches the photosensitive sensor of the pixel sensor, the more intense the color of its corresponding pixel in the original Bayer pattern image. These principles apply similarly to the green and blue pixel sensors in the APS array.

[0046] The raw Bayer pattern image data undergoes video processing to convert (1) a raw Bayer pattern image, such as one captured by an endoscope, into (2) a red-green-blue (RGB) image suitable for display. In one embodiment, this video processing involves a demosaicing module. Generally, RGB images are based on an additive color model. The color of each pixel in an RGB image is determined by adding red, green, and blue in appropriate intensities together. Starting with the raw Bayer pattern image, the demosaicing module creates an RGB image by adjusting the color of each red, green, or blue pixel in the raw Bayer pattern image using information about pixels of different colors surrounding each red, green, or blue pixel. For example, red pixels in the raw Bayer pattern image are adjusted by adding color data from (1) one or more adjacent blue pixels and (2) one or more adjacent green pixels to the red pixels in the raw Bayer pattern image. Using this method, the red pixels of the raw Bayer pattern image are converted into RGB image pixels with red, green, and blue components. These principles are similarly applied to converting the green and blue pixels of the raw Bayer pattern image into RGB image pixels.

[0047] One goal of the image processing algorithm in the desmosaic processing module is to generate an RGB image by adjusting each pixel individually, where each pixel of the RGB image is an accurate approximation of the ratio of red, green, and blue light exposed to the corresponding pixel sensor in the APS array (i.e., the endoscope). As discussed earlier, each pixel sensor in the APS array includes a filter that allows only red light to pass through when the pixel sensor is a red pixel sensor, only green light when the pixel sensor is a green pixel sensor, or only blue light when the pixel sensor is a blue pixel sensor. Non-red light reaching the red pixel sensor is filtered out by the red pixel sensor's filter and never reaches the red pixel sensor's photodetector. The same applies to non-green light reaching the green pixel sensor and non-blue light reaching the blue pixel sensor. Therefore, the information detected by the photodetector of any single pixel sensor (whether red, green, or blue) corresponds only to the intensity of a single color of light (i.e., the color of light allowed to pass through a particular filter).

[0048] An APS array consists of red, green, and blue pixel sensors arranged in a periodic pattern. Typically, pixel sensors of different colors are not arranged adjacent to each other. For example, in one embodiment, each red pixel sensor is flanked by one or more green pixel sensors and one or more blue pixel sensors. One reason for this type of pixel sensor arrangement is to allow the image processing algorithms of the demosaicing module to use information from neighboring pixel sensors to approximate the color component of the light arriving at a particular pixel sensor. In one example, a green pixel sensor is positioned to the left and right of a red pixel sensor, with a blue pixel sensor above and below it. The red pixel sensor includes a filter that allows only red light to reach the photodetector. All non-red light (e.g., green, blue, etc.) reaching the pixel sensor is filtered out and never reaches the photodetector. Therefore, the signal from the photodetector of the red pixel sensor indicates the intensity of the red light exposed to the red pixel sensor. However, the photodetector of the red pixel sensor does not provide any information about the intensity of non-red light (e.g., green or blue light) reaching the red pixel sensor, because all non-red light is filtered out by the red pixel sensor's filter and never reaches the photodetector. However, in this example, two green pixel sensors and two blue pixel sensors surround the red pixel sensor. The photodetector of each of the two green pixel sensors provides information about the intensity of green light exposed to each green pixel sensor. The photodetector of each of the two blue pixel sensors provides information about the intensity of blue light exposed to each blue pixel sensor. This information from adjacent green and blue pixel sensors can be used to approximate the intensity of green and blue light exposed to the red pixel sensor. This approximate estimate of the green and blue light exposed to the red pixel sensor, along with information about the intensity of red light provided by the red pixel sensor's photodetector, is used to generate the RGB image pixels corresponding to the red pixel sensor. The colors of the RGB image pixels corresponding to the green and blue pixel sensors can be determined in a similar manner.

[0049] After video processing, the data is sent to a remote surgical system (e.g., in...). Figure 1 One or more users of the remote-controlled surgical system 10 shown herein display processed images. (See reference...) Figure 1These images can be displayed to the surgeon 18 seated at the surgeon's console. Alternatively, these images can be displayed to other surgical staff on a monitor located on the electronics cart 24. In one embodiment, the processed images constitute SXGA video output at 60 frames per second (fps). The image data of the processed images is transmitted to various displays via a DVI-D connection. In one embodiment, the processed images are additionally stored in a memory device. The processed images can be stored in their compressed and / or uncompressed formats. In one embodiment, the processed images are stored on a memory device 59 located on the electronics cart 56 (in... Figure 6 (as shown in the illustration). In alternative embodiments, the processed image is stored on a memory device located on the surgeon's console 16 or the patient-side trolley 22, or the storage device is a removable memory device (i.e., a "thumb drive") that communicates data with a computer processor performing the video processing. Alternatively, the generated unprocessed raw Bayer pattern image data may also be stored in such or another storage device.

[0050] In one embodiment, a series of images output by image processing is stored as a medical video. A timestamp of the start time of video recording is embedded in the medical video data. Video recording can be automatically commanded when an image capture device (e.g., an endoscope) is mounted on a remote surgical system for use. In an alternative embodiment, video recording begins when the user initiates it. The computer processor of the remote surgical system, configured to perform content searches on the recorded medical video, can use this timestamp. This aspect will be discussed in more detail below.

[0051] Fill the surgical event log Surgical event logs describe various system events that occur on a medical system (e.g., in...). Figure 8 The example event log shown here (a portion of which is an example) can be used to improve speed and accuracy, and can be used to perform content searches on video recordings of medical surgeries performed using a medical system. That is, clinically significant system events in the surgical event log can be used to identify video segments (i.e., “video clips”) in medical surgical video recordings that may contain content of interest.

[0052] In one embodiment, the medical system includes a remote surgical system. The surgical event log of the remote surgical system is typically controlled by the computer processor 58 of the remote surgical system (in...). Figure 6 (As shown) is continuously maintained and stored in the electronic equipment cart 56 (in Figure 6The memory device 59 (shown in the diagram) is located on the system. In an alternative embodiment, one or both of the computer and the memory device may be located on another component of the remote surgical system. The surgical event log preferably includes entries for system events, which include the following: • Powering on / off the remote surgical system • Install the image capture device on the remote surgical system for use. • Video recording is initiated by the user of the remote surgical system. • Mount surgical instruments on a remote surgical system • Removing surgical instruments from a remote surgical system • Function of actuating electrocautery devices • Adjust the input and output ratio of the remote surgical system. • Activate the special imaging mode of the remote surgical system This list is by no means exhaustive. The event log may include entries corresponding to the occurrence of many other system events that can be detected by the remote surgical system.

[0053] A flow of an algorithm for automatically generating and maintaining surgical event logs for a medical system is disclosed. The algorithm detects the occurrence of system events. If a system event is detected, a corresponding entry is created in the surgical event log maintained on the medical system. The algorithm updates the surgical event log by creating an additional entry for each new system event detected. In one embodiment of the surgical event log, for each entry associated with the occurrence of a clinical system event, the following fields are recorded: (1) a timestamp of the event; (2) a brief description of the event; and (3) a unique code associated only with a specific type of event. In one example, a surgical instrument (e.g., a needle-driven instrument) is installed for use on a remote surgical system (e.g., at 10:00 AM local time), which detects the installation and identifies it as a system event. An associated entry is created and entered into the surgical event log. The brief description field of the event log entry will read “Needle-driven instrument installed”. A unique code (e.g., 300006) associated with each installation of the needle-driven instrument will also be recorded. Finally, the 10:00 AM timestamp of the entry is generated. If the event was previously programmed in local time, the local time of the event can be determined locally on the remote surgical system. Alternatively, if the remote surgical system is connected to the Internet, the local time of the event can be determined by performing an Internet query in which the Internet Protocol (IP) address associated with the remote surgical system can be used to provide location information.

[0054] exist Figure 8The diagram shows a portion of a surgical event with the data structure described above. As discussed earlier, in one embodiment, event logs are automatically generated and maintained without requiring input from, for example, a user of the medical system implementing the algorithm. This generation and maintenance of event logs occurs in parallel with other operations of the medical system. These other operations may include video recording of medical procedures performed using the medical system, as well as video processing and storage of these surgical video recordings.

[0055] Search for captured videos.

[0056] Figure 7 This is a flowchart illustrating algorithm 703 used to search recorded medical surgical videos. Algorithm 703 utilizes entries from the surgical event log. (See also...) Figure 7 At point 735, algorithm 703 receives a command issued by a user of the medical system to search for recorded medical videos on a specific medical topic of interest. In one embodiment, the medical system includes a remote surgical system. The user can issue this command by entering text describing the medical event of interest via a text input interface, speaking a phrase describing the medical event of interest via a voice control user interface or other user interfaces known in the art. In one embodiment, the surgical surgeon (e.g., in...) Figure 1 The surgeon (18) performs this search during surgery, and the search is limited to records of instantaneous medical procedures. In an alternative embodiment, various medical surgical video recordings and their associated event logs are stored in a storage device. The contents of these stored medical surgical video recordings can be searched by medical personnel during or after surgery. In one example, a medical personnel wishing to search the contents of these stored medical surgical video recordings do so via a keyboard (e.g., on the keyboard). Figure 6 The computer 620 (shown here) allows users to input a text describing a medical event of interest to issue a search command.

[0057] At 740, for one or more system events indicating the occurrence of a medical event of interest, algorithm 703 uses the information input at 735 to search one or more surgical event logs containing system events related to immediate medical surgery. In one embodiment, 740 limits the event log search to system events related to emergency medical surgery by searching only system events that occurred after the most recent power-on of the remote surgical system. In an alternative embodiment, 740 searches for system events in an archived set of surgical event logs. Each archived surgical event log is associated with a medical procedure performed using the device, and in one embodiment, a video recording of the associated medical procedure (surgical video recording) is archived with its corresponding surgical event log. In one embodiment, at the time of device manufacturing, the correlation between (1) various medical events of interest and (2) system events in the event logs is pre-programmed on the device implementing algorithm 703. The information on which these algorithmic correlations are based can be derived from the knowledge of experienced medical trainees using the remote surgical system. As more correlations between medical events of interest and system events become known to the device manufacturer, an updated version of algorithm 703 incorporating these correlations can be made available to remote surgical systems in the art using an internet connection. After examining video recordings of medical surgeries and event log data retrieved from examples of remote surgical systems used in the field, device manufacturers can make these additional correlations.

[0058] Based on the above discussion, in one example, at 735, a medical professional (e.g., surgeon 18 using remote surgical system 10) performs a medical procedure using a remote surgical system implementing algorithm 703. Figure 1 (Both are shown here) To learn more about the early stages of a surgery in which the patient experienced significant blood loss, the medical personnel initiated a content search of the video recording of the immediate medical surgery by saying "significant blood loss" to a voice-controlled search interface. The remote surgical system was programmed to associate significant blood loss with the use of electrocautery instruments, which are commonly used for hemostasis. At 740, algorithm 703 searches the event log for entries corresponding to the immediate medical surgery, the descriptions of which include "cauterization" and "installation." (See reference...) Figure 8In one example, the search at 740 generates a first system event 810, a second system event 820, and a third system event 830. These system events indicate three separate instances in which cauterization instruments are mounted on a remote-controlled surgical system. At 745, the timestamp of each search result generated at 740 is retrieved. In this example, the timestamps of the first system event 810, the second system event 820, and the third system event 830 are 10:00 AM, 11:00 AM, and 12:00 PM, respectively. At 750, the time span of interest is determined for each of the retrieved timestamps. In cases of massive blood loss, the remote-controlled surgical system is programmed to identify a 3-minute video segment corresponding to each of the retrieved timestamps as the time span of interest. Each 3-minute video segment begins one minute before each retrieved timestamp and lasts for three minutes. These video segments are identified as candidate video clips. In this example, the three 3-minute video segments of interest (i.e., candidate video segments) are the following segments of the medical video: (1) the segment from 9:59 a.m. to 10:02 a.m.; (2) the segment from 10:59 a.m. to 11:02 a.m.; and (3) the segment from 11:59 a.m. to 12:02 p.m. Different time spans of interest can be associated with different medical events of interest and with different systemic events.

[0059] At position 755, an image search algorithm analyzing the image data is applied to the frames of the video segments of interest (i.e., candidate video segments) identified at position 750. In one embodiment, the first frame of each candidate video segment is located by determining the duration elapsed since the start of video recording. As an example, see [reference]. Figure 8 Video recording is initiated at point 805, and the video recording has a timestamp of 9:00 AM. Point 750 identifies portions of the video recording occurring between 9:59 AM and 10:02 AM as the first candidate video segment. Therefore, the first frame of the first candidate video segment occurs at 59 minutes into the video recording. Similarly, the first frames of the second and third video segments of interest (the second candidate video segment) occur at 1 hour and 59 minutes and 2 hours and 59 minutes into the video recording, respectively. Likewise, the last frames of the first, second, and third video segments of interest (the third candidate video segment) occur at 1 hour and 2 minutes, 2 hours and 2 minutes, and 3 hours and 2 minutes, respectively.

[0060] A discussion of exemplary algorithms for determining image subjects using image data analysis can be found in U.S. Patent 7,623,709, entitled "Method and System for Segmenting ImageData," granted to Gering. Typically, these algorithms search the pixels of an image for certain patterns indicating a subject of interest. In this example, 755 is programmed to search frames of the three candidate video segments identified above for pixel patterns indicating large blood loss. In one embodiment, the image search algorithm precisely identifies large blood loss when the sequence of frames constituting the candidate video segments contains an increasing number of dark red adjacent pixels compared to the immediate preceding frame. In this example, 755 precisely identifies large blood loss as shown in the second candidate video segment (from 10:59 AM to 11:02 AM) compared to... Figure 8 The video segment corresponding to the second system event 820 shown is displayed, and it is determined that the massive blood loss was not shown in the first candidate video segment (from 9:59 a.m. to 10:02 a.m.). Figure 8 The video segment corresponding to the first system event 810 shown at the location (or the third candidate video segment (from 11:59 AM to 12:02 PM) is shown in the video segment corresponding to the first system event 810 shown at the location) or in the third candidate video segment (from 11:59 AM to 12:02 PM). Figure 8 The video segment corresponding to the third system event 830 shown here.

[0061] At location 760, candidate video segments identified at location 755 (i.e., the video segment from 10:59 AM to 11:02 AM) are presented to medical personnel for viewing. In one embodiment, this video segment is displayed automatically. In an alternative embodiment, a link is provided on the display for medical personnel to view (e.g., at a location located at...). Figure 1 (As shown on the monitor on the surgeon's console 16). Links are configured to play back to the medical personnel the portion of the recorded video corresponding to the identified video segment. Similarly, if more than one candidate video segment is identified at 755, more than one link can be displayed. Note that identified video segments and / or one or more links configured to play back identified video segments can be presented to the medical personnel on any of the multiple monitors other than the monitor located on the surgeon's console 16. Additionally or alternatively, this can be done on the monitor connected to computer 620 ( Figure 6 This information is displayed on the associated monitor (as shown in the image).

[0062] In another example, 740 of algorithm 703 includes analysis of the actuation frequency of the ablation function of the electrocautery instrument. This can be performed via a surgeon's console located on a remotely controlled surgical system (e.g., in...). Figure 1The foot-activated pedal on the surgeon's console 52 (shown at 740) actuates the cauterization function. As previously discussed, cauterization instruments are commonly used for hemostasis. Generally, the more severe the bleeding, the more frequently the cauterization function is actuated in an effort to stop the bleeding. In one embodiment, the remote surgical system is programmed to correlate significant blood loss with the frequency of actuation of the cauterization function of the electrocautery instrument. In one example, a medical professional watching a video recording of a medical procedure attempts to learn more about a part of the procedure during which the patient experienced significant blood loss. The medical professional initiates a video search in a manner similar to that discussed above. In this example, each actuation of the cauterization function is a system event, and an entry is created in the surgical event log for each system event. At 740, algorithm 703 searches the surgical event log corresponding to the immediate medical procedure for surgical event log entries whose descriptions include "cauterization" and "application". In one example, the search at 740 yields 30 entries. At 745, the timestamp of each search result in the search results generated by 740 is retrieved. In this example: (1) the timestamp of the first entry is 10:00 AM; (2) the timestamps of entries 2-29 are closely spaced and span the period between 11:00 AM and 11:02 AM; and (3) the timestamp of entry 30 is 12:00 PM. At 750, the timestamp retrieved at 745 is used to determine the time span of interest. In this example, the remote surgical system is programmed to identify 3-minute video segments (i.e., candidate video segments) corresponding to instances where the cauterization function is actuated more than 10 times within a 2-minute time period as the time span of interest. Each candidate video segment begins one minute before the timestamp of the event log entry corresponding to the first cauterization actuation in the cauterization actuation sequence and continues for two minutes after that timestamp. In this example, the frequency of cauterization actuation between 11:00 AM and 11:02 AM is the only instance that satisfies the condition of more than 10 times within a 2-minute test. Therefore, only one candidate video segment is identified: the segment from 10:59 AM to 11:02 AM. 755 and 760 operate in a manner similar to that described previously.

[0063] In another example, a medical professional performing a cholecystectomy using a remote surgical system implementing algorithm 703 (e.g., surgeon 18 using remote surgical system 10, in...) Figure 1(The text shows both examples.) The medical staff wanted to learn more about gallbladder loosening from the duodenal wall during surgery. They initiated a content search of the video recording of the cholecystectomy by saying "gallbladder loosening" to a voice-controlled search interface. The remote surgical system was programmed to associate gallbladder loosening (the medical event of interest) with the use of fluorescence imaging modes (system events recorded in the surgical event log). Fluorescence imaging is frequently used to enhance the visualization of an anatomical structure during the gallbladder loosening portion of a cholecystectomy. Therefore, at 740, algorithm 703 performs a search on a portion of the event log corresponding to the immediate cholecystectomy, searching for event log entries whose descriptions include "fluorescent" and "on". In one example, the search at 740 produces a first entry, a second entry, and a third entry. At 745, the timestamp of each search result generated by 740 is retrieved. In this example, the timestamps of the first, second, and third entries are 10:00 AM, 11:00 AM, and 12:00 PM, respectively. At 750, the time span of interest is determined for each of the retrieved timestamps. In the case of gallbladder loosening, the remote surgical system is programmed to identify 10-minute video segments corresponding to each of the retrieved timestamps as time spans of interest. Each 10-minute video segment begins at the time of the retrieved timestamp and continues for ten minutes after said timestamp. In this example, the three 10-minute video segments of interest (candidate video segments) are the following segments of the medical video: (1) the segment from 10:00 AM to 10:10 AM; (2) the segment from 11:00 AM to 11:10 AM; and (3) the segment from 12:00 PM to 12:10 PM. 755 and 760 operate in a manner similar to that described previously.

[0064] In another example, a medical professional performing mitral valve repair surgery using a remote surgical system implementing algorithm 703 (e.g., surgeon 18 using remote surgical system 10, in...) Figure 1(Both are shown here) To learn more about the reconstruction work performed during mitral valve repair surgery, the medical personnel initiate a content search of the video recording of the mitral valve repair surgery by saying "valve reconstruction" to the voice control search interface. The remote surgical system is programmed to associate valve reconstruction (the medical event of interest) with the use of fine control mode (a system event recorded in the surgical event log). Compared to coarse control mode, fine control mode gives the user of the remote surgical system the ability to command smaller movements of the surgical instruments. As discussed earlier, in one embodiment of the remote surgical system, the surgical instruments coupled to the driven robotic manipulator move in response to the surgeon's movement of two control inputs located on the surgeon's console. In coarse control mode, the surgical instruments may move 5 mm in response to a 1 cm movement of the control inputs by the surgeon. In fine control mode, the same 1 cm movement of the control inputs may produce only a 2.5 mm movement of the surgical instruments. Fine control mode is frequently used by surgeons operating in areas of fine anatomy. Compared to coarse control modes, the finer proportion between (1) the movement of the control input and (2) the movement of the corresponding surgical instruments enables surgeons to perform smaller dissections and smaller sutures.

[0065] Therefore, at 740, algorithm 703 performs a search on a portion of the event log corresponding to the immediate valve repair surgery, searching for event log entries whose descriptions include "fine control mode". In one example, the search at 740 produces a first entry, a second entry, and a third entry. At 745, the timestamp of each search result in the search results generated by 740 is retrieved. In this example, the timestamps of the first, second, and third entries are 10:00 AM, 11:00 AM, and 12:00 PM, respectively. At 750, the time span of interest is determined for each of the retrieved timestamps. In the case of valve reconstruction, the remote surgical system is programmed to identify a 30-minute video segment corresponding to each of the retrieved timestamps as the time span of interest. Each 30-minute video segment begins at the time of the retrieved timestamp and continues for 30 minutes after said timestamp. In this example, the three 30-minute video segments of interest (i.e., candidate video segments) are the following segments of the medical video: (1) the segment from 10:00 AM to 10:30 AM; (2) the segment from 11:00 AM to 11:30 AM; and (3) the segment from 12:00 PM to 12:30 PM. 755 and 760 operate in a manner similar to that described previously.

[0066] For each example in the previous examples, candidate video clips are evaluated to determine, using the image data of each candidate video clip, whether the pixels of the frames that make up each candidate video clip actually indicate the subject of interest that the user expects. Candidate video clips used to make a positive determination are presented to the user for viewing.

[0067] Figure 9 A flowchart of algorithm 900 is shown, which guides the data collection and maintenance of the device's database. In one embodiment, the device is a remote surgical system, and the database includes: (1) one or more video recordings (surgical video recordings) of surgical procedures performed using the surgical system; and (2) a surgical event log for each of the one or more surgical video recordings. The database may be stored on a storage device of the remote surgical system. Alternatively or additionally, the database may be stored in: (1) a storage device located on a remote server that communicates with the remote surgical system; (2) a storage device in a cloud computing system that communicates with the remote surgical system; or (3) any other electronic storage medium option.

[0068] In one embodiment, a new surgical event log is created whenever the surgical system is powered on. In one instance, powering on the surgical system creates a new surgical event log but does not begin recording of video captured by an image capture device associated with the surgical system; user input is required via the surgical system's user interface to begin recording the video captured by the image capture device. In another instance, if an image capture device (e.g., an endoscope) is installed on the surgical system when it is powered on, powering on the surgical system may additionally cause the surgical system to begin recording the video captured by the image capture device. In yet another instance, recording of the video captured by the image capture device is automatically initiated when the image capture device is installed for use on a surgical system that is already powered on. These examples are merely exemplary and are not intended to be limiting.

[0069] like Figure 9 As shown, algorithm 900 starts when a start signal is received at 910. The start signal is the result of user input (e.g., actuating a power on / off switch) received via user interface 905. If no start signal is received, algorithm 900 loops back and repeatedly queries at 910.

[0070] If a start signal is received at 910, algorithm 900 continues to 915, where it queries the system clock 920 to read the current time. At 925, algorithm 900 receives an image from an image capture device 930 that is in data communication with the computer processor executing algorithm 900; the image is a frame of a video recording captured by the image capture device 930. At 935, the image received at 925 is digitally timestamped with the current time read at 915, and the timestamped image is saved as a frame of the video recording to memory device 940.

[0071] Algorithm 900 also updates and maintains the surgical event log. As discussed previously, in one embodiment, the surgical event log is created each time the device executing Algorithm 900 (e.g., a remote surgical system) is powered on. At 945, Algorithm 900 queries the surgical system to determine whether one of a plurality of system events of interest has been detected at that time. Preferably, the surgical system's memory device includes a lookup table containing a list of pre-identified system events to be recorded in the surgical event log. If 945 detects the occurrence of one of the system events included in the lookup table, the identified system event is retrieved and, at 950, digitally timestamped with the current time determined at 915. This system event and its associated timestamp are saved as an entry in the surgical event log to memory device 840. Algorithm 900 then proceeds to 955, which will be discussed in more detail below.

[0072] If 945 determines that none of the system events included in the lookup table have occurred, algorithm 900 proceeds directly to 955, where algorithm 900 queries whether a stop signal has been received. If a stop signal is received, algorithm 900 stops video recording and ceases updating the surgical event log for the existing surgical procedure. Preferably, the stop signal is the result of user input via a user interface (e.g., user interface 905). In one embodiment, the stop signal is received when the system is powered off (e.g., when the user actuates a power on / off switch). These examples are merely exemplary and are not intended to be limiting.

[0073] If a stop signal is received, algorithm 900 continues to 960, where the images of each frame constituting the surgical video recording are compiled to produce a visual video recording, and the system events constituting the entries in the surgical event log are compiled to form a complete surgical event log. The compiled surgical video recording and surgical event log are then saved to memory device 940 for later use. If no stop signal is received, algorithm 900 loops back to run 915 again.

[0074] Figure 10 An algorithm 1000 is shown that performs video content search using a combination of image data and auxiliary data. In one application, algorithm 1000 can be used in various surgical video recordings (e.g., in...). Figure 9 The algorithm 1000 searches for a specific topic of interest within surgical video recordings generated by algorithm 900, as shown below. In this application, data contained in the surgical event log (each piece of which is associated with a specific surgical video recording) constitutes auxiliary data that can be used to facilitate the search of surgical video recordings for a specific topic of interest. Algorithm 1000 can be executed by a computer processor of the device executing algorithm 900 (e.g., a computer processor as part of a remote surgical system). Algorithm 1000 can also be executed by a computer processor that communicates data with the device executing algorithm 900 (e.g., a personal computing device).

[0075] In one embodiment, Algorithm 1000 leverages (1) known correlations between specific system events recorded in the surgical event log and (2) specific medical topics of interest to facilitate user searches for portions of surgical video recordings containing specific medical topics of interest. As discussed previously, this approach of Algorithm 1000 improves upon various drawbacks associated with earlier search algorithms that relied solely on image data analysis.

[0076] Reference Figure 10 In one embodiment, algorithm 1000 is initiated by a user request commanded via user interface 1005. At user interface 1005, the user requests to search for certain surgical video recordings on a topic of interest to the user. In one example, the user is a surgeon preparing to perform a specific surgical procedure involving a particular surgical technique, and the surgeon wants to view archived surgical video recordings showing portions of the specific surgical technique. User interface 1005 may be a microphone, which the user can use to command a search for a topic of interest in previously recorded video. Alternatively, the user interface may be a touchpad or some alternative user interface that provides text input. These examples are merely exemplary and are not intended to be limiting.

[0077] In one embodiment, at 1010, the query mapper associates one or more terms included in the user search requested at 1005 with one of the system events typically recorded by the surgical event log. Preferably, the query mapper 1010 operates by searching a lookup table containing known associations between (1) various key terms or phrases and (2) individual system events or patterns among several system events. As additional associations become known, this lookup table can be updated to reflect the additional associations.

[0078] In one example, at 1005, a user requests to search for surgical techniques used to reduce patient bleeding during a surgical procedure within surgical video recordings. At 1010, the query mapper operates on this user query and determines system events indicating frequent application of electrocautery functions associated with cauterization instruments during a given surgical procedure, corresponding to common surgical techniques used to reduce patient bleeding. Continuing this example, at 1015, the event searcher searches the surgical event log archived in database 1035 for recurring applications of electrocautery within a given time period (e.g., at least 10 applications of electrocautery energy within a 1-minute time span).

[0079] Reference Figure 10 Example database 1035 shows surgical event logs 1040a, 1041a, and 1042a. Each surgical event log in surgical event logs 1040a, 1041a, and 1042a is associated with one of surgical video recordings 1040b, 1041b, and 1042b. Continuing with the above example, event searcher 1015 searches through surgical event logs 1040a, 1041a, and 1042a for instances of repeated applications of electrocautery within a given time period, and finds that only surgical event logs 1040a and 1041a contain time periods of electrocautery applications that meet a specified frequency.

[0080] As discussed previously regarding Algorithm 900, in one embodiment, each system event in the surgical event log (e.g., surgical event logs 1040a, 1041a, 1042a) is archived with a corresponding timestamp, and each frame in the surgical video recordings (e.g., surgical video recordings 1040b, 1041b, 1042b) is archived with a corresponding timestamp. Video clip extractor 1020 operates on the output of event searcher 1015 using one or more timestamps associated with system events in the surgical event logs to retrieve corresponding clips from the surgical video recordings. Continuing with the example above, using the output of event searcher 1015, video clip extractor 1020 retrieves candidate video clips from surgical video recordings 1040b, 1041b for each portion of the electrocautery application in surgical event logs 1040a, 1041a that meets a specified frequency.

[0081] Each candidate video segment retrieved by video segment extractor 1020 is submitted to video analyzer 1025 for analysis. Video analyzer 1025 analyzes the image data corresponding to each candidate video segment to determine whether each candidate video segment actually contains the topic of interest sought by the user. Continuing the example above, video analyzer 1025 determines that only one candidate video segment from surgical video recording 1040b actually contains the topic of interest sought by the user, and identifies this candidate video segment as the identified video segment. At 1030, this identified video segment is presented to the user for viewing.

[0082] Although illustrative embodiments have been shown and described, a wide range of modifications, alterations, and substitutions are conceived in the foregoing disclosure and some examples, and some features of the described embodiments may be employed without corresponding use of other features. Those skilled in the art will recognize many variations, alternatives, and modifications. Therefore, the scope of this disclosure should be limited only by the following claims, and it is appropriate to interpret the appended claims broadly in accordance with the scope of this disclosure.

Claims

1. A method comprising: Receive user commands to locate one or more video segments from one or more surgical video recordings that display a medical event of interest; Identify one or more system events from one or more surgical event logs that may correspond to the occurrence of the medical event of interest; One or more candidate video segments are identified from the one or more surgical video recordings using one or more timestamps corresponding to each of the one or more identified system events; By analyzing the image data of the one or more candidate video segments, it is determined whether each of the one or more candidate video segments is an identified video segment containing the medical event of interest; as well as Present at least one identified video segment to the user.

2. The method of claim 1, wherein, Receiving user commands for locating one or more video clips displaying a medical event of interest includes: Present multiple medical events of interest for selection via the user interface; The system allows users to select one of several medical events of interest presented.

3. The method of claim 1, wherein, Receiving user commands for locating one or more video clips displaying a medical event of interest includes receiving the user commands during surgery, while the user is performing a medical procedure.

4. The method of claim 1, wherein, Receiving user commands for locating one or more video clips displaying a medical event of interest includes receiving the user commands before or after surgery.

5. The method of claim 1, wherein, Identifying one or more system events that may correspond to the occurrence of the medical event of interest includes searching a lookup table.

6. The method of claim 5, wherein, The lookup table is periodically and automatically updated to interpret the correlation between system events learned through machine learning and medical events of interest.

7. The method of claim 1, wherein, The one or more system events that may correspond to the occurrence of the medical event of interest include the occurrence of a single system event.

8. The method of claim 1, wherein, The one or more system events that may correspond to the occurrence of the medical event of interest include the occurrence of two or more system events corresponding to a specific pattern.

9. The method of claim 1, wherein, The one or more system events that may correspond to the occurrence of the medical event of interest include the occurrence of a single system event for an extended duration.

10. The method of claim 1, wherein, One or more candidate video segments are identified using one or more timestamps corresponding to each of the one or more identified system events, including identifying video segments of surgical video recordings that occur between two timestamps.

Images