SYSTEM, METHOD, AND COMPUTER PROGRAM PRODUCT FOR CONTROLLING IMAGE CAPTURE DEVICES INTRA-SURGICAL OPERATIONS - Patent application

Abstract

A system for controlling a medical image capture device during surgery is provided, the system including circuitry configured to acquire first image data from the medical image capture device, the first image data being an appearance of a surgical scene at a first time instance, determine one or more desired image capture characteristics of the medical image capture device based on a predicted appearance of the surgical scene at a second time instance after the first time instance based on the first image data, and control the medical image capture device at a third time instance between the first time instance and the second time instance in accordance with the one or more desired image capture characteristics of the medical image capture device.

Description

[Technical Field]

[0001] SUMMARY The present disclosure relates to methods, devices, and systems for controlling image capture devices during surgery. [Background technology]

[0002] The "Background" discussion provided herein is intended to generally present the context for the present disclosure. The inventors' work described in the Background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are not admitted, expressly or implicitly, as prior art to the present disclosure.

[0003] Significant technological advances have been made in medical systems and devices in recent years. Computer-assisted surgical systems, such as robotic systems, now often work alongside human surgeons during surgery. These computer-assisted surgical systems include master-slave robotic systems, where the human surgeon operates a master console to control the operation of slave devices during surgery.

[0004] Computer-aided camera systems, such as robotic camera systems, are used to provide visual information to a human operator or surgeon in a surgical environment. These computer-aided camera systems can include a single camera that captures and provides a view of the surgical activity within a scene. Alternatively, these computer-aided camera systems can include multiple cameras, each capturing a given view of the surgical activity within a scene.

[0005] However, the view of the surgical scene is often obstructed by elements such as tools, the surgeon's hands, tissue deformation, or other dynamic elements. That is, surgical scenes are often highly complex and include many dynamic elements that can obstruct the view obtained from a medical image capture device such as an endoscope or microscope.

[0006] These obstructions can delay the capture and provision of important visual information needed to inform the surgeon or robotic system of changes in the scene. Summary of the Invention [Problem to be solved by the invention]

[0007] In a surgical environment, delays in providing important visual information to the surgeon can have serious consequences.

[0008] The purpose of this disclosure is to address these issues. [Means for solving the problem]

[0009] According to a first aspect of the present disclosure, there is provided a system for controlling a medical image capture device during surgery, the system including circuitry configured to acquire first image data from the medical image capture device, the first image data being an appearance of a surgical scene at a first time instance, determine one or more desired image capture characteristics of the medical image capture device based on a predicted appearance of the surgical scene at a second time instance after the first time instance based on the first image data, and control the medical image capture device at a third time instance between the first time instance and the second time instance in accordance with the one or more desired image capture characteristics of the medical image capture device.

[0010] According to a second aspect of the present disclosure, there is provided a method for controlling a medical image capture device during surgery, the method including the steps of: acquiring first image data from the medical image capture device, the first image data being an appearance of a surgical scene at a first time instance; determining one or more desired image capture characteristics of the medical image capture device based on a predicted appearance of the surgical scene at a second time instance after the first time instance based on the first image data; and controlling the medical image capture device at a third time instance between the first time instance and the second time instance in accordance with the one or more desired image capture characteristics of the medical image capture device.

[0011] According to a third aspect of the present disclosure, there is provided a computer program product including instructions that, when executed by the computer, cause the computer to perform a method for controlling a medical image capture device, the method including the steps of: acquiring first image data from the medical image capture device, the first image data being an appearance of a surgical scene at a first time instance; determining one or more desired image capture characteristics of the medical image capture device based on a predicted appearance of the surgical scene at a second time instance after the first time instance based on the first image data; and controlling the medical image capture device at a third time instance between the first time instance and the second time instance in accordance with the one or more desired image capture characteristics of the medical image capture device.

[0012] Aspects of the present disclosure enable a computer-aided camera system to predictively optimize camera position so that the impact of foreseeable future changes to the surgical usability and predictability of the captured scene is compensated for by camera system movement before the changes occur, thereby reducing delays in capturing and providing critical visual information to a surgeon or surgical robotic system during surgery.

[0013] The foregoing paragraphs have been provided by way of general introduction and are not intended to limit the scope of the claims that follow. The described embodiments, together with further advantages, will be best understood by reference to the following detailed description considered in conjunction with the accompanying drawings, in which:

[0014] A more complete appreciation of the present disclosure and many of the attendant advantages will be readily obtained as the same becomes better understood by reference to the following detailed description, when considered in connection with the accompanying drawings. [Brief explanation of the drawings]

[0015] [Figure 1]FIG. 1 is a diagram showing an example of the schematic configuration of an endoscopic surgery system to which a medical support arm device according to the present disclosure can be applied. [Figure 2] FIG. 2 is a block diagram showing an example of a functional configuration of the camera head and the CCU (camera control unit) shown in FIG. [Figure 3] FIG. 3 is an explanatory diagram showing an example of use of the master device according to the present disclosure. [Figure 4] FIG. 4 illustrates an apparatus for controlling an image capture device during surgery according to an embodiment of the present disclosure. [Figure 5A] FIG. 5A illustrates an exemplary situation in which embodiments of the present disclosure can be applied. [Figure 5B] FIG. 5B illustrates exemplary first image data according to an embodiment of the present disclosure. [Figure 5C] FIG. 5C illustrates a timeline for generating a predicted appearance of a surgical scene according to an embodiment of the present disclosure. [Figure 5D] FIG. 5D illustrates exemplary second image data according to an embodiment of the present disclosure. [Figure 5E] FIG. 5E illustrates an exemplary image obtained by an image capture device during surgery according to an embodiment of the present disclosure. [Figure 6A] FIG. 6A illustrates an apparatus for controlling an image capture device during surgery according to an embodiment of the present disclosure. [Figure 6B] FIG. 6B illustrates an exemplary situation in which embodiments of the present disclosure can be applied. [Figure 7A] FIG. 7A illustrates an exemplary configuration of a computer-assisted surgery system according to the current embodiment. [Figure 7B] FIG. 7B illustrates an exemplary situation in which embodiments of the present disclosure may be applied. [Figure 7C] FIG. 7C illustrates an exemplary situation in which embodiments of the present disclosure may be applied. [Figure 7D] FIG. 7D illustrates an exemplary situation in which embodiments of the present disclosure may be applied. [Figure 7E] FIG. 7E illustrates an exemplary situation in which embodiments of the present disclosure may be applied. [Figure 8] FIG. 8 illustrates a method for controlling an image capture device during surgery according to an embodiment of the present disclosure. [Figure 9] FIG. 9 illustrates a computing device for controlling an image capture device during surgery according to an embodiment of the present disclosure. [Figure 10] FIG. 10 shows a schematic diagram of a first example of a computer-assisted surgery system to which the present technology can be applied. [Figure 11] FIG. 11 shows a schematic diagram of a second example of a computer-assisted surgery system to which the present technology can be applied. [Figure 12] FIG. 12 shows a schematic diagram of a third example of a computer-assisted surgery system to which the present technology can be applied. [Figure 13] FIG. 13 shows a schematic diagram of a fourth example of a computer-assisted surgery system to which the present technology can be applied. [Figure 14] FIG. 14 shows a schematic diagram of an example of an arm unit. DETAILED DESCRIPTION OF THE INVENTION

[0016] Referring now to the drawings, like reference numerals refer to the same or corresponding parts throughout the several views.

[0017] <<1.Basic configuration>> First, the basic configuration of an endoscopic surgery system to which an embodiment of the present disclosure can be applied will be described with reference to FIGS. 1 to 4 of the present disclosure.

[0018] <1.1. Example of endoscopic surgery system configuration> Fig. 1 is a diagram showing an example of the general configuration of an endoscopic surgery system 5000 to which the technology according to the present disclosure can be applied. Fig. 1 shows a state in which an operator (doctor) 5067 is performing surgery on a patient 5071 on a patient bed 5069 using the endoscopic surgery system 5000. As shown in the figure, the endoscopic surgery system 5000 is composed of an endoscope 5001, other surgical tools 5017, a support arm device 5027 that supports the endoscope 5001, and a cart 5037 on which various devices for endoscopic surgery are mounted.

[0019] In endoscopic surgery, instead of cutting the abdominal wall to open the abdominal cavity, the abdominal wall is punctured with a plurality of tubular drilling instruments called trocars 5025a-5025d. Then, the lens barrel 5003 of the endoscope 5001 and other surgical tools 5017 are inserted into the body cavity of the patient 5071 through the trocars 5025a-5025d. In the illustrated example, the other surgical tools 5017 inserted into the body cavity of the patient 5071 include an air supply tube 5019, an energy treatment tool 5021, and forceps 5023. Furthermore, the energy treatment tool 5021 is a treatment tool that uses high-frequency current or ultrasonic vibration to perform tasks such as incising and dissecting tissue, sealing blood vessels, etc. However, the illustrated surgical tool 5017 is merely an example, and various surgical tools commonly used in endoscopic surgery, such as tweezers and retractors, can be used as the surgical tool 5017.

[0020] An image of the surgical site inside the body cavity of the patient 5071, captured by the endoscope 5001, is displayed on the display device 5041. An operator 5067 performs a procedure, for example, excising an affected area, using the energy treatment tool 5021 or forceps 5023 while viewing the image of the surgical site displayed in real time on the display device 5041. Although not shown, the air supply tube 5019, the energy treatment tool 5021, and the forceps 5023 are supported by the operator 5067 or an assistant during surgery.

[0021] (Support arm device) The support arm device 5027 includes an arm unit 5031 extending from a base unit 5029. In the illustrated example, the arm unit 5031 is a multi-joint arm composed of joints 5033a, 5033b, and 5033c and links 5035a and 5035b, and is driven under control of an arm control device 5045. The arm unit 5031 has a distal end to which an endoscope 5001 can be connected. The endoscope 5001 is supported by the arm unit 5031, and its position and attitude are controlled. This configuration makes it possible to stably fix the position of the endoscope 5001.

[0022] (Endoscopy) The endoscope 5001 is composed of a lens barrel 5003 having a region of a predetermined length from the distal end that is inserted into the body cavity of the patient 5071, and a camera head 5005 connected to the proximal end of the lens barrel 5003. In the illustrated example, the endoscope 5001 is configured as a so-called rigid lens barrel having a solid lens barrel 5003, but the endoscope 5001 may also be configured as a so-called flexible lens barrel having a flexible lens barrel 5003.

[0023] An opening into which an objective lens is fitted is provided at the distal end of the lens barrel 5003. A light source device 5043 is connected to the endoscope 5001, and light generated by the light source device 5043 is guided to the distal end of the lens barrel by a light guide extending inside the lens barrel 5003, and is emitted through the objective lens toward an object to be observed inside the body cavity of the patient 5071. The endoscope 5001 may be a direct-viewing endoscope, an oblique-viewing endoscope, or a side-viewing endoscope.

[0024] An optical system and an image sensor are provided inside the camera head 5005, and light reflected from the object of observation (observation light) is collected onto the image sensor by the optical system. The observation light is photoelectrically converted by the image sensor to generate an electrical signal corresponding to the observation light, i.e., an image signal corresponding to the observed image. The image signal is transmitted to a camera control unit (CCU) 5039 as RAW data. The camera head 5005 has a function for adjusting the magnification and focal length by appropriately driving the optical system.

[0025] For example, to support stereoscopic vision (3D display), multiple imaging elements can be provided in camera head 5005. In this case, multiple relay optical systems are provided inside lens barrel 5003 to guide observation light to each of the multiple imaging elements.

[0026] (Various devices provided on the cart) The CCU 5039 is configured using a central processing unit (CPU) or a graphics processing unit (GPU), and integrally controls the operations of the endoscope 5001 and the display device 5041. Specifically, the CCU 5039 performs various types of image processing, such as development processing (demosaic processing), on the image signal received from the camera head 5005 in order to display an image based on the image signal. The CCU 5039 supplies the image signal after image processing to the display device 5041. The CCU 5039 also transmits control signals to the camera head 5005 to control the driving of the camera head 5005. The control signals may include information regarding imaging conditions such as magnification and focal length.

[0027] The display device 5041, under the control of the CCU 5039, displays an image based on an image signal that has been image processed by the CCU 5039. If the endoscope 5001 is an endoscope that is capable of capturing high-resolution images such as 4K (3840 horizontal pixels × 2160 vertical pixels) or 8K (7680 horizontal pixels × 4320 vertical pixels) and / or is an endoscope that is capable of 3D display, the display device 5041 can be a device capable of high-resolution display and / or a device capable of 3D display that is compatible with the endoscope. In the case of an endoscope that is capable of capturing high-resolution images such as 4K and 8K, a more immersive feeling can be achieved by using a display device 5041 that is 55 inches or larger. Furthermore, multiple display devices 5041 with different resolutions and sizes may be provided depending on the application.

[0028] The light source device 5043 is configured using a light source such as a light emitting diode (LED), and supplies the endoscope 5001 with illumination light when photographing the surgical site.

[0029] The arm control device 5045 is configured using a processor such as a CPU, and operates according to a predetermined program to control the driving of the arm unit 5031 of the support arm device 5027 according to a predetermined control method.

[0030] The input device 5047 is an input interface for the endoscopic surgery system 5000. A user can input various types of information and instructions to the endoscopic surgery system 5000 via the input device 5047. For example, the user inputs various types of information related to the surgery, such as information about the patient's body and information about the surgical procedure, via the input device 5047. Furthermore, for example, the user inputs via the input device 5047 an instruction to drive the arm unit 5031, an instruction to change the imaging conditions using the endoscope 5001 (type of irradiation light, magnification, focal length, etc.), an instruction to drive the energy treatment tool 5021, etc.

[0031] The type of input device 5047 is not limited, and may be any of various known input devices. For example, a mouse, a keyboard, a touch panel, a switch, a foot switch 5057, and / or a lever may be applied as the input device 5047. When a touch panel is used as the input device 5047, the touch panel may be provided on the display surface of the display device 5041.

[0032] Alternatively, the input device 5047 may be a device worn by the user, such as a glasses-type wearable device or a head-mounted display (HMD), and various inputs are made in response to the user's gestures or gaze detected by these devices. Furthermore, the input device 5047 may include a camera capable of detecting the user's movements, and various inputs are made in response to the user's gestures or gaze detected from images captured by the camera. Furthermore, the input device 5047 may include a microphone capable of collecting the user's voice, and various inputs are made by voice via the microphone. In this manner, the input device 5047 is configured to input various types of information in a contactless manner, and in particular, a user (e.g., operator 5067) in a clean area can operate equipment in a non-clean area in a contactless manner. Furthermore, the user can operate the equipment without taking their hands off the surgical tool, thereby improving user convenience.

[0033] The treatment tool control device 5049 controls the operation of the energy treatment tool 5021 for cauterizing tissue, incising, sealing blood vessels, etc. The air supply device 5051 supplies gas to the body cavity of the patient 5071 through the air supply tube 5019 to inflate the body cavity for the purpose of ensuring visibility for the endoscope 5001 and ensuring workspace for the operator. The recorder 5053 is a device that can record various types of information related to the surgery. The printer 5055 is a device that can print various types of information related to the surgery in various formats, such as text, images, and graphs.

[0034] Below, the particularly characteristic configuration of the endoscopic surgery system 5000 will be described in further detail.

[0035] (Support arm device) The support arm device 5027 includes a base unit 5029 as a foundation and an arm unit 5031 extending from the base unit 5029. In the illustrated example, the arm unit 5031 is configured with multiple joints 5033a, 5033b, and 5033c and multiple links 5035a and 5035b connected by the joint 5033b; however, for convenience, FIG. 1 illustrates the configuration of the arm unit 5031 in a simplified manner. In practice, the shape, number, and arrangement of each of the joints 5033a to 5033c and the links 5035a and 5035b, as well as the direction of the rotation axis of each of the joints 5033a to 5033c, are appropriately set so that the arm unit 5031 has a desired degree of freedom. For example, the arm unit 5031 can be configured to have preferably six or more degrees of freedom. This configuration allows the endoscope 5001 to be moved freely within the movable range of the arm unit 5031, and therefore the lens barrel 5003 of the endoscope 5001 can be inserted into the body cavity of the patient 5071 from the desired direction.

[0036] The joints 5033a to 5033c are provided with actuators, and the joints 5033a to 5033c are configured to be rotatable around predetermined rotation axes by the drive of the actuators. When the drive of the actuators is controlled by the arm control device 5045, the rotation angle of each of the joints 5033a to 5033c is controlled, and the drive of the arm unit 5031 is controlled. This configuration makes it possible to control the position and attitude of the endoscope 5001. At this time, the arm control device 5045 can control the drive of the arm unit 5031 by various known control methods such as force control or position control.

[0037] For example, when the operator 5067 appropriately executes an operation input via the input device 5047 (including the foot switch 5057) and the drive of the arm unit 5031 is appropriately controlled by the arm control device 5045 in accordance with the operation input, the position and attitude of the endoscope 5001 can be controlled. By such control, the endoscope 5001 at the distal end of the arm unit 5031 can be moved from any position to any position and then fixedly supported at the position after movement. Note that the arm unit 5031 can be operated in a so-called master-slave manner. In this case, the arm unit 5031 can be remotely controlled by a user via the input device 5047 installed in a location away from the operating room.

[0038] Furthermore, when force control is applied, the arm control device 5045 can execute so-called power assist control for receiving an external force from a user and driving the actuators of the joints 5033a to 5033c so that the arm unit 5031 moves smoothly in response to the external force. With this configuration, when the user moves the arm unit 5031 while directly touching the arm unit 5031, the arm unit 5031 can be moved with a relatively light force. Therefore, the endoscope 5001 can be moved more intuitively with simpler operations, improving user convenience.

[0039] Here, the endoscope 5001 is generally supported by a doctor called a scopist during endoscopic surgery. In this regard, by using the support arm device 5027, it becomes possible to more reliably fix the position of the endoscope 5001 without relying on human hands, and therefore it becomes possible to obtain stable images of the surgical site and perform the surgery smoothly.

[0040] It should be noted that the arm control device 5045 does not necessarily have to be provided on the cart 5037. Furthermore, the arm control device 5045 does not necessarily have to be a single device. For example, an arm control device 5045 can be provided for each of the joints 5033a to 5033c of the arm unit 5031 of the support arm device 5027, or drive control of the arm unit 5031 may be realized by a plurality of arm control devices 5045 cooperating with each other.

[0041] (Light source device) The light source device 5043 supplies the endoscope 5001 with illumination light for capturing images of the surgical site. The light source device 5043 is configured using a white light source configured, for example, by an LED, a laser light source, or a combination of these. In this case, if the white light source is configured by a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision, and the white balance of the captured image can be adjusted by the light source device 5043. Furthermore, in this case, it is also possible to capture images corresponding to each of the RGB colors in a time-division manner by irradiating the object of observation with laser light from each of the RGB laser light sources in a time-division manner and controlling the drive of the image sensor of the camera head 5005 in synchronization with the irradiation timing. This method makes it possible to obtain a color image without providing a color filter to the image sensor.

[0042] Furthermore, the driving of the light source device 5043 can be controlled so as to change the intensity of the light output at predetermined time intervals. By controlling the driving of the image sensor of the camera head 5005 in synchronization with the timing of the change in the light intensity, it is possible to acquire images in a time-division manner and synthesize the images to generate an image with a high dynamic range that is free from so-called blocked-up shadows and blown-out highlights.

[0043] Furthermore, the light source device 5043 may be configured to supply light in a predetermined wavelength band suitable for special light observation. Special light observation, for example, utilizes the wavelength dependence of light absorption in body tissues. By emitting light in a narrower band than the illumination light (i.e., white light) used in normal observation, so-called narrow band imaging (NBI) can be performed to capture high-contrast images of specific tissues, such as blood vessels on the surface of mucous membranes. Alternatively, special light observation can also be performed using fluorescence observation, in which images are obtained using fluorescence generated by emitting excitation light. Fluorescence observation can involve irradiating body tissues with excitation light and observing the fluorescence from the body tissue (autofluorescence observation), or locally injecting a reagent such as indocyanine green (ICG) into the body tissue and then irradiating the body tissue with excitation light corresponding to the fluorescent wavelength of the reagent to obtain a fluorescence image. The light source device 5043 can be configured to supply narrow band light and / or excitation light suitable for such special light observation.

[0044] (camera head and CCU) The functions of the camera head 5005 and the CCU 5039 of the endoscope 5001 will be described in further detail with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the functional configuration of the camera head 5005 and the CCU 5039 shown in Fig. 1.

[0045] Referring to FIG. 2, the camera head 5005 has, as its functions, a lens unit 5007, an imaging unit 5009, a drive unit 5011, a communication unit 5013, and a camera head control unit 5015.

[0046] Furthermore, the CCU 5039 has, as its functions, a communication unit 5059, an image processing unit 5061, and a control unit 5063. The camera head 5005 and the CCU 5039 are connected via a transmission cable 5065 so as to enable two-way communication.

[0047] First, the functional configuration of the camera head 5005 will be described. The lens unit 5007 is an optical system provided at the connection point with the lens barrel 5003. Observation light taken in from the distal end of the lens barrel 5003 is guided to the camera head 5005 and enters the lens unit 5007. The lens unit 5007 is configured by combining multiple lenses, including a zoom lens and a focus lens. The optical characteristics of the lens unit 5007 are adjusted so that the observation light is focused on the light receiving surface of the image sensor of the imaging unit 5009. Furthermore, the zoom lens and focus lens are configured so that the magnification and focal length of the captured image can be adjusted by moving their positions on the optical axis.

[0048] The imaging unit 5009 is composed of an imaging element and is arranged after the lens unit 5007. Observation light passing through the lens unit 5007 is condensed on the light receiving surface of the imaging element, and an image signal corresponding to the observed image is generated by photoelectric conversion. The image signal generated by the imaging unit 5009 is sent to the communication unit 5013.

[0049] As the imaging element constituting the imaging unit 5009, for example, a complementary metal oxide semiconductor (CMOS) type image sensor having a Bayer array and capable of color imaging can be used. Note that as the imaging element, for example, an imaging element suitable for capturing high-resolution images of 4K or higher can be used. Since high-resolution images of the surgical site can be obtained, the operator 5067 can grasp the situation of the surgical site in more detail, and the surgery can proceed more smoothly.

[0050] Furthermore, the imaging element constituting the imaging unit 5009 is configured to have a pair of imaging elements for acquiring image signals for the right eye and the left eye, respectively, that are compatible with 3D display. 3D display allows the operator 5067 to more accurately grasp the depth of the biological tissue at the surgical site. When the imaging unit 5009 is configured as a multi-plate type, multiple lens units 5007 are provided corresponding to the respective imaging elements.

[0051] Furthermore, the imaging unit 5009 does not necessarily have to be provided in the camera head 5005. For example, the imaging unit 5009 can be provided inside the lens barrel 5003 immediately after the objective lens.

[0052] The drive unit 5011 is configured using an actuator, and moves the zoom lens and focus lens of the lens unit 5007 by a predetermined distance along the optical axis under the control of the camera head control unit 5015. This movement makes it possible to appropriately adjust the magnification and focal length of the image captured by the imaging unit 5009.

[0053] The communication unit 5013 is configured using a communication device and transmits and receives various types of information to and from the CCU 5039. The communication unit 5013 transmits image signals obtained from the imaging unit 5009 as RAW data to the CCU 5039 via a transmission cable 5065. In this case, it is preferable that the image signals be transmitted via optical communication to display the captured images of the surgical site with low latency. During surgery, the operator 5067 performs the surgery while observing the condition of the affected area through the captured images. For safer and more reliable surgery, moving images of the surgical site must be displayed as quickly as possible in real time. When optical communication is performed, an optoelectronic conversion module that converts electrical signals into optical signals is provided in the communication unit 5013. The image signals are converted into optical signals by the optoelectronic conversion module and then transmitted to the CCU 5039 via the transmission cable 5065.

[0054] Furthermore, the communication unit 5013 receives control signals from the CCU 5039 for controlling the operation of the camera head 5005. The control signals include information related to imaging conditions, such as information for specifying the frame rate of the captured image, information for specifying the exposure value during imaging, and / or information for specifying the magnification and focal length of the captured image. The communication unit 5013 provides the received control signals to the camera head control unit 5015. Note that the control signals from the CCU 5039 may also be transmitted by optical communication. In this case, the communication unit 5013 includes an opto-electrical conversion module for converting optical signals into electrical signals, and the control signals are converted into electrical signals by the opto-electrical conversion module and then transmitted to the camera head control unit 5015.

[0055] The imaging conditions such as the frame rate, exposure value, magnification, and focal length described above are automatically set by the control unit 5063 of the CCU 5039 based on the acquired image signal. That is, the endoscope 5001 has so-called automatic exposure (AE) function, autofocus (AF) function, and automatic white balance (AWB) function.

[0056] The camera head control unit 5015 controls the driving of the camera head 5005 based on a control signal received from the CCU 5039 via the communication unit 5013. For example, the camera head control unit 5015 controls the driving of the image sensor of the imaging unit 5009 based on information specifying the frame rate of the captured image and / or information specifying the exposure during imaging. Furthermore, for example, the camera head control unit 5015 appropriately moves the zoom lens and focus lens of the lens unit 5007 via the drive unit 5011 based on information specifying the magnification and focal length of the captured image. The camera head control unit 5015 may also have a function to store information for identifying the lens barrel 5003 and the camera head 5005.

[0057] Incidentally, by arranging components such as the lens unit 5007 and the imaging unit 5009 in a sealed structure that is highly airtight and waterproof, the camera head 5005 can be made resistant to autoclave sterilization.

[0058] Next, the functional configuration of the CCU 5039 will be described. The communication unit 5059 is configured using a communication device and transmits and receives various types of information to and from the camera head 5005. The communication unit 5059 receives image signals transmitted from the camera head 5005 via a transmission cable 5065. In this case, as described above, the image signals can be suitably transmitted by optical communication. In this case, to be compatible with optical communication, the communication unit 5059 includes an opto-electrical conversion module that converts optical signals into electrical signals. The communication unit 5059 provides the converted image signals to the image processing unit 5061.

[0059] Furthermore, the communication unit 5059 transmits a control signal to the camera head 5005 for controlling the driving of the camera head 5005. This control signal may also be transmitted by optical communication.

[0060] The image processing unit 5061 performs various types of image processing on the image signal, which is RAW data transmitted from the camera head 5005. For example, the image processing includes various types of known signal processing, such as development processing, image quality improvement processing (e.g., band enhancement processing, super-resolution processing, noise reduction (NR) processing, and / or image stabilization processing), and / or enlargement processing (electronic zoom processing). Furthermore, the image processing unit 5061 performs detection processing on the image signal to perform AE, AF, and AWB.

[0061] The image processing unit 5061 is configured using a processor such as a CPU and a GPU, and can execute the above-described image processing and detection processing when the processor operates according to a predetermined program. Note that, when the image processing unit 5061 is configured with multiple GPUs, the image processing unit 5061 appropriately divides information related to the image signal and executes image processing in parallel using the multiple GPUs.

[0062] The control unit 5063 performs various types of control related to the imaging of the surgical site using the endoscope 5001 and the display of such captured images. For example, the control unit 5063 generates a control signal for controlling the driving of the camera head 5005. At this time, if the imaging conditions are input by the user, the control unit 5063 generates the control signal based on the user's input. Alternatively, if the endoscope 5001 has an AE function, an AF function, and an AWB function, the control unit 5063 appropriately calculates the optimal exposure value, focal length, and white balance according to the result of the detection processing by the image processing unit 5061, and generates the control signal.

[0063] Furthermore, the control unit 5063 causes the display device 5041 to display an image of the surgical site based on the image signal processed by the image processing unit 5061. At this time, the control unit 5063 uses various image recognition technologies to recognize various objects in the image of the surgical site. For example, the control unit 5063 can recognize surgical tools such as forceps, specific body parts, bleeding, mist generated when using the energy treatment tool 5021, and the like by detecting the shape and color of the edges of objects included in the image of the surgical site. When the control unit 5063 causes the display device 5041 to display the image of the surgical site, the control unit 5063 uses the recognition results to superimpose various types of surgical assistance information on the image of the surgical site. The surgical assistance information is superimposed and presented to the operator 5067, enabling the surgery to proceed more safely and reliably.

[0064] The transmission cable 5065 connecting the camera head 5005 and the CCU 5039 is an electrical signal cable for communication of electrical signals, an optical fiber for optical communication, or a composite cable of these.

[0065] Here, in the illustrated example, communication is performed wired using a transmission cable 5065, but communication between the camera head 5005 and the CCU 5039 may be performed wirelessly. When communication between them is performed wirelessly, there is no need to lay the transmission cable 5065 in the operating room, which eliminates the situation where the transmission cable 5065 interferes with the movement of medical staff in the operating room.

[0066] An example of an endoscopic surgery system 5000 to which the technology according to the present disclosure can be applied has been described above. Note that although the endoscopic surgery system 5000 has been described here as an example, systems to which the technology according to the present disclosure can be applied are not limited to this example. For example, the technology according to the present disclosure can also be applied to a flexible endoscope system for inspection or a microsurgery system.

[0067] Alternatively, aspects of the present disclosure are applicable to medical robotic systems, including master-slave medical robotic systems, in which a user (such as a doctor 5067) operates a master device (surgeon's console) to send motion commands to and remotely control a slave device (a bedside cart) via wired or wireless communication means. The medical robotic system may further include a separate cart containing several supporting hardware and software components, such as an electrosurgical unit (ESU), a suction / irrigation pump, and a light source for an endoscope / microscope.

[0068] FIG. 3 shows an example of how to use the master device 60 according to the present disclosure. In FIG. 3, two master devices 60R and 60L, one for the right hand and one for the left hand, are provided. The surgeon rests both arms or elbows on the support base 50 and grasps the control units 100R and 100L with his or her right and left hands, respectively. In this state, the surgeon operates the control units 100R and 100L while viewing the monitor 210 showing the surgical site. By displacing the position or direction of each control unit 100R and 100L, the surgeon can remotely control the position or direction of surgical instruments attached to slave devices (none of which are shown), or perform grasping operations using each surgical instrument.

[0069] The basic configuration of an exemplary surgical system to which embodiments of the present disclosure can be applied has been described above with reference to Figures 1 to 4 of the present disclosure. Specific embodiments of the present disclosure will be described below.

[0070] <Device for controlling image capture device during surgery> As noted above, it would be desirable to provide an apparatus that reduces delays in providing important visual information to an operating or robotic surgeon due to dynamically changing surgical environments. Thus, in accordance with an embodiment of the present disclosure, an apparatus is provided for controlling an image capture device during surgery.

[0071] The apparatus for controlling image capture devices during surgery is applicable to an exemplary endoscopic surgery, such as the endoscopic surgery described with reference to FIG. 1 of the present disclosure. Accordingly, embodiments of the present disclosure will be described with reference to this exemplary surgery. However, it will be understood that the present disclosure is not intended to be limited to this particular surgery. Rather, embodiments of the present disclosure are applicable to any such surgery. Indeed, embodiments of the present disclosure are applicable to any surgery that includes computer-assisted surgery systems and devices.

[0072] <Example Situation> Returning now to FIG. 1 of the present disclosure, in this exemplary procedure, a medical doctor (or, as used herein, a surgeon) 5067 is performing an endoscopic procedure on a patient 5071. The surgeon 5067 is unable to see the interior of the patient's 5071 body cavity with his or her own eyes. Rather, the surgeon relies on images captured by the endoscopic device 5001 that are displayed on the display screen 5041. Thus, in this example, the endoscopic device captures images of the surgical scene and provides those images to the surgeon. This allows the surgeon 5067 to use a surgical tool (e.g., energy treatment tool 5021) to perform a surgical procedure inside the patient's 5071 body cavity, even though the surgeon 5067 cannot directly see the interior of the patient's body cavity.

[0073] In this sense, the endoscopic device captures important visual information of the surgical scene for display to the surgeon.

[0074] 1 , the endoscopic device 5001 is supported by a support arm 5027. The support arm therefore holds the endoscopic device 5001 in a predetermined position so that the endoscopic device captures an image of the surgical scene from an initial viewpoint. This initial viewpoint can be determined by the surgeon 5067 before the surgeon 5067 begins surgery. Movement of the support arm 5027 to position the endoscopic device 5001 in an initial position (corresponding to the initial viewpoint) to provide this first viewpoint of the surgical scene is controlled by the arm control device 5045.

[0075] Consider the insertion of an energy treatment tool 5021 into a body cavity of a patient by a surgeon 5067 during surgery, as may be necessary to perform a particular stage of surgery. The surgeon 5067 is guided in inserting the energy treatment tool by an image of the surgical scene captured by the endoscopic device 5001.

[0076] In this exemplary situation, once the energy treatment tool 5021 is inserted into the patient's body at the desired location, the surgeon 5067 begins to manipulate the energy treatment tool 5021 .

[0077] During operation, the energy treatment tool 5021 heats a target area of the patient, which may be done by the surgeon 5067 to cauterize bleeding. It will be appreciated that use of the energy treatment tool may generate mist or smoke within the body cavity of the patient 5071. This mist or smoke may obscure the view of the surgical scene from the perspective of the endoscopic device 5001, and as a result, the endoscopic device may no longer be able to obtain a clear image of the surgical scene. Thus, when using the energy treatment tool 5021, the surgeon (viewing images captured by the endoscopic device 5001 on the display screen 5041) may no longer be able to see a clear image of the surgical scene inside the patient's body.

[0078] In other words, the surgeon may not be able to receive important visual information from inside the patient's body cavity because the endoscope device 5001 can no longer capture a clear image of the surgical scene from the initial viewpoint during the operation of the energy treatment tool 5021. Therefore, at this time, the surgeon may not be able to recognize the occurrence of an important surgical event (such as additional bleeding due to surgery) during the operation of the energy treatment tool 5021.

[0079] Thus, the surgeon may have to stop operation of the energy treatment tool 5021 until the mist and / or smoke clears so that a clear image of the scene can be restored on the display device 5041. However, this slows the progress of the surgery. Alternatively, the surgeon 5067 (or the surgeon's 5067's human or computer assistant) may have to control the endoscopic device 5001 to attempt to reconfigure the endoscopic device 5001 so that a clear image of the scene can be obtained despite the mist and / or smoke caused by the operation of the energy treatment tool 5021. However, while this allows the surgeon 5067 to continue the surgery, there may be a delay between the loss of a clear image of the scene (due to the mist and / or smoke) and the restoration of a clear image of the scene. During this period, the surgeon may miss important safety-critical visual information about the surgical scene.

[0080] Thus, in accordance with an embodiment of the present disclosure, there is provided an apparatus in accordance with an embodiment of the present disclosure for controlling an image capture device during surgery.

[0081] Device: FIG. 4 illustrates an apparatus / system for controlling an image capture device, such as a medical image capture device, during surgery according to an embodiment of the present disclosure.

[0082] The apparatus 800 includes an acquisition unit 810 configured to acquire first image data, which is an appearance of a surgical scene at a first time instance, from a medical image capture device; a determination unit 820 configured to determine one or more desired image capture characteristics of the medical image capture device based on a predicted appearance of the surgical scene at a second time instance after the first time instance based on the first image data; and a control unit 830 configured to control the image capture device at a third time instance between the first time instance and the second time instance according to the one or more desired image capture characteristics of the medical image capture device.

[0083] In certain exemplary embodiments, controlling the image capture device includes controlling the position of an articulated arm supporting the image capture device. That is, the articulated arm supporting the image capture device can be controlled by the device / system 800 to control the position of the image capture device within a surgical scene. That is, returning to the exemplary situation of FIG. 1 of the present disclosure, the device 800 can be connected to the arm control unit 5045 to control the movement of the endoscopic device 5001. Alternatively, the device 800 can be connected to or form part of the CCU 5039.

[0084] 4, the apparatus 800 may optionally include a generating unit 820a configured to generate second image data, the second image data being a predicted appearance of the surgical scene at a second time instance after the first time instance, according to the first image data, thereby enabling the second image data being the predicted appearance of the surgical scene at the second time instance to be stored and used in subsequent processing (such as training a machine learning system).

[0085] The operation of the device 800 will now be described with reference to an exemplary endoscopic surgical context as shown with reference to Figure 1 of the present disclosure, however, it will be understood that the device may be applied to any surgical context as desired.

[0086] Acquisition Unit: During surgery, the acquisition unit 810 of the device 800 acquires a first image (or image data) of the surgical scene from the endoscopic device 5001. This first image provides the device 800 with information about the appearance of the surgical scene at the time the image was taken by the endoscopic device 5001. In this example, this first image is the same image that is displayed to the surgeon on the display device 5041. That is, the first image shows the current appearance of the surgical scene.

[0087] It will be appreciated that the manner in which the acquisition unit 810 acquires the first image data is not particularly limited. For example, the acquisition unit 810 can acquire the image data from an image capture device (such as the endoscopic device 5001) by any suitable wired or wireless means. Furthermore, the actual form of the image data will depend on the type of image capture device used to capture the image data. In this example (described with reference to FIG. 1 of the present disclosure), the image capture device is the endoscopic device 5001. Thus, in this example, the image data acquired by the acquisition unit 810 may be a high-resolution image, a 4K image, or an 8K image of the scene.

[0088] Now, in this exemplary situation, consider that the surgery has progressed to the stage where the surgeon 5067 has just inserted the energy treatment tool 5021 into the patient's body cavity. This exemplary surgical scene is shown in Figure 5A of the present disclosure.

[0089] 5A shows a target treatment area 9000 within a body cavity of a patient 5017. An energy treatment tool 5021 (inserted by a surgeon 5067) is shown approaching the target treatment area 9000. An endoscopic device 5001 captures an image of the surgical scene from a first perspective. The area of the surgical scene captured by the endoscopic device 5001 is indicated by area 9002. The image of this area captured by the endoscopic device 5001 is acquired by device 800 to form a first image of the surgical scene.

[0090] At this stage, the first image acquired by the device 800 includes a clear image of the surgical scene, i.e., the first image provides an unobstructed view of the surgical scene (specifically, of the target treatment area 9000) from the perspective of the endoscopic device 5001. An exemplary illustration of this first image data is shown in Figure 5B. As can be seen from the illustration in Figure 5B, the image from the perspective of the endoscopic device 5001 includes the target treatment area 9000 and at least a portion of the energy treatment tool 5021.

[0091] Decision Unit: Then, according to an embodiment of the present disclosure, the determination unit 820 of the device 800 is configured to determine one or more desired image capture characteristics of the image capture device 5001 based on a predicted appearance of the surgical scene at a second time instance after the first time instance based on the first image data.

[0092] The time step or time gap between the time of the first image and the time at which the predicted appearance of the first image of the scene is determined can vary depending on the circumstances. The duration of the time gap can be preset during initial configuration of the device 800. Alternatively, the time gap can be dynamically adjusted by the device 800 or the surgeon 5067 during surgery.

[0093] However, in this example, the time gap may be approximately 1 second. Time gaps much shorter and much larger than this exemplary time gap are also contemplated.

[0094] In other words, in this example, the device 800 determines a predicted appearance of the surgical scene about one second into the future, which is determined according to the acquired first image data (which shows the current appearance of the surgical scene).

[0095] <Predicted Appearance> In certain examples, the determination unit 820 may receive a predicted image of the scene from an external computing device or server, however, in other embodiments, the determination unit 820 of the device 800 uses a scene prediction algorithm to generate a prediction (i.e., predicted data or image data) of how the visual characteristics of the surgical scene will appear after a time step.

[0096] The actual form of the predicted data (which is the predicted appearance of the surgical scene) is not particularly limited and may vary depending on the context in which the embodiments of the present disclosure are applied. However, in a particular example, as described in further detail below, the predicted data is composed of the same type of data as the image data acquired by the acquisition unit 810 (i.e., if the image acquired by the acquisition unit 810 is composed of image sensor pixel values such as RGB pixel values, then such pixel values). Furthermore, although a particular example is described using predicted data including a single predicted appearance, it will be understood that multiple predictions may instead be made such that the asserted appearance data includes predicted appearances of the surgical scene at successively increasing time steps.

[0097] Advantageously, if the predicted appearance of the scene is in a similar form to the captured image data (i.e., if the predicted appearance of the scene forms a predicted image of the scene), advanced image processing techniques can be used by apparatus 800 to analyze the content of the predicted image. Analysis of the predicted image by the determination unit to determine one or more desired image capture characteristics of the image capture device is described in further detail below.

[0098] Note that in some embodiments, the predictive algorithm used by the determining unit 820 to generate the predicted appearance of a scene may include a known machine learning algorithm, such as a generative adversarial network (GAN). However, it will be appreciated that the present disclosure is not limited to GAN machine learning algorithms in particular, and any machine learning algorithm may be used as desired. These machine learning algorithms can generate realistic predictions of a scene's appearance at future times based on a database of past video and image data of similar scenarios. Thus, any new image provided to the algorithm generates a prediction of the scene's appearance in the near future (i.e., a given time step into the future from the image provided to the algorithm).

[0099] In the example of a surgical environment, the database of historical video and image data used to train the machine learning algorithm can include video and images of previous surgeries performed by a surgeon (either a computer-assisted surgery system or a human surgeon). The training database can also include video and images of previous surgeries performed by other surgeons. In some situations, the training database can also include verified photorealistic simulations of surgical scenes. These verified photorealistic simulations of surgical scenes can be generated specifically for the purpose of training the machine learning algorithm. The training database can also include depth data indicating depth information of the surgical scene. The training database can also include configuration data of the articulated arm supporting the image capture device and / or data regarding the posture of the articulated arm, such as the rotation angle of the arm's joints and the lengths of the links connected by the arm's joints. That is, the machine learning system can be trained with any surgical data obtained in a surgical scenario as needed.

[0100] In certain examples, deep learning models can be used to generate realistic predicted appearance data. These deep learning models are constructed using neural networks. These neural networks include an input layer and an output layer. Several hidden layers are located between the input layer and the output layer. Each layer includes several individual nodes. The nodes in the input layer are connected to the nodes in the first hidden layer. The nodes in the first hidden layer (and each subsequent hidden layer) are connected to the nodes in the subsequent hidden layers. The nodes in the last hidden layer are connected to the nodes in the output layer. In other words, each node in a layer is connected to all nodes in the previous layers of the neural network.

[0101] Of course, it will be appreciated that both the number of hidden layers used in the model and the number of individual nodes within each layer may vary according to the size of the training data and the individual requirements of the prediction data.

[0102] Here, each node receives a number of inputs and produces an output, with each input brought to the node (via connections to previous layers of the neural network) having a weighting factor applied to it.

[0103] In a neural network, the input layer receives several inputs, which may include surgical data obtained in a surgical scenario as described above. That is, in this example, the input layer may receive one or more of images from past surgical scenarios, verified simulations of the surgical scenario, and / or previous images of the current surgical scenario information regarding actions taken by the surgeon in the previous surgical scenario, and / or image capture characteristics of the medical image capture device used in the previous surgical scenario as input to the input layer. These inputs are then processed in the hidden layer using weights adjusted during training. The output layer then generates predictions from the neural network.

[0104] Specifically, during training, training data can be divided into inputs and targets. The input data is all data except for the targets (which are the appearances of images of surgical scenes that the model is trained to predict). The input data is then analyzed by the neural network during training to adjust the weights between each node of the neural network. In some examples, adjusting the weights during training can be achieved by a linear regression model. However, in other examples, nonlinear methods can be implemented to adjust the weights between nodes to train the neural network.

[0105] Effectively, during training, weighting factors applied to the nodes of the neural network are adjusted to determine the values of the weighting factors that produce the best match with the target data for the provided input data. That is, during training, both input and target output are provided. The network then processes the input and compares the resulting output with the target data. The differences between the output and the target data are then backpropagated through the neural network, causing the neural network to adjust the weights of each node of the neural network (backpropagation).

[0106] Of course, the number of training cycles (or epochs) used to train a model may vary depending on the situation. In some examples, the model may be continuously trained with the training data until the model produces outputs that are within a predetermined threshold of the target data.

[0107] Once trained, new input data can be provided to the input layer of the neural network, which will cause the model to generate a predicted output for the given input data (e.g., the predicted appearance of a surgical scene at a particular time step in the future) (based on the weights applied to each node of the neural network during training).

[0108] Of course, it will be appreciated that the present embodiments are not specifically limited to deep learning models (such as neural networks), and that any such machine learning algorithm may be used in accordance with embodiments of the present disclosure, depending on the circumstances.

[0109] Furthermore, the actual data used to train the machine learning algorithm is not particularly limited and may vary depending on the type of surgical scenario to which the embodiments of the present disclosure are applied.

[0110] An event recognition network can be used to automatically train a prediction network on a large pool of footage or images (although, of course, other data obtained during surgery can also be used). For example, for a given image or series of images, the prediction network can output a prediction of what comes next (i.e., how the next image in the sequence is predicted to appear). This prediction (e.g., the predicted data forming the predicted appearance of the scene) is then declared correct or incorrect by event recognition network analysis of subsequent images in the sequence. This allows machine learning systems to be automatically trained on large databases in a short amount of time, thereby improving the quality of predictions made by the machine learning system.

[0111] Here, the decision unit 820 can form a prediction of the appearance of the surgical scene at a future time point based on the occurrence of events in the training database and the current appearance of the surgical scene. That is, the decision unit 820 can learn from the training data that, for example, if a first surgical tool (such as a scalpel) is introduced into a scene, a second surgical tool (such as a suction device) is likely to be introduced into the scene in a similar location a short time after the introduction of the first surgical tool. Using this information and the current appearance of the surgical scene, the decision unit 820 can predict the appearance of the surgical scene a short time after the introduction of the scalpel.

[0112] Furthermore, in some embodiments, the determining unit 820 can be configured to determine and output a confidence value of the prediction, which may be determined for each portion of the predicted appearance of the surgical scene. Portions of the prediction with a confidence value below a predetermined threshold can then be excluded from subsequent analysis. This ensures that only regions of the surgical scene predicted with high confidence (or certainty) are included in subsequent analysis by the device 800, thus improving the accuracy of the device 800.

[0113] Of course, the predictive algorithm used by the determining unit 820 can include a generative adversarial network, although the present disclosure is not intended to be particularly limited in this respect. Alternatively, a variational autoencoder can be used to generate the predicted image in accordance with embodiments of the present disclosure. As a further alternative, an autoregressive model can be used as the predictive algorithm in accordance with embodiments of the present disclosure.

[0114] In fact, depending on the circumstances, the determination unit 820 may use any such method as needed to generate a predicted appearance of the surgical scene according to the acquired first image data.

[0115] As mentioned above, in certain exemplary embodiments, the apparatus 800 may further include an optional generation unit 820a configured to generate second image data that is a predicted appearance of the surgical scene (at a second time instance) according to the first image data.

[0116] In this regard, Figure 5C shows a timeline for generating a predicted appearance of a surgical scene at a future time point, where time is shown on the horizontal axis and increases from left to right.

[0117] First image data I1 (showing the current appearance of the surgical scene from the perspective of the endoscopic device 5001) is acquired by the device 800 at time T1. At this time (T1), the device 800 generates second image data I2 showing a predicted future appearance of the surgical scene at time T2. The future time T2 at which the prediction of the scene's appearance is made is a time Δt later than the current time T1. As mentioned above, in this example, the time difference Δt may be approximately 1 second, so that the second image data I2 shows the predicted appearance of the surgical scene at a time approximately 1 second in the future.

[0118] An example of the second image data I2 generated by the device 800 is shown in Figure 5D. This exemplary second image data I2 shows a predicted appearance of the surgical scene at a future time point T2. In this predicted image generated by the device 800, it can be seen that the surgical site 9000 from the perspective of the endoscopic device 5001 is predicted to be obscured by fog 9004 at time point T2 (i.e., a time Δt in the future from the current time point T1).

[0119] In other words, in this example, the device 800 uses the first image data to confirm the insertion of the energy treatment tool 5021 and, based on this first image data, predicts that at a future time period Δt, the surgical scene from the perspective of the endoscopic device 5001 will have the appearance of the predicted second image data I2 shown in Figure 5D. That is, due to the presence of the energy treatment tool 5021, the device 800 can predict that at a time point approximately 1 second from now, the view of the surgical site 9000 will be obscured by fog (such as mist and / or smoke).

[0120] The predicted image data I2 can be used directly to determine one or more desired image capture characteristics of the image, or generation unit 820a may first store the predicted image data in memory so that it can be retrieved later as needed.

[0121] 5 of the present disclosure, the device 800 analyzes the second image data I2 and determines that the surgical site 9000 is predicted to be obscured by fog 9004 at time T2, and therefore the fog may prevent the surgeon 5067 from observing important visual information of the surgical site at the future time T2. Accordingly, the device 800 determines the configuration of image capture characteristics of the endoscopic device 5001 required to avoid the loss of important visual information due to the predicted fog 9004.

[0122] That is, in this example, the device 800 determines that it is desirable for the endoscopic device 5001 to change from observing in the visible region of the electromagnetic spectrum to observing in the infrared region of the electromagnetic spectrum. This is because the device 800 determines that while fog obscures the visual image of the surgical site, a clear image of the surgical site can still be obtained by observing the surgical site using a different portion of the electromagnetic spectrum (such as the infrared region). Thus, the desired image capture characteristic of the endoscopic device at time T2 is for the endoscopic device 5001 to switch to capturing images in the infrared region of the electromagnetic spectrum before the occurrence of the second time instance.

[0123] <Image capture characteristics> The one or more desired image capture characteristics of the image capture device may include determining desired imaging conditions of the image capture device. The imaging conditions of the image capture device may include one or more of optical imaging system conditions and image processing conditions. For example, the optical imaging system conditions and / or image processing conditions may include one or more of desired image zoom, image focus, image aperture, image contrast, and / or image brightness. That is, the optical imaging system conditions may include optical image zoom, etc. In contrast, the image processing conditions may include digital image zoom, etc., performed by image processing circuitry at the time of image capture or applied to a captured image in post-processing. Alternatively or additionally, the one or more desired image capture characteristics of the image capture device may include a desired position or movement of the image capture device. Of course, the disclosure is not particularly limited in this respect, and any such desired image capture characteristics may be determined by the determination unit as appropriate. As a further alternative, as described above, the image capture characteristics may include the type of image capture (such as whether a visual image of the scene is captured or a hyperspectral image (using information from the entire electromagnetic spectrum) of the scene is captured).

[0124] Here, in an embodiment of the present disclosure, the determination unit 820 of the device 800 can use one or more camera characteristic algorithms to determine optimal one or more image capture characteristics of an image capture device (such as the endoscopic device 5001) with respect to the predicted appearance of the surgical scene.

[0125] The one or more image capture characteristic algorithms used to determine the one or more image capture characteristics may, in some examples, comprise a machine learning system trained with input data including past surgical footage, validated simulations, data measured during a surgeon's surgery (such as surgical tool and camera position information, an environment map (which may be generated by simultaneous localization and mapping (SLAM)), and tool type information). That is, a machine learning algorithm trained with an appropriate training set can be used to determine desired image capture characteristics for an image capture device.

[0126] Additionally, deep learning algorithms, including neural networks (such as those described in detail above), may be used in accordance with embodiments of the present disclosure to determine one or more image capture characteristics of an image capture device.

[0127] Additionally, machine learning algorithms can be trained using labeled (either manually or automatically) data. The labeled data can include assessments of the operation, navigation, or control of the imaging device during the surgical workflow. Additionally, the labeled data can include assessments of the visibility of important features, such as active tools and events (e.g., bleeding) within the surgical scene. This labeling can be performed by the surgeon 5067 or other medical staff. Additionally, the labeled data can include other objectives related to the usability of the image for the human visual system, such as viewpoint stability, consistent orientation, and lighting. In this way, the machine learning algorithm can learn to determine desired image capture characteristics optimized for use by the surgeon 5067.

[0128] Optionally, the algorithm for determining the desired image capture characteristics may be a rules-based algorithm, where the predicted appearance of the surgical scene (e.g., predicted image data) can be combined with current image data and contextual data and analyzed to detect the occurrence of known scenarios.

[0129] A lookup table of ideal camera characteristics and behaviors for the detected scenarios can then be used to determine the desired image capture characteristics. For a surgical scene, some example scenarios (or events) that may be detected in the predicted image data, and the resulting desired image capture characteristics, are as follows:

[0130] 1. Moving a tool or object in the scene Movement of tools or objects within the scene can cause obstructions to the view. For example, movement of a tool (such as a suction tool) can obscure or partially obscure the image from the perspective of the image capture device within the scene. According to embodiments of the present disclosure, given the position of the tool in the predicted image data, a desired movement of the imaging device can be calculated that avoids the obstruction and maintains visibility of the surgical scene and tool.

[0131] 2. Moving tools that are actively used The future movement of the tool in the predicted image data can be used to determine the desired movement of the imaging device to keep the tool at the center of the captured image in an automatic camera tracking function. The state of the tool (in use or not) can be detected based on data indicative of the state and changed by the tool's activation switch, etc.

[0132] 3. Organ retention / slippage The predicted image data can depict events such as the organ being held by the retractor dropping or moving. The organ movement can be tracked in the image, and a desired movement of the imaging device can be selected that keeps both the retractor and the organ in view. Alternatively, the organ can be kept in view by changing the level of zoom used by the image capture device.

[0133] 4. Anticipated bleeding events A desired movement of the imaging device can be selected that keeps the location of the bleeding source as close to the center of the captured image as possible while keeping other important image features, such as the surgical tool, in view. This can be achieved by a combination of lateral and angular changes in camera position.

[0134] 5. Tool change Desired image capture characteristics can be selected when a tool change occurs (i.e., when a used tool is removed or a new tool is introduced into the scene). The desired image capture characteristics are determined to provide visibility of the area where the tool will be introduced while maintaining visibility of other important features in the predicted image data. For example, a camera angle can be selected that provides visibility of the area below the trocar entrance into the patient's body cavity while maintaining visibility of the surgical scene.

[0135] 6. Change in direction of contraction pull (with non-dominant hand) There will be a change in the direction / position of the ablation / resection, and the tool will move into an area where tension is created by holding the tool with the non-dominant hand. The desired image capture characteristics of the image capture device can be determined to account for this change.

[0136] It will be appreciated that the present disclosure is not intended to be limited to these above-described examples, and that there are numerous other surgical scenarios that may be used in a lookup table to determine one or more desired image capture characteristics of an image capture device in accordance with embodiments of the present disclosure.

[0137] Although the determination unit has been described using both machine learning algorithms and look-up tables, it will be appreciated that any suitable method may be used by determination unit 820, as appropriate, to determine desired image capture characteristics of the image capture device. Those skilled in the art will be able to apply existing techniques for automatic camera position systems to the predicted image data generated by generation unit 820a to determine desired image capture characteristics, such as, for example, a desired movement or a desired image capture position of the image capture device.

[0138] It will be appreciated that there may be situations in which the predicted image data includes multiple scenarios or events. That is, the predicted image data may include several likely predictions that may in fact require very different image capture characteristics. Thus, in certain instances, it may be desirable for the camera or image capture device to use image capture characteristics that capture all possible outcomes, or to use capture characteristics that allow the camera to quickly assume an ideal viewpoint for each scenario as soon as its occurrence is confirmed. For example, this may be a midpoint between two desired viewpoints for different scenarios.

[0139] Furthermore, in particular examples, the determination unit 820 can be configured to calculate weights for the image capture characteristics of the image capture device according to one or more factors related to the image capture characteristics, and determine the image capture characteristic with the largest weight coefficient as the desired image capture characteristic of the image capture device. That is, a range of image capture characteristics (such as a range of viewing positions) can be generated, and each of these viewing positions can be weighted according to factors such as its advantage (e.g., how well the surgeon can view the target area of the surgical scene at that position, or what percentage of the image the target area of the surgical scene occupies). The image capture characteristic (or, in this example, the position) with the largest calculated weight is then considered the most advantageous image capture characteristic and selected as the desired image capture characteristic of the surgical scene.

[0140] Alternatively, the weighting of the image capture characteristics of the image capture device is based on a comparison of the image capture characteristics of the image capture device with a set of target image capture characteristics for the image capture device, the closer the image capture characteristics of the image capture device are to the target image capture characteristics, the higher the weighting factor.

[0141] Control unit: Once the determination unit determines the desired image capture characteristics of the image capture device, the control unit is configured to control the image capture device in accordance with the one or more desired image capture characteristics of the image capture device at a third time instance between the first time instance and the second time instance.

[0142] Thus, in this example, the device 800 controls the endoscopic device 5001 at a third time instance between the first time instance and the second time instance according to one or more desired image capture characteristics of the endoscopic device (i.e., the endoscopic device captures images using the infrared portion of the electromagnetic spectrum).

[0143] Returning to FIG. 5C , a third time instance T3 (control time point) is shown on the time chart between the current time point T1 and the second time point T2 (when fog is predicted to obscure the image). Accordingly, the device 800 controls the endoscopic device so that, at time point T3, the endoscopic device switches to capturing images using the infrared portion of the electromagnetic spectrum. The actual location of time point T3 on the time chart is not particularly limited, so long as it is between the current time point T1 and the predicted time point T2, and is sufficiently prior to time point T2 so that the device 800 can adjust the image capture characteristics of the endoscopic device to correlate with the determined image capture characteristics before reaching the second time point T2.

[0144] Thus, in this example, when the second time point T2 is reached (the surgeon 5067 begins using the energy treatment tool 5021 as expected), the device 800 has already controlled the endoscopic device 5001 so that the endoscopic device 5001 captures images in the infrared region of the electromagnetic spectrum rather than the visible region of the electromagnetic spectrum. Thus, at time point T2, the actual image captured by the endoscopic device 5001 shows a clear image of the surgical site 5001, despite the presence of mist and / or smoke caused by operation of the energy treatment tool 5021. A diagram of the actual image acquired by the endoscopic device 5001 at time point T2 (i.e., after controlling the image capture characteristics of the endoscopic device 5001) is shown in FIG. 5E.

[0145] In some embodiments, the control unit is configured to compare current image capture characteristics of the image capture device (such as the endoscope 5001) with desired image capture characteristics of the image capture device, and then use the comparison of these characteristics to generate image controller instructions that cause the image capture device to achieve the desired image capture characteristics at a desired time. For example, by comparing the current position of the image capture device with the desired position of the image capture device, the control unit can determine corresponding actuation instructions that can be used to move the image capture device to the desired position.

[0146] Favorable effects: According to embodiments of the present disclosure, an apparatus for controlling image capture devices during surgery enables a computer-assisted surgery system to predictively optimize the image capture characteristics of a camera, such that the effects of foreseeable future changes in the surgical scene can be addressed by adjusting the image capture characteristics and / or configuring the image capture device before the predicted changes actually occur in the surgical scene. In this way, adverse effects on imaging of the surgical scene (e.g., preventing the provision of important image information to the surgeon) can be avoided without delaying the surgery.

[0147] Of course, the present disclosure is not specifically limited to these advantageous technical effects, and other effects may exist as will be apparent to those skilled in the art upon reviewing the present disclosure.

[0148] Further changes: It will be understood that the image capture device may include any medical image capture device as needed under the circumstances. That is, while the configuration of device 800 is described above with reference to Figures 5-6 of the present disclosure, it will be understood that embodiments of the present disclosure are not limited to this particular example. For example, while embodiments of the present disclosure have been described with reference to an endoscopic imaging device, embodiments of the present disclosure are also applicable to telescopic imaging devices, microscopic imaging devices, exoscopic imaging devices, etc., as needed. Additionally, some further modifications to the configuration of the device are described below.

[0149] Context Detection System: It will be appreciated that while the above-described method used by the determination unit to generate a predicted appearance of a scene utilizes acquired image data to generate the prediction, the present disclosure is not particularly limited in this respect. Rather, certain additional information may be used by apparatus 800 in generating the predicted data and determining one or more desired image capture characteristics of the image capture device. This additional information may, in certain examples, provide contextual information that further enhances the predictive capabilities of apparatus 800.

[0150] In some examples, context information can be provided to the generation unit to assist in generating predicted images of a scene. For example, a scene prediction algorithm can rely on cues from the context information to generate a depiction of scene changes, and multiple data sources can be used to generate any particular example of predicted image data. Alternatively, a context detection system can be configured to perform an analysis of the context information, which is used by the decision unit 820 when determining desired image capture characteristics of the image capture device. As a further alternative, the context information can be provided directly to the decision unit, which incorporates the context information into its analysis.

[0151] Therefore, optionally, according to an embodiment of the present disclosure, the acquisition unit 810 may include a context detection system 850 configured to collect additional information regarding the surgical context that may be related to changes occurring in the surgical scene, as shown in FIG. 6A of the present disclosure.

[0152] The context sensing system may include a number of context sensing means (including various separate cameras and sensors) configured to collect context information about a procedure being performed by a surgeon. The context information obtained by the context sensing means may include at least one of the following: positions of objects in a scene, movements of objects in a scene, types of objects present in a scene, and / or actions being performed by people in a scene.

[0153] However, the configuration of the context detection system 850 is not particularly limited and may vary depending on the particular situation in which an embodiment of the present disclosure is applied.

[0154] Some specific examples of context information that can be obtained by the context detection means of the context detection system 850 are provided below.

[0155] In some examples, the context sensing system 850 can include cameras and / or microphones within the operating room and outside the patient's body. The cameras and / or microphones monitor events occurring within the operating room and outside the patient's body. Audio recordings from the microphones can enable the context sensing system to monitor conversations within the operating room (including commands from the surgeon, such as "hand over the forceps" or "there's bleeding"). Alternatively, images from the context sensing system's cameras can enable the context sensing system to determine the orientation of the operating table, room lighting, the relative positions of staff within the operating room, etc. Alternatively, data from the microphones and cameras can be used by the context sensing unit to determine the state of staff within the operating room, such as stress levels or current task levels for the surgical staff.

[0156] Thus, by using a camera and / or microphone to monitor the environment outside the patient's body, the device 800 can gain deeper contextual awareness of the progress of the surgery, which can improve both the predictive capabilities of the generation unit 820a and the selection of image capture characteristics by the decision unit 820.

[0157] Alternatively or additionally, the context sensing system may include patient sensors, such as blood pressure, respiration, and heart rate sensors. These sensors may provide additional contextual information regarding the patient's condition during surgery. For example, a drop in the patient's blood pressure may signal the occurrence of bleeding, and this information may be used by generating unit 820a and determining unit 820b when generating predicted images and desired image capture characteristics to further improve device 800's ability to maintain a clear image of the surgical scene. Alternatively, contextual information regarding rhythms associated with tissue motion (e.g., heart rate), respiratory cycles, pneumoperitoneum pressure, etc. may be monitored by separate sensors in context sensing system 850 to further improve device 800's determination of the image capture characteristics of the image capture device.

[0158] Alternatively, or in addition, a tool that communicates the current tool's operating state or other parameters can be coupled to the context sensing system. For example, the context sensing system can be configured to receive status updates from a surgical tool (such as the energy treatment tool 5021). These status updates, which may be received via a wired or wireless interface, can inform the context sensing system whether the tool is active or inactive. In the example of an energy treatment tool, the context sensing system 850 can know that when the energy treatment tool is active, there is a likelihood of an increased amount of mist and / or smoke within the patient's body cavity. Thus, the device 800 can determine desired image capture characteristics of the image capture device accordingly.

[0159] Alternatively, the context system 850 may include one or more sensors and / or circuitry configured to determine which tools an assisting surgeon has ready for use in the next stage of surgery, the number of tool changes that have occurred within a particular time period, tool type information (including information about how the tool is manipulated by the surgeon), tool movement information such as tool speed and trajectory, information about how the surgeon or assisting staff is holding the tool, information about tool status and operational settings that may be related to an impending visual change (e.g., amount of suction / irrigation / aspiration), etc.

[0160] This additional context information can be used to assist apparatus 800 in determining the image capture characteristics of the image capture device. For example, the context detection unit can use the information received from the tool as cues that are used to improve the accuracy of the predicted image of the scene generated by generation unit 820a.

[0161] Alternatively, or in addition, the context detection system 850 can be configured to receive manual input from a surgeon or other medical staff. The manual input can be received by the context detection system 850, for example, via operation of a touchscreen device or a computer keyboard. This information can be used to inform the device 800 of the current stage of the surgery (e.g., "entering stage 2"). Determining the stage of the surgery in this manner aids the device 800 in determining the image capture characteristics of the image capture device. For example, by knowing which stage of the surgery has begun, the device 800 can more accurately determine and predict which tools are likely to be introduced into or removed from the surgical scene.

[0162] Virtual Camera Unit: In some examples, device 800 may further include a virtual viewpoint system 860. The virtual viewpoint system 860 may be used to generate a virtual viewpoint having a coordinate position in 3D space (i.e., a virtual camera position) that may be separate from the camera position. In other words, a synthetic virtual viewpoint is generated through images taken during movement of the image capture device, where the virtual viewpoint is separate and offset from the actual position of the image capture device at a given time instance. To generate the virtual viewpoint, the virtual viewpoint system 860 is configured to combine image data from the movement of the image capture device using image stitching and viewpoint virtualization algorithms known in the art.

[0163] In this way, a human controller controls the virtual camera position while the virtual viewpoint system 860 can operate autonomously to control the actual camera position of the image capture device. In some examples, the virtual camera position can be controlled completely autonomously, allowing data to be collected rapidly using autonomous control of the image capture device while meeting the image stability goals of the human visual system.

[0164] In instances where a virtual camera viewpoint is used, an improved virtual camera viewpoint that is favorable for viewing the scene by the surgeon can be predicted by the device 800. Furthermore, the actual camera movement required to generate the virtual viewpoint can be determined by the determination unit 820 as one or more desired image capture characteristics of the image capture device.

[0165] Ability Detection Unit: In some examples, device 800 may further include a capabilities detection unit 870. The capabilities detection unit 870 may be configured to interact with the determination unit 820 and the control unit 830 to determine optimal image capture characteristics that can be achieved for a given set of desired image capture characteristics. That is, the capabilities detection unit 870 may determine that one or more limitations or limitations of the image capture device or the surgical environment prevent the image capture device from achieving the desired image capture characteristics determined by the determination unit 820 within the required time frame. In this case, the capabilities detection unit 870 indicates to the control unit 830 the optimal image capture characteristics achievable for the image capture device. In other words, the capabilities detection unit takes into account the limitations of the image capture device when determining the desired / optimal image capture characteristics of the image capture device.

[0166] Consider an example where the desired image capture characteristic of an image capture device is a desired image capture position. This is shown in FIG. 6B of the present disclosure. In this example, device 800 has generated a desired position L1 of the image capture device for taking an image at a future time T. Once the desired position has been determined (by determining unit 820), capability detecting unit 870 evaluates the capabilities of the image capture device (including the current position L0 of the image capture device and the maximum velocity V of the image capture device) to determine whether the image capture device can achieve the desired position L1 by time T (a time point Δt in the future).

[0167] However, in this example, the capability detection unit determines that the image capture device will not achieve position L1 by time T. Rather, based on the capability of the image capture device, the image capture device will only achieve position L2 by time T. Therefore, the capability detection unit determines that the image capture device will not be able to achieve position L1 by time T.

[0168] However, position L2 is not the optimal position at time T compared to position L1. That is, in contrast to L1, position L2 may not provide a clear view of the scene. Therefore, the capability detection unit interacts with the determination unit 820 to determine the optimal position of the image capture device within the range of movement of the image capture device that can be achieved within time step Δt. Based on this, position L3 is determined to be the optimal position of the image capture device that can be achieved within time step Δt (i.e., that can be achieved by time T).

[0169] Therefore, since it is determined that the image capture device cannot achieve the desired position L1 due to limitations in the image capture device's capabilities, a second desired position L3 that is within the capabilities of the image capture device is determined, thereby avoiding a situation in which the image capture device reaches a suboptimal position L2 due to its inability to achieve the desired position L1 in the required time.

[0170] It will be further appreciated that the capability sensing unit may take into account other constraints, such as environmental constraints, when determining whether the image capture device can achieve the desired image capture characteristics. For example, the capability sensing unit may determine that the image capture device cannot achieve the desired image capture position due to a potential collision with the patient's tissue, a surgeon's surgical tools, etc.

[0171] Optionally, the capability detection unit 870 can be implemented according to one or more rules of a surgical scenario. That is, the device can recognize a particular surgical scenario (such as cardiac surgery) through acquired image data and / or acquired context data. Recognition of the surgical scenario can then be used to query a lookup table of predetermined constraints that can be applied to possible movements or desired image capture characteristics of the image capture device. For example, in cardiac surgery, constraints may be that a particular portion of the heart is kept within the field of view of the image capture device, or that the position of the image capture device does not obstruct the surgeon's ability to access a particular portion of the heart.

[0172] In example embodiments of the present disclosure, the capability sensing means 870 and / or the control unit 830 may be configured to determine a movement pattern to a desired image capture location according to the location of one or more objects present in the scene, thereby ensuring that the image capture device does not collide with objects in the scene on its way to the desired location.

[0173] Specifically, in some embodiments, once a movement pattern to a desired image capture position is determined, the control unit is configured to control the position and / or orientation of an articulated arm supporting the image capture device according to the determined movement pattern in order to efficiently move the image capture device to the desired position without adversely affecting the surgical scene (e.g., by colliding with objects within the scene) on the path to the desired position.

[0174] Of course, although the capabilities of the image capture device have been described with respect to desired locations, it will be appreciated that the capabilities sensing unit may be applied to other desired image capture characteristics of the image capture device as desired.

[0175] Exemplary configuration: An exemplary configuration of a computer-assisted surgery system according to an embodiment of the present disclosure is described with reference to Figure 7A of the present disclosure, which may be used, for example, in the context of endoscopic surgery (as described with reference to Figure 1 of the present disclosure).

[0176] In this exemplary configuration, a robotic camera system 1100 is provided. The robotic camera system includes an image capture device 1102 and a drive system 1104. The image capture device is coupled to the drive system by a support arm 1106. In other words, the drive system 1104 is used to move the support arm 1106 to position the image capture device 1102 so that the image capture device captures an image of a surgical scene.

[0177] An image acquisition unit (not shown) acquires image data from an image capture device and provides the image data to a scene prediction unit (such as the generation unit 820a of the device 800). Furthermore, a context detection unit 1100 acquires context information of the surgical scene and provides the context information to the scene prediction unit 1108.

[0178] The scene prediction unit 1108 then generates, based on this information, predicted image data representing the appearance of the surgical scene at a future time T (i.e., at a time T prior to the time the image data was acquired by the image capture device of the robotic camera system).

[0179] This predicted image data is provided to a camera positioning unit 1112 (such as determination unit 820 of device 800), which also receives a virtual camera position from a virtual viewpoint system 1114 (generated based on image data received from image capture device 1100).

[0180] Based on this information (as well as the current position of the support arm of the robotic camera system 1100), the camera positioning unit 1112 calculates the desired movement of the imaging device, which is provided to a drive design unit 1116 (such as the control unit 830 of the device 800) which designs a series of drive commands to be provided to the drive unit of the robotic camera system to move the robotic camera system to the desired position before time T.

[0181] In this manner, the position of the robotic camera system can be controlled so that the robotic camera system achieves a desired position before time T. This allows the computer-assisted surgery system to predictively optimize the camera position such that the effects of foreseeable future changes in the surgical scene are compensated for by moving the camera system before the changes occur.

[0182] An exemplary illustration of the application of this computer-assisted surgery system to a surgical scenario is shown with reference to Figure 7B. In Figure 7B, a surgeon 1120 is performing a computer-assisted surgical procedure on a target area 1122 of a patient 1124. The surgeon 1120 observes images of the surgical scene captured by a robotic camera system 1126 on a display (not shown). The field of view of the robotic camera system is indicated by the area surrounded by line 1128 in Figure 7B. Images from the robotic camera system 1126 are also provided to a scene prediction unit 1108, as shown in Figure 7A.

[0183] In this example, consider that the surgeon 1120 has just introduced a scalpel 1130 into the surgical scene. At this stage, the scene prediction unit (which may be the generation unit of the apparatus 800) predicts that in a subsequent video frame, within a time period of 2 seconds, the surgeon 1120 will introduce a suction tube 1132 into the surgical scene. This is shown in Figure 7C. Analysis of the predicted image reveals that the placement of the suction tube 1132 in the predicted surgical scene will occlude the robotic camera system 1126, thus preventing a clear image of the scene.

[0184] Thus, given the time step of the predicted image (i.e., 2 seconds ahead of the current time), the camera positioning unit 1112 of Figure 7A calculates the possible movements within the three-dimensional volume surrounding the robotic camera system 1126 that can be achieved by the robotic camera system 1126 and that will provide a clear image of the target area 1122. This is shown in Figure 7D.

[0185] Finally, once the desired image capture position of the robotic camera system is determined, the drive design unit 1116 and the drive unit 1104 interact to control the robotic camera system 1126 so that it adjusts its position and angle to maintain a clear view of the target area 1122 before the surgeon 1120 actually places the suction tube 1132 into the surgical scene. This is shown in Figure 7E.

[0186] method: According to an embodiment of the present disclosure, there is provided a method for controlling an image capture device during surgery. An illustration of a method for controlling an image capture device during surgery according to an embodiment of the present disclosure is shown in FIG.

[0187] A method for controlling an image capture device, such as a medical image capture device, during surgery begins in step S1200 and proceeds to step S1202.

[0188] In step S1202, the method includes obtaining first image data from a medical image capture device, the first image data being an appearance of a surgical scene at a first time instance.

[0189] Once the first image data is acquired, the method proceeds to step S1204.

[0190] In step S1204, the method includes determining one or more desired image capture characteristics of the medical image capture device based on a predicted appearance of the surgical scene at a second time instance after the first time instance based on the first image data.

[0191] Once one or more desired image capture characteristics of the image capture device have been determined, the method proceeds to step S1206.

[0192] In step S1206, the method includes controlling the image capture device according to one or more desired image capture characteristics of the medical image capture device at a third time instance between the first time instance and the second time instance.

[0193] Once the image capture device has been controlled at the third time instance, the method proceeds to step S1208, where it ends.

[0194] It will be appreciated that in some circumstances, the desired image capture characteristics of the image capture device may be image capture characteristics that the image capture device already has, in which case no changes are made to the current image capture characteristics of the image capture device.

[0195] It will be further understood that in some circumstances, upon completion of step S1206, the method returns to step S1202. In this manner, the desired image capture characteristics of the image capture device are continuously or periodically evaluated and updated as necessary.

[0196] Computing equipment: Referring now to FIG. 9 , a computing device 1300 according to an embodiment of the present disclosure is shown. The computing device 1300 may be a computing device for controlling an image capture device during surgery. Typically, the computing device may be a device such as a personal computer or a terminal connected to a server. Indeed, in some embodiments, the computing device may be a server. The computing device 1300 is controlled using a microprocessor or other processing circuitry 1302.

[0197] The processing circuit 1302 may be a microprocessor that executes computer instructions or may be an application specific integrated circuit. The computer instructions are stored on a storage medium 1304, which may be a magnetically readable medium, an optically readable medium, or a solid-state circuit. The storage medium 1304 may be integrated into the computing device 1300 (as shown) or may be separate from the computing device 1300 and connected to the computing device 1300 using either a wired or wireless connection. The computer instructions may be embodied as computer software including computer readable code that, when loaded into the processor circuit 1302, configures the processor circuit 1302 of the computing device 1300 to perform a method for controlling an image capture device intraoperatively according to an embodiment of the present disclosure.

[0198] A user input (not shown) is also connected to the processor circuitry 1302. The user input may be a touch screen, or a mouse or stylist-type input device. The user input may also be a keyboard or any combination of these devices.

[0199] A network connection 1306 is further coupled to the processor circuit 1302. The network connection 1306 may be a connection to a local area network or a wide area network such as the Internet or a virtual private network. The network connection 1306 may be connected to a medical device infrastructure that enables the processor circuit 1302 to communicate with other medical devices to obtain or provide relevant data to other medical devices. The network connection 1306 may be located behind a firewall or some other form of network security.

[0200] Additionally, coupled to the processing circuit 1302 is a display device 1308. While the display device is shown integrated into the computing device 1300, it may also be separate from the computing device 1300 and may be a monitor or some other device that allows a user to visualize the operation of the system. Additionally, the display device 1300 may be a printer or some other device that allows relevant information generated by the computing device 1300 to be viewed by the user or a third party (such as a medical support assistant).

[0201] While the above has been described in the context of a "master-slave" robotic system, the disclosure is not so limited. In some cases, a surgical robot can work independently of a human surgeon, with the human surgeon present in an overseeing role. Furthermore, in an endoscopy or laparoscopy, the scopist may be the robot, with the human surgeon supervising the robot. In some embodiments, the robotic system may be a multi-robot surgical system in which a primary surgeon uses the robotic arm and an assistant surgeon teleoperates an auxiliary robotic arm. The robotic system may also be a single surgical system consisting of a pair of cooperating autonomous robotic arms holding surgical instruments. In this case, the human surgeon may use a master-slave configuration.

[0202] Exemplary Systems FIG. 10 schematically illustrates an example of a computer-assisted surgery system 11260 to which the present technology can be applied. The computer-assisted surgery system is a master-slave system incorporating an autonomous arm 11000 and one or more surgeon-controlled arms 11010. The autonomous arm holds an imaging device 11020 (e.g., a medical scope such as an endoscope, microscope, or exoscope). Each of the one or more surgeon-controlled arms 11010 holds a surgical device 11030 (e.g., a cutting tool, etc.). The imaging device of the autonomous arm outputs an image of the surgical scene to an electronic display 11100 viewable by the surgeon. The autonomous arm autonomously adjusts the field of view of the imaging device as the surgeon performs surgery using the one or more surgeon-controlled arms, allowing the surgeon to properly view the surgical scene in real time.

[0203] The surgeon controls one or more surgeon-controlled arms 11010 using a master console 11040. The master console includes a master controller 11050. The master controller 11050 includes one or more force sensors 11060 (e.g., torque sensors), one or more rotation sensors 11070 (e.g., encoders), and one or more actuators 11080. The master console includes an arm (not shown) with one or more joints and a manipulator. The surgeon can grasp and move the manipulator to move the arm about one or more joints. One or more force sensors 11060 detect the force the surgeon applies to the manipulator of the arm about one or more joints. One or more rotation sensors detect the angle of rotation of one or more joints of the arm. The actuators 11080 drive the arm about one or more joints, allowing the arm to provide tactile feedback to the surgeon. The master console includes a natural user interface (NUI) input / output for receiving input information from the surgeon and providing output information to the surgeon. The NUI input / output includes an arm that the surgeon moves to provide input information and also provides haptic feedback to the surgeon as output information. The NUI input may also include voice input, eye gaze input, and / or gesture input. The master console includes an electronic display 11100 for outputting images captured by the imaging device 11020.

[0204] The master console 11040 communicates with the autonomous arm 11000 and each of the one or more surgeon-controlled arms 11010 via a robotic control system 11110. The robotic control system is connected to the master console 11040, the autonomous arm 11000, and the one or more surgeon-controlled arms 11010 by wired or wireless connections 11230, 11240, and 11250. The connections 11230, 11240, and 11250 allow for the exchange of wired or wireless signals between the master console, the autonomous arm, and the one or more surgeon-controlled arms.

[0205] The robotic control system includes a control processor 11120 and a database 11130. The control processor 11120 processes signals received from one or more force sensors 11060 and one or more rotation sensors 11070 and outputs control signals in response to which one or more actuators 11160 drive one or more surgeon control arms 11010. In this manner, movement of the controls of the master console 11040 causes corresponding movement of one or more surgeon control arms.

[0206] Additionally, the control processor 11120 outputs control signals, and in response to the control signals, one or more actuators 11160 drive the autonomous arm 11000. The control signals output to the autonomous arm are determined by the control processor 11120 in response to signals received from one or more of the master console 11040, the one or more surgeon-controlled arms 11010, the autonomous arm 11000, and any other signal sources (not shown). The received signals are signals indicative of an appropriate position of the autonomous arm for capturing an image of an appropriate field of view by the imaging device 11020. The database 11130 stores the values of the received signals and the corresponding positions of the autonomous arm.

[0207] For example, for a given combination of values of signals received from one or more force sensors 11060 and rotation sensors 11070 of the master controller (which in turn represent corresponding movements of one or more surgeon-controlled arms 11010), the corresponding position of the autonomous arm 11000 is set so that images captured by the imaging device 11020 are not obstructed by one or more surgeon-controlled arms 11010.

[0208] As another example, if the signal output by one or more force sensors 11170 (e.g., torque sensors) of the autonomous arm indicates that the autonomous arm is facing resistance (e.g., due to an obstacle in the path of the autonomous arm), the corresponding position of the autonomous arm is configured such that an image is acquired by the imaging device 11020 from a different field of view (e.g., a field of view in which the autonomous arm can move along a different path that does not include the obstacle).

[0209] It will be appreciated that there may be other types of received signals that indicate the approximate position of the autonomous arm.

[0210] The control processor 11120 looks up the value of the received signal in a database 11130 to obtain information indicative of the corresponding position of the autonomous arm 11000. This information is then processed to generate a further signal in response to which the actuator 11160 of the autonomous arm moves the autonomous arm to the instructed position.

[0211] The autonomous arm 11000 and the one or more surgeon-controlled arms 11010 each include an arm unit 11140. The arm unit includes an arm (not shown), a control unit 11150, one or more actuators 11160, and one or more force sensors 11170 (e.g., torque sensors). The arm includes one or more links and joints that enable movement of the arm. The control unit 11150 sends signals to and receives signals from the robot control system 11110.

[0212] In response to signals received from the robotic control system, the control unit 11150 controls one or more actuators 11160 to drive the arm about one or more joints to move it into the appropriate position. For one or more surgeon-controlled arms 11010, the received signals are generated by the robotic control system based on signals received from the master console 11040 (e.g., by a surgeon controlling the arm on the master console). For the autonomous arm 11000, the received signals are generated by the robotic control system looking up appropriate autonomous arm position information in a database 11130.

[0213] In response to signals output by one or more force sensors 11170 about one or more joints, the control unit 11150 outputs signals to the robotic control system. For example, this can cause the robotic control system to send signals to the master console 11040 representing the resistance faced by one or more surgeon-controlled arms 11010 and provide corresponding haptic feedback to the surgeon (e.g., such that the resistance faced by one or more surgeon-controlled arms causes the master console's actuators 11080 to create corresponding resistance in the master console's arms). As another example, the robotic control system can search the database 11130 for appropriate autonomous arm position information (e.g., to find alternative positions for the autonomous arm if one or more force sensors 11170 indicate that an obstacle is present in the path of the autonomous arm).

[0214] The imaging device 11020 of the autonomous arm 11000 includes a camera control unit 11180 and an imaging unit 11190. The camera control unit controls the imaging unit to capture images and controls various parameters of the captured images, such as zoom level, exposure value, white balance, etc. The imaging unit captures images of the surgical scene. The imaging unit includes all components necessary for capturing images, including one or more lenses and an image sensor (not shown). The field of view of the surgical scene from which an image is captured depends on the position of the autonomous arm.

[0215] The surgical device 11030 of one or more surgeon-controlled arms includes a device control unit 11200, a manipulator 11210 (eg, including one or more motors and / or actuators), and one or more force sensors 11220 (eg, torque sensors).

[0216] The device control unit 11200 controls the manipulator to perform a physical action (e.g., a cutting action if the surgical device 11030 is a cutting tool) in response to signals received from the robotic control system 11110. The signals are generated by the robotic control system in response to signals received from the master console 11040, which are generated by the surgeon entering information into the NUI input / output 11090 to control the surgical device. For example, the NUI input / output may include one or more buttons or levers included as part of the controls on the arm of the master console that the surgeon can manipulate to cause the surgical device to perform a predetermined action (e.g., turning on or off an electric blade if the surgical device is a cutting tool).

[0217] Additionally, the device control unit 11200 receives signals from one or more force sensors 11220. In response to the received signals, the device control unit provides a corresponding signal to the robotic control system 11110, which in turn provides a corresponding signal to the master console 11040. The master console provides tactile feedback to the surgeon via the NUI input / output 11090. Thus, the surgeon receives tactile feedback from the surgical device 11030 as well as from one or more surgeon control arms 11010. For example, if the surgical device is a cutting tool, the tactile feedback may include a button or lever operating the cutting tool providing more resistance to manipulation when signals from the one or more force sensors 11220 indicate a greater force being applied to the cutting tool (e.g., occurring when cutting hard material such as bone), and less resistance to manipulation when signals from the one or more force sensors 11220 indicate a lesser force being applied to the cutting tool (e.g., occurring when cutting softer material such as muscle). The NUI input / output 11090 includes one or more suitable motors, actuators, etc. for providing haptic feedback in response to signals received from the robotic control system 11110.

[0218] 11 shows a schematic diagram of another example of a computer-assisted surgery system 12090 to which the present technology can be applied. The computer-assisted surgery system 12090 is a surgical system in which a surgeon performs tasks via a master-slave system 11260 and a computerized surgical device 12000 performs the tasks autonomously.

[0219] The master-slave system 11260 is the same as in Figure 10 and will not be described further, however, the system may be different from that of Figure 10 in alternative embodiments, or may be omitted entirely (in which case the system 12090 operates autonomously while the surgeon performs conventional surgery).

[0220] The computerized surgical device 12000 includes a robotic control system 12010 and a tool holder arm device 12100. The tool holder arm device 12100 includes an arm unit 12040 and a surgical device 12080. The arm unit includes an arm (not shown), a control unit 12050, one or more actuators 12060, and one or more force sensors 12070 (e.g., torque sensors). The arm includes one or more joints that enable movement of the arm. The tool holder arm device 12100 sends and receives signals to the robotic control system 12010 via a wired or wireless connection 12110. The robotic control system 12010 includes a control processor 12020 and a database 12030. Although shown as separate robotic control systems, the robotic control system 12010 and the robotic control system 11110 may be one and the same robotic control system. Surgical device 12080 has the same components as surgical device 11030, which are not shown in FIG.

[0221] In response to control signals received from the robotic control system 12010, the control unit 12050 controls one or more actuators 12060 to drive the arm about one or more joints to move it into an appropriate position. Furthermore, the operation of the surgical device 12080 is controlled by control signals received from the robotic control system 12010. The control signals are generated by the control processor 12020 in response to signals received from one or more of the arm unit 12040, the surgical device 12080, and any other signal sources (not shown). The other signal sources may include an imaging device (e.g., the imaging device 11020 of the master-slave system 11260) that captures images of the surgical scene. The signal values received by the control processor 12020 are compared with signal values stored in the database 12030 along with corresponding arm position and / or surgical device operating state information. The control processor 12020 retrieves the arm position and / or surgical device operating state information associated with the received signal values from the database 12030. The control processor 12020 then uses the retrieved arm position and / or surgical device operating state information to generate control signals that are sent to the control unit 12050 and the surgical device 12080.

[0222] For example, if a signal received from an imaging device capturing images of a surgical scene indicates a predetermined surgical scenario (e.g., via a neural network image classification process, etc.), the predetermined surgical scenario is looked up in database 12030, and arm position information and / or surgical device motion state information associated with the predetermined surgical scenario is retrieved from the database. As another example, if the signal indicates a resistance value measured by one or more force sensors 12070 about one or more joints of arm unit 12040, the resistance value is looked up in database 12030, and arm position information and / or surgical device motion state information associated with the resistance value is retrieved from the database (e.g., to allow the arm to be repositioned to an alternate position if resistance increases in response to an obstacle in the arm's path). In either case, the control processor 12020 then sends a signal to the control unit 12050 to control one or more actuators 12060 to change the position of the arm to the position indicated by the retrieved arm position information, and / or sends a signal to the surgical device 12080 to control the surgical device 12080 to enter the operating state indicated by the retrieved operating state information (e.g., turning the electric blade to the "on" or "off" state if the surgical device 12080 is a cutting tool).

[0223] FIG. 12 schematically illustrates another example of a computer-assisted surgery system 13000 to which the present technology can be applied. The computer-assisted surgery system 13000 is a computer-assisted medical scope system in which an autonomous arm 11000 holds an imaging device 11020 (e.g., a medical scope such as an endoscope, microscope, or exoscope). The imaging device of the autonomous arm outputs an image of the surgical scene to an electronic display (not shown) visible to the surgeon. The autonomous arm autonomously adjusts the field of view of the imaging device while the surgeon is performing the surgery, allowing the surgeon to properly view the surgical scene in real time. The autonomous arm 11000 is the same as the autonomous arm in FIG. 10 , so a description thereof will be omitted. However, in this case, the autonomous arm is provided as part of a standalone computer-assisted medical scope system 13000, rather than as part of the master-slave system 11260 in FIG. 10 . Therefore, the autonomous arm 11000 can be used in many different surgical environments, including, for example, laparoscopic surgery (where the medical scope is an endoscope) and open surgery.

[0224] The computer-aided medical scope system 13000 further includes a robotic control system 13020 for controlling the autonomous arm 11000. The robotic control system 13020 includes a control processor 13030 and a database 13040. Wired or wireless signals are exchanged between the robotic control system 13020 and the autonomous arm 11000 via a connection 13010.

[0225] In response to control signals received from the robot control system 13020, the control unit 11150 controls one or more actuators 11160 to drive the autonomous arm 11000 to move to an appropriate position for capturing an image of an appropriate field of view by the imaging device 11020. The control signals are generated by the control processor 13030 in response to signals received from one or more of the arm unit 11140, the imaging device 11020, and any other signal sources (not shown). The signal values received by the control processor 13030 are compared with signal values stored in the database 13040 along with corresponding arm position information. The control processor 13030 retrieves arm position information associated with the received signal values from the database 13040. The control processor 13030 then generates control signals to be sent to the control unit 11150 using the retrieved arm position information.

[0226] For example, if the signals received from the imaging device 11020 indicate a predetermined surgical scenario (e.g., via a neural network image classification process, etc.), then the predetermined surgical scenario is looked up in the database 13040, and arm position information associated with the predetermined surgical scenario is retrieved from the database. As another example, if the signals indicate a value of resistance measured by one or more force sensors 11170 of the arm unit 11140, then the resistance value is looked up in the database 12030, and arm position information associated with the resistance value is retrieved from the database (e.g., so that the position of the arm can be changed to an alternate position if the resistance increases in response to an obstacle in the path of the arm). In either case, the control processor 13030 then sends a signal to the control unit 11150 to control one or more actuators 1116 to reposition the arm to the position indicated by the retrieved arm position information.

[0227] 13 schematically illustrates another example of a computer-assisted surgery system 14000 to which the present technology can be applied. The system includes one or more autonomous arms 11000 having an imaging unit 11020 and one or more autonomous arms 12100 having a surgical device 12100. The one or more autonomous arms 11000 and the one or more autonomous arms 12100 are the same as the autonomous arms described above. Each of the autonomous arms 11000 and 12100 is controlled by a robotic control system 14080, which includes a control processor 14090 and a database 14100. Wired or wireless signals are transmitted between the robotic control system 14080 and each of the autonomous arms 11000 and 12100 via connections 14110 and 14120, respectively. The robot control system 14080 performs the functions of the robot control systems 11110 and / or 13020 described above for controlling each of the autonomous arms 11000, and performs the functions of the robot control system 12010 described above for controlling each of the autonomous arms 12100.

[0228] The autonomous arms 11000 and 12100 perform at least a portion of the surgery fully autonomously (e.g., when the system 14000 is an open surgical system). The robotic control system 14080 controls the autonomous arms 11000 and 12100 to perform predetermined actions during the surgery based on input information representing a current stage of the surgery and / or events occurring in the surgery. For example, the input information includes images taken by the image capture device 11000. Additionally, the input information may include sounds captured by a microphone (not shown), detection of a surgical instrument in use based on motion sensors included in the surgical instrument (not shown), and / or any other suitable input information.

[0229] The input information is analyzed using a suitable machine learning (ML) algorithm (e.g., a suitable artificial neural network) implemented by the machine learning based surgical planning device 14020. The planning device 14020 comprises a machine learning processor 14030, a machine learning database 14040, and a trainer 14050.

[0230] The machine learning database 14040 includes information indicative of classifications of surgical steps (e.g., making an incision, removing an organ, or applying a suture) and / or surgical events (e.g., bleeding or a patient parameter falling outside a predetermined range) and a priori known input information corresponding to those classifications (e.g., one or more images taken by the imaging device 11020 during each classification of surgical step and / or surgical event). The machine learning database 14040 is populated during a training phase by providing the information indicative of each classification and the corresponding input information to a trainer 14050. The trainer 14050 then uses this information to train a machine learning algorithm (e.g., by using the information to determine appropriate artificial neural network parameters). The machine learning algorithm is implemented by the machine learning processor 14030.

[0231] Once trained, previously unseen input information (e.g., newly captured images of a surgical scene) can be classified by the machine learning algorithm to determine the surgical stage and / or surgical event associated with the input information. Furthermore, the machine learning database includes action information indicating the action each autonomous arm 11000 and 12100 should take in response to each surgical stage and / or surgical event stored in the machine learning database (e.g., for the surgical stage "making an incision," control the autonomous arm 12100 to form an incision at the associated location, and for the surgical event "bleeding," control the autonomous arm 12100 to perform an appropriate cauterization). Thus, the machine learning-based surgical planner 14020 can determine the associated action each autonomous arm 11000 and / or 12100 should take in response to the surgical stage and / or surgical event classification output by the machine learning algorithm. The information indicating the associated action is provided to the robot control system 14080, which then sends a signal to the autonomous arms 11000 and / or 12100 to execute the associated action.

[0232] The planning device 14020 may be included in a control unit 14010 with a robotic control system 14080, allowing direct electronic communication between the planning device 14020 and the robotic control system 14080. Alternatively or additionally, the robotic control system 14080 may receive signals from other devices 14070 via a communication network 14050 (e.g., the Internet). This allows the autonomous arms 11000 and 12100 to be remotely controlled based on processing performed by these other devices 14070. In one example, the device 14070 is a cloud server with sufficient processing power to rapidly execute complex machine learning algorithms to arrive at more reliable classifications of surgical stages and / or surgical events. Different machine learning algorithms can be implemented by different respective devices 14070 using the same training data stored in an external (e.g., cloud-based) machine learning database 14060 accessible to each of these devices. Thus, each device 14070 does not need its own unique machine learning database (such as the machine learning database 14040 of the planning device 14020), and training data can be updated centrally and made available to all devices 14070. Each device 14070 further includes a trainer (such as trainer 14050) and a machine learning processor (such as machine learning processor 14030) to implement its respective machine learning algorithm.

[0233] 14 shows an example of an arm unit 11140. An arm unit 12040 is configured in the same manner. In this example, the arm unit 11140 supports an endoscope as the imaging device 11020. However, in other examples, other imaging devices 11020 or surgical devices 11030 (in the case of arm unit 11140) or 12080 (in the case of arm unit 12040) are supported.

[0234] The arm unit 11140 includes a base 7100 and an arm 7200 extending from the base 7100. The arm 7200 includes a plurality of active joints 721a-721f and supports the endoscope 11020 at the distal end of the arm 7200. The links 722a-722f are substantially rod-shaped members. Ends of the plurality of links 722a-722f are connected to each other by the active joints 721a-721f, a passive slide mechanism 7240, and a passive joint 7260. The base unit 7100 functions as a fulcrum so that the arm shape extends from the base 7100.

[0235] The position and attitude of the endoscope 11020 are controlled by driving and controlling the actuators provided on the active joints 721a to 721f of the arm 7200. According to this example, the distal end of the endoscope 11020 enters a patient's body cavity, which is the treatment site, and captures an image of the treatment site. However, the endoscope 11020 may also be another device, such as another imaging device or a surgical device. More generally, the device held at the end of the arm 7200 is referred to as a distal unit or distal device.

[0236] Here, the arm unit 7200 will be described as follows by defining coordinate axes as shown in FIG. 14. Furthermore, the vertical direction, longitudinal direction, and horizontal direction are defined according to the coordinate axes. In other words, the vertical direction relative to the base 7100 placed on the floor is defined as the z-axis direction and vertical direction. Furthermore, the direction perpendicular to the z-axis and in which the arm 7200 extends from the base 7100 (in other words, the direction in which the endoscope 11020 is positioned relative to the base 7100) is defined as the y-axis direction and longitudinal direction. Furthermore, the direction perpendicular to the y-axis and z-axis is defined as the x-axis direction and horizontal direction.

[0237] The active joints 721a to 721f connect the links to each other so that they can rotate. The active joints 721a to 721f each have an actuator and a rotation mechanism that is driven to rotate around a predetermined rotation axis by driving the actuator. When the rotational drive of each of the active joints 721a to 721f is controlled, the drive of the arm 7200 can be controlled so that, for example, the arm unit 7200 is extended or contracted (folded).

[0238] The passive slide mechanism 7240 is one aspect of a passive shape changing mechanism, and connects the link 722c and the link 722d so that they can move back and forth in a predetermined direction. The passive slide mechanism 7240 is operated by a user, for example, to move back and forth, and can change the distance between the active joint 721c on one end of the link 722c and the passive joint 7260. With this configuration, the overall shape of the arm unit 7200 can be changed.

[0239] The passive joint 7360 is one aspect of a passive shape changing mechanism, and connects the link 722d and the link 722e to each other so that they can rotate. The passive joint 7260 can be rotated by, for example, a user to change the angle formed between the link 722d and the link 722e. With this configuration, the overall shape of the arm unit 7200 can be changed.

[0240] In one embodiment, the arm unit 11140 has six active joints 721a to 721f, and six degrees of freedom are realized for driving the arm 7200. In other words, the drive control of the arm unit 11140 is realized by drive control of the six active joints 721a to 721f, while the passive slide mechanism 7260 and the passive joint 7260 are not subject to drive control.

[0241] 14, the active joints 721a, 721d, and 721f are provided so that their rotation axes correspond to the longitudinal axes of the connected links 722a and 722e and the intake direction of the connected endoscope 11020. The active joints 721b, 721c, and 721e are provided so that their rotation axes correspond to the x-axis direction, which is the direction in which the connection angles of the connected links 722a to 722c, 722e, and 722f and the endoscope 11020 change within the yz plane (the plane defined by the y-axis and z-axis). In this way, the active joints 721a, 721d, and 721f have a function of performing so-called yawing, and the active joints 421b, 421c, and 421e have a function of performing so-called pitching.

[0242] Since six degrees of freedom are realized for driving the arm 7200 of the arm unit 11140, it is possible to freely move the endoscope 11020 within the movable range of the arm 7200. Fig. 14 shows a hemisphere as an example of the movable range of the endoscope 11020. If the center point RCM (Remote Center of Motion) of the hemisphere is the imaging center of the treatment site imaged by the endoscope 11020, then by moving the endoscope 11020 on the spherical surface of the hemisphere while fixing the imaging center of the endoscope 11020 to the center point of the hemisphere, it is possible to image the treatment site from a variety of angles.

[0243] Embodiments of the present disclosure are also defined by the following numbered clauses: (1) 1. A system for controlling a medical image capture device during surgery, comprising: acquiring first image data from the medical image capture device, the first image data being an appearance of a surgical scene at a first time instance; determining one or more desired image capture characteristics of the medical image capture device based on a predicted appearance of the surgical scene at a second time instance after the first time instance based on the first image data; controlling the medical image capture device at a third time instance between the first time instance and the second time instance according to the one or more desired image capture characteristics of the medical image capture device. A system including a circuit configured as follows: (2) 10. The system of claim 1, wherein controlling the medical image capture device includes controlling the position of an articulated arm that supports the medical image capture device. (3) The system described in clause 1 or 2, wherein the circuitry is configured to determine a desired position of the medical image capture device as one of the one or more desired image capture characteristics of the medical image capture device based on the predicted appearance of the surgical scene. (4) The system described in clause 2 or 3, wherein the circuitry is further configured to determine a movement pattern to the desired image capture position according to the position of one or more objects present in the scene, and to control the position and / or orientation of an articulated arm supporting the medical image capture device according to the determined movement pattern. (5) The system of any of clauses 1 to 3, wherein the circuitry is configured to determine a desired imaging condition of the image capture device as one of the one or more desired image capture characteristics of the medical image capture device. (6) The system of clause 5, wherein the desired imaging conditions include one or more of optical imaging system conditions and image processing conditions. (7) 7. The system of clause 6, wherein the optical imaging system conditions and the image processing conditions include at least one of image zoom, image focus, image aperture, image contrast, and / or image brightness of the medical image capture device. (8) The system of any one of claims 1 to 7, wherein the circuitry is further configured to generate second image data, the second image data being the predicted appearance of the surgical scene at the second time instance, according to the first image data. (9) 9. The system of claim 8, wherein the circuitry is further configured to generate the second image data according to the first image data and information about a current state of the scene. (10) 10. The system of claim 9, wherein the circuitry is further configured to obtain information regarding a current state of the scene, including at least one of the positions of objects in the scene, the movement of objects in the scene, the types of objects present in the scene, and / or the actions being performed by people in the scene. (11) 11. The system of any of clauses 1 to 10, wherein the circuitry is configured to control the medical image capture device, which is one of an endoscope, a microscope, or an exoscope. (12) 12. The system of any of clauses 1-11, wherein the circuitry is further configured to take into account constraints of the medical image capture device when determining the desired image capture characteristics of the medical image capture device. (13) 13. The system of any of clauses 1-12, wherein the circuitry is further configured to generate the predicted appearance of the surgical scene using a machine learning system trained on surgical data obtained in a surgical scenario. (14) The system described in clause 13, wherein the surgical data obtained in the surgical scenario includes one or more of images of past surgical scenarios, verified simulations of surgical scenarios, and / or previous images of the current surgical scenario information regarding actions performed by the surgeon in previous surgical scenarios, and / or image capture characteristics of a medical image capture device used in the previous surgical scenario. (15) 15. The system of any of clauses 1 to 14, wherein the circuitry is configured to calculate weights for image capture characteristics of the medical image capture device according to one or more factors related to the image capture characteristics, and determine the image capture characteristic with the largest weight coefficient as the desired image capture characteristic of the medical image capture device. (16) The system described in any of clauses 1 to 15, wherein the circuitry is configured to calculate a range of motion that can be completed in the time between the third time instance and the second time instance, and to determine the desired image capture characteristics of the medical image capture device according to the calculation. (17) 1. A method for controlling a medical image capture device during surgery, comprising: acquiring first image data from the medical image capture device, the first image data being an appearance of a surgical scene at a first time instance; determining one or more desired image capture characteristics of the medical image capture device based on a predicted appearance of the surgical scene at a second time instance after the first time instance based on the first image data; controlling the medical image capture device at a third time instance between the first time instance and the second time instance according to the one or more desired image capture characteristics of the medical image capture device; A method comprising: (18) A computer program product containing instructions, The instructions, when executed by a computer, cause the computer to perform a method for controlling a medical image capture device; The method comprises: acquiring first image data from the medical image capture device, the first image data being an appearance of a surgical scene at a first time instance; determining one or more desired image capture characteristics of the medical image capture device based on a predicted appearance of the surgical scene at a second time instance after the first time instance based on the first image data; controlling the medical image capture device at a third time instance between the first time instance and the second time instance according to the one or more desired image capture characteristics of the medical image capture device; a computer program product,

[0244] Obviously, numerous modifications and variations of the present disclosure are possible in light of the above teachings. It is therefore to be understood that, within the scope of the appended claims, the present disclosure may be practiced other than as specifically described herein.

[0245] To the extent that embodiments of the present disclosure are described as being at least partially implemented by a data processing apparatus controlled by software, it will be understood that non-transitory machine-readable media, such as optical disks, magnetic disks, semiconductor memories, and the like, carrying such software are also considered to represent embodiments of the present disclosure.

[0246] It will be appreciated that the above description has, for clarity, described embodiments with reference to different functional units, circuits and / or processors. However, it will be apparent that any suitable distribution of functionality between different functional units, circuits and / or processors may be used without detracting from the embodiments.

[0247] The described embodiments may be implemented in any suitable form including hardware, software, firmware, or any combination of these. The described embodiments may optionally be implemented, at least in part, as computer software running on one or more data processors and / or digital signal processors. The elements and components of any embodiment may be physically, functionally, and logically implemented in any suitable way. Indeed, functionality may be implemented in a single unit, or in multiple units or as part of other functional units. Thus, the disclosed embodiments may be implemented in a single unit or may be physically and functionally distributed between different units, circuits, and / or processors.

[0248] While the present disclosure has been described in connection with several embodiments, it is not intended to be limited to the specific form set forth herein. Moreover, while certain features may appear to be described in connection with particular embodiments, those skilled in the art will recognize that the various features of the described embodiments can be combined in any manner suitable for practicing the present technology.

Claims

1. 1. A system for controlling a medical image capture device during surgery, comprising: acquiring first image data from the medical image capture device, the first image data being an appearance of a surgical scene at a first time instance; determining one or more desired image capture characteristics of the medical image capture device based on a predicted appearance of the surgical scene at a second time instance after the first time instance based on the first image data; controlling the medical image capture device at a third time instance between the first time instance and the second time instance according to the one or more desired image capture characteristics of the medical image capture device. a circuit configured to: The circuit comprises: configured to calculate weights for image capture characteristics of the medical image capture device according to one or more factors related to the image capture characteristics, and determine the image capture characteristic with the largest weight coefficient as the desired image capture characteristic of the medical image capture device; determining a confidence value for each location of the predicted appearance, excluding locations from the predicted appearance where the confidence value is below a predetermined threshold, and determining one or more desired image capture characteristics of the medical image capture device based on the predicted appearance from which the locations where the confidence value is below the predetermined threshold have been excluded; The desired image capture characteristics include at least one of image zoom, image focus, image aperture, image contrast, and image brightness of the medical image capture device. system.

2. The system of claim 1 , wherein controlling the medical image capture device comprises controlling the position of an articulated arm that supports the medical image capture device.

3. 3. The system of claim 1, wherein the circuitry is configured to determine a desired position of the medical image capture device as one of the one or more desired image capture characteristics of the medical image capture device based on the predicted appearance of the surgical scene.

4. 4. The system of claim 3, wherein the circuitry is further configured to determine a movement pattern for the medical image capture device to a desired position according to the positions of one or more objects present within the surgical scene, and to control the position and / or orientation of an articulated arm supporting the medical image capture device according to the determined movement pattern.

5. The system of any one of claims 1 to 4, wherein the circuitry is further configured to generate second image data, the second image data being the predicted appearance of the surgical scene at the second time instance, according to the first image data.

6. The system of claim 5 , wherein the circuitry is further configured to generate the second image data according to the first image data and information about a current state of the surgical scene.

7. 7. The system of claim 6, wherein the circuitry is further configured to obtain information regarding a current state of the surgical scene including at least one of a position of an object within the surgical scene, a movement of an object within the surgical scene, a type of object present within the surgical scene, and an action being performed by a person within the surgical scene.

8. The system of any one of claims 1 to 7, wherein the circuitry is configured to control the medical image capture device, which is one of an endoscope, a microscope, and an exoscope.

9. The system of any one of claims 1 to 8, wherein the circuitry is further configured to take into account constraints of the medical image capture device when determining the desired image capture characteristics of the medical image capture device.

10. 10. The system of claim 1, wherein the circuitry is further configured to generate the predicted appearance of the surgical scene using a machine learning system trained on surgical data obtained in a surgical scenario.

11. The system of claim 10, wherein the surgical data obtained in the surgical scenario includes one or more of images of past surgical scenarios, verified simulations of surgical scenarios, previous images of the current surgical scenario information regarding actions performed by the surgeon in previous surgical scenarios, and image capture characteristics of medical image capture devices used in previous surgical scenarios.

12. 12. The system of claim 1, wherein the circuitry is configured to calculate a range of motion that can be completed in a time between the third time instance and the second time instance, and to determine the desired image capture characteristics of the medical image capture device according to the calculation.

13. 1. A method for controlling a medical image capture device during surgery, comprising: The computer acquiring first image data from the medical image capture device, the first image data being an appearance of a surgical scene at a first time instance; determining one or more desired image capture characteristics of the medical image capture device based on a predicted appearance of the surgical scene at a second time instance after the first time instance based on the first image data; controlling the medical image capture device at a third time instance between the first time instance and the second time instance according to the one or more desired image capture characteristics of the medical image capture device; Including, The computer calculating weights for the image capture characteristics of the medical image capture device according to one or more factors related to the image capture characteristics, and determining the image capture characteristic with the largest weight coefficient as the desired image capture characteristic of the medical image capture device; determining a confidence value for each location of the predicted appearance, excluding locations from the predicted appearance where the confidence value is below a predetermined threshold, and determining one or more desired image capture characteristics of the medical image capture device based on the predicted appearance from which the locations where the confidence value is below the predetermined threshold have been excluded; The desired image capture characteristics include at least one of image zoom, image focus, image aperture, image contrast, and image brightness of the medical image capture device. method.

14. A computer program product containing instructions, The instructions, when executed by a computer program product, cause the computer to perform a method for controlling a medical image capture device; The method comprises: acquiring first image data from the medical image capture device, the first image data being an appearance of a surgical scene at a first time instance; determining one or more desired image capture characteristics of the medical image capture device based on a predicted appearance of the surgical scene at a second time instance after the first time instance based on the first image data; controlling the medical image capture device at a third time instance between the first time instance and the second time instance according to the one or more desired image capture characteristics of the medical image capture device; Including, calculating weights for the image capture characteristics of the medical image capture device according to one or more factors related to the image capture characteristics, and determining the image capture characteristic with the largest weight coefficient as the desired image capture characteristic of the medical image capture device; determining a confidence value for each location of the predicted appearance, excluding locations from the predicted appearance where the confidence value is below a predetermined threshold, and determining one or more desired image capture characteristics of the medical image capture device based on the predicted appearance from which the locations where the confidence value is below the predetermined threshold have been excluded; further comprising The desired image capture characteristics include at least one of image zoom, image focus, image aperture, image contrast, and image brightness of the medical image capture device. Computer program products.

Images