Methods, apparatuses, and systems for controlling image capture devices during surgical procedures

Abstract

A system for controlling a medical image capture device during a surgical procedure, the system comprising: circuitry configured to: receive a first image of a surgical scene captured by a medical image capture device from a first viewpoint and additional information for the scene; determine one or more candidate viewpoints for the medical image capture device from the additional information and previous viewpoint information for the surgical scene, images of the surgical scene being obtainable from the one or more candidate viewpoints; provide a simulated image of the surgical scene from each of the one or more candidate viewpoints from the first image of the surgical scene; control the medical image capture device to obtain an image of the surgical scene from a candidate viewpoint corresponding to a selection of one of the one or more simulated images of the surgical scene.

Description

Technical Field

[0001] This disclosure relates to a method, apparatus, and system for controlling an image capture device during surgical procedures. Background Technology

[0002] The description of the “Background Art” provided herein is for the purpose of presenting the overall context of this disclosure. Within the scope described in the Background section, the work of the presently named inventors and aspects of the description that do not qualify as prior art at the time of filing are neither expressly nor impliedly acknowledged as prior art to this invention.

[0003] Significant technological advancements have been achieved in medical systems and equipment in recent years. Computer-aided surgical systems, such as robotic surgical systems, now typically work alongside human surgeons during surgical procedures. These computer-aided surgical systems include master-slave robotic systems, in which the human surgeon operates the master device to control the operation of the slave device during surgery.

[0004] Computer-aided camera systems, such as robotic camera systems, are used in surgical settings to provide critical visual information to human operators or surgeons. These computer-aided camera systems may be equipped with a single camera that captures and provides a field of view of surgical actions within the scene. Alternatively, these computer-aided camera systems may include multiple cameras, each capturing a given field of view of surgical actions within the scene.

[0005] In some cases, it may be necessary to reposition a medical image capture device supported by an articulated arm during surgery (e.g., by moving the articulated arm). This may be required if the field of view of the surgical scene provided by a computer-assisted camera system becomes obstructed. Alternatively, this may be required when a surgeon is performing a surgical procedure, as the computer-assisted camera system in the surgical scene may have different field-of-view requirements at different stages of the surgical procedure.

[0006] However, surgical scenarios are inherently complex, involving multiple independently moving components. Unnecessary repositioning of the camera system can delay the procedure and introduce unnecessary risks to the patient.

[0007] Furthermore, the resistance to repositioning a medical image capturing device may cause surgeons to tolerate certain suboptimal viewpoints during surgical procedures. Specifically, this may result in situations where the surgeon cannot easily identify the improved camera position. The purpose of this disclosure is to address these problems. Summary of the Invention

[0008] According to a first aspect of this disclosure, a system is provided for controlling a medical image capture device during surgical procedures. The system includes: circuitry configured to: receive additional information about a scene and a first image of a surgical scene captured by the medical image capture device from a first viewpoint; determine one or more candidate viewpoints for the medical image capture device based on the additional information and previous viewpoint information of the surgical scene, from which an image of the surgical scene is obtained; provide a simulated image of the surgical scene from each of the one or more candidate viewpoints based on the first image of the surgical scene; and control the medical image capture device to obtain an image of the surgical scene from a candidate viewpoint corresponding to the selection of one of the one or more simulated images of the surgical scene.

[0009] According to a second aspect of this disclosure, a method for controlling a medical image or device during surgery is provided, the method comprising: receiving additional information about a scene and a first image of a surgical scene captured from a first viewpoint by a medical image capturing device; determining one or more candidate viewpoints for the medical image capturing device based on the additional information and previous viewpoint information of the surgical scene, from which an image of the surgical scene is obtained; providing a simulated image of the surgical scene from each of the one or more candidate viewpoints based on the first image of the surgical scene; and controlling the medical image capturing device to obtain an image of the surgical scene from a candidate viewpoint corresponding to the selection of one of the one or more simulated images of the surgical scene.

[0010] According to a third aspect of this disclosure, a computer program product including instructions is provided, which, when executed by a computer, cause the computer to perform a method of controlling a medical image capture device during surgery. The method includes: receiving additional information about a scene and a first image of a surgical scene captured by the medical image capture device from a first viewpoint; determining one or more candidate viewpoints for the medical image capture device based on the additional information and previous viewpoint information of the surgical scene, from which an image of the surgical scene is obtained; providing a simulated image of the surgical scene from each of the one or more candidate viewpoints based on the first image of the surgical scene; and controlling the medical image capture device to obtain an image of the surgical scene from a candidate viewpoint corresponding to the selection of one of the one or more simulated images of the surgical scene.

[0011] According to embodiments of this disclosure, the apparatus for controlling an image capture device during surgical procedures enables surgeons to consider alternative viewpoints of a computer-assisted camera system during surgery without repeatedly repositioning the camera. This allows for the optimization of the computer-assisted camera system viewpoint strategy without causing unnecessary delays in the surgical procedure. This disclosure is not specifically limited to these advantageous effects, and other effects may become apparent to those skilled in the art upon reading this disclosure.

[0012] The foregoing paragraphs have been provided in general description and are not intended to limit the scope of the following claims. The described embodiments and further advantages can be well understood together with reference to the following detailed description, which unfolds in conjunction with the accompanying drawings. Attached Figure Description

[0013] A more complete understanding of this disclosure and its many additional advantages is readily available when considered in conjunction with the accompanying drawings, as will become more readily understood by reference to the following detailed description.

[0014] Figure 1 This is a diagram illustrating an example of a schematic configuration of an endoscopic surgical system capable of applying the medical support arm device according to this disclosure.

[0015] Figure 2 It is shown Figure 1 The block diagram shown is an example of the functional configuration of the camera head and CUU.

[0016] Figure 3 This is an explanatory diagram showing a main device according to an example of use of this disclosure.

[0017] Figure 4 Exemplary surgical scenarios in which embodiments of this disclosure can be applied are illustrated.

[0018] Figure 5 An example is shown of an image captured from a first viewpoint by an image capture device according to an embodiment of the present disclosure.

[0019] Figure 6 An apparatus for controlling an image capture device during surgical procedures, according to an embodiment of the present disclosure, is shown.

[0020] Figure 7 An exemplary lookup table that can be used to determine candidate viewpoints according to an embodiment of the present disclosure is shown.

[0021] Figure 8 An exemplary illustration shows a simulated image of a candidate viewpoint according to an embodiment of the present disclosure.

[0022] Figure 9An exemplary diagram showing a user interface according to an embodiment of the present disclosure is provided.

[0023] Figure 10 An exemplary diagram is shown of an image captured by an image capturing device after a candidate viewpoint has been selected, according to an embodiment of the present disclosure.

[0024] Figure 11 An apparatus for controlling an image capture device during surgical procedures, according to an embodiment of the present disclosure, is shown.

[0025] Figure 12 An exemplary diagram showing a user interface according to an embodiment of the present disclosure is provided.

[0026] Figure 13 An exemplary setup of a computer-assisted surgical system according to an embodiment of the present disclosure is shown.

[0027] Figure 14 A method for controlling an image capture device during surgical procedures according to an embodiment of the present disclosure is shown.

[0028] Figure 15 A computing device for controlling an image capture device during surgical procedures is shown according to an embodiment of the present disclosure.

[0029] Figure 16 A first example of a computer-aided surgical system to which this technology can be applied is illustrated schematically.

[0030] Figure 17 A second example of a computer-assisted surgical system to which this technology can be applied is illustrated schematically.

[0031] Figure 18 A third example of a computer-aided surgical system to which this technology can be applied is illustrated schematically.

[0032] Figure 19 A fourth example of a computer-aided surgical system to which this technology can be applied is illustrated schematically.

[0033] Figure 20 An example of an arm unit is shown schematically. Detailed Implementation

[0034] Referring now to the accompanying drawings, where similar reference numerals refer to the same or corresponding parts throughout several drawings.

[0035] <<1. Basic Configuration>>

[0036] First, refer to this disclosure Figures 1 to 3 The basic configuration of an endoscopic surgical system to which embodiments of the present disclosure can be applied is described.

[0037] <1.1. Configuration Example of Endoscopic Surgical System>

[0038] Figure 1 This is a diagram illustrating an example of a schematic configuration of an endoscopic surgical system 5000 capable of applying the technology according to this disclosure. Figure 1 The illustration shows an operator (doctor) 5067 performing surgery on a patient 5071 on a bed 5069 using an endoscopic surgical system 5000. As shown, the endoscopic surgical system 5000 consists of an endoscope 5001, other surgical instruments 5017, a support arm device 5027 supporting the endoscope 5001, and a trolley 5037 on which various devices for endoscopic surgery are mounted.

[0039] In endoscopic surgery, the abdominal wall is opened by puncturing multiple tubular opening instruments called trocars (5025a to 5025d) instead of cutting the abdominal wall.

[0040] Then, the endoscope tube 5003 of the endoscope 5001 and other surgical instruments 5017 are inserted into the body cavity of the patient 5071 through cannulas 5025a to 5025d. In the example shown, an air tube 5019, an energy therapy tool 5021, and forceps 5023 are inserted into the body cavity of the patient 5071 as other surgical instruments 5017. Further, the energy therapy tool 5021 refers to a treatment tool that uses high-frequency current or ultrasonic vibration to perform tissue cutting and dissection, vascular suturing, etc. However, the surgical instruments 5017 shown are merely examples, and various surgical instruments commonly used in endoscopic surgery (e.g., forceps, retractors) can be used as surgical instruments 5017.

[0041] An image of the surgical site within the body cavity of the patient 5071, captured by the endoscope 5001, is displayed on the display device 5041. For example, while observing the image of the surgical site displayed on the display device 5041 in real time, the operator 5067 performs treatment by removing the infected site using the energy therapy tool 5021 or forceps 5023. It should be noted that, although not shown, during the surgical procedure, the air tube 5019, the energy therapy tool 5021, and the forceps 5023 are supported by the operator 5067, assistants, etc.

[0042] (Support arm equipment)

[0043] The support arm device 5027 includes an arm unit 5031 extending from the base unit 5029. In the illustrated example, the arm unit 5031 refers to a multi-joint arm composed of joints 5033a, 5033b, and 5033c, and links 5035a and 5035b, and is driven by control from an arm control device 5045. The arm unit 5031 has a distal end capable of being connected to an endoscope 5001. The endoscope 5001 is supported by the arm unit 5031, and its position and orientation are controlled. With this configuration, stable fixation of the endoscope 5001 can be achieved.

[0044] (Endoscope)

[0045] Endoscope 5001 comprises a tube 5003 and a camera head 5005. The tube 5003 has a predetermined length region distal to the end inserted into the body cavity of the patient 5071, and the camera head 5005 is connected to the proximal end of the tube 5003. Although the example shown depicts an endoscope 5001 configured with a so-called rigid body having a rigid tube 5003, endoscope 5001 can be configured with a so-called flexible body having a flexible tube 5003.

[0046] An opening for the objective lens adapter is provided at the distal end of the endoscope tube 5003. A light source device 5043 is connected to the endoscope 5001, and light generated by the light source device 5043 is guided to the distal end of the endoscope tube via a light guide extending within the endoscope tube 5003, and then emitted towards the object of observation within the body cavity of the patient 5071 via the objective lens. It should be noted that the endoscope 5001 can be a direct-viewing endoscope, an oblique-viewing endoscope, or a side-viewing endoscope.

[0047] An optical system and an imaging element are installed within the camera head 5005. The optical system focuses reflected light (observation light) from the object being observed onto the imaging element. The imaging element performs photoelectric conversion on the observation light and generates an electrical signal corresponding to the observation light; in other words, an image signal corresponding to the observed image. The image signal is then sent as raw data to the camera control unit (CCU) 5039.

[0048] It should be noted that the camera head 5005 is equipped with the ability to adjust the magnification and focal length by properly driving the optical system.

[0049] It should be noted that, for example, multiple imaging elements can be arranged in the camera head 5005 to address stereoscopic observation (3D display), etc. In this case, multiple relay optical systems are arranged within the lens barrel 5003 to guide the observation light to each of the multiple imaging elements.

[0050] (The various equipment provided in the handcart)

[0051] The CCU 5039 is configured with a central processing unit (CPU), a graphics processing unit (GPU), etc., and the CCU 5039 controls the operation of the endoscope 5001 and the display device 5041 as a whole.

[0052] Specifically, the CCU 5039 performs various types of image processing on the image signal received from the camera head 5005, such as development processing (de-mosaic processing), to display an image based on the image signal. The CCU 5039 provides the image-processed image signal to the display device 5041. Further, the CCU 5039 sends control signals to the camera head 5005 and controls the driving of the camera head 5005. The control signals may include information about imaging conditions such as magnification and focal length.

[0053] Display device 5041 displays an image based on an image signal processed by CCU 5039 under the control of CCU 5039. In the case where endoscope 5001 is an endoscope compatible with high-resolution capture, such as 4K (3840 horizontal pixels × 2160 vertical pixels), 8K (7680 horizontal pixels × 4320 vertical pixels), and / or in the case of an endoscope compatible with 3D displays, devices capable of high-resolution display and / or devices capable of 3D display can be used as display devices 5041 compatible with the aforementioned endoscopes. In the case of endoscopes compatible with high-resolution capture such as 4K and 8K, a greater sense of immersion can be achieved by using display devices 5041 with a size of 55 inches or larger. Furthermore, multiple display devices 5041 with different resolutions and sizes can be provided depending on the application.

[0054] For example, a light source such as a light-emitting diode (LED) is used to configure the light source device 5043, and the illumination light is supplied to the endoscope 5001 when capturing the surgical site.

[0055] The arm control device 5045 is configured using a processor (e.g., CPU, etc.) and operates according to a predetermined program to control the drive of the arm unit 5031 of the support arm device 5027 according to a predetermined control method.

[0056] Input device 5047 refers to the input interface related to the endoscopic surgical system 5000. Users can input various types of information and commands into the endoscopic surgical system 5000 via input device 5047. For example, users can input various types of information about surgery via input device 5047, such as information about the patient's body and information about surgical techniques. Furthermore, for example, users can input commands to the drive arm unit 5031, commands to change imaging conditions (type of illumination light, magnification, focal length, etc.) using the endoscope 5001, commands to drive the energy therapy tool 5021, etc., via input device 5047.

[0057] The type of input device 5047 is not limited, and input device 5047 can be various known input devices. For example, a mouse, keyboard, touchpad, switch, foot switch 5057, and / or joystick can be used as input device 5047. When using a touchpad as input device 5047, the touchpad can be provided on the display surface of display device 5041.

[0058] Alternatively, for example, the input device 5047 is a device installed on the user, such as an eyeglass-type wearable device and a head-mounted display (HMD), and performs various inputs based on the user's gestures or gaze detected by these devices. Further, the input device 5047 includes a camera capable of detecting the user's movement, and performs various inputs based on the user's gestures or gaze detected from images captured by the camera.

[0059] Furthermore, the input device 5047 includes a microphone capable of capturing the user's voice, and various inputs are performed using the voice through the microphone. Thus, the input device 5047 is configured to input various types of information non-contactly, and specifically, users in clean areas (e.g., operators 5067) can operate equipment in uncleaned areas non-contactly. Further, users can operate the equipment without removing surgical instruments from their hands, thereby improving user convenience.

[0060] Treatment tool control device 5049 controls the drive of energy therapy tool 5021 for procedures such as tissue ablation, resection, and vascular suturing. To ensure the field of vision of endoscope 5001 and to ensure the operator's workspace, air blowing device 5051 delivers gas into the body cavity via air blowing tube 5019 to inflate the body cavity of patient 5071. Recorder 5053 refers to a device capable of recording various types of information about the surgical procedure. Printer 5055 refers to a device capable of printing various types of information about the surgical procedure in various forms such as text, images, and charts.

[0061] The specific features of the endoscopic surgical system 5000 will be described in more detail below.

[0062] (Support arm equipment)

[0063] The support arm device 5027 includes a base unit 5029 as a base and an arm unit 5031 extending from the base unit 5029. In the example shown, the arm unit 5031 is composed of a plurality of joints 5033a, 5033b, and 5033c, and a plurality of links 5035a and 5035b connected by joints 5033b.

[0064] Figure 1 For simplicity, the configuration of the arm unit 5031 is shown in a simplified manner. In practice, the shapes, numbers, and arrangements of the joints 5033a to 5033c and the links 5035a and 5035b, as well as the rotation axis directions of the joints 5033a to 5033c, are appropriately configured to give the arm unit 5031 the desired degrees of freedom. For example, the arm unit 5031 can preferably be configured to have six or more degrees of freedom. With this configuration, the endoscope 5001 can move freely within the movable range of the arm unit 5031, thereby allowing the endoscope tube 5003 of the endoscope 5001 to be inserted into the body cavity of the patient 5071 from the desired direction.

[0065] Actuators are disposed in joints 5033a to 5033c, and joints 5033a to 5033c are configured to rotate about a predetermined rotation axis by the actuators. Each rotation angle of joints 5033a to 5033c is controlled by the actuator drive via the arm control device 5045, and the drive of arm unit 5031 is also controlled. With this configuration, control of the position and orientation of endoscope 5001 can be achieved. At this time, the arm control device 5045 can control the drive of arm unit 5031 using various known control methods such as force control or position control.

[0066] For example, when operator 5067 appropriately performs operational input via input device 5047 (including foot switch 5057), the position and posture of endoscope 5001 can be controlled, and the arm control device 5045 appropriately controls the drive of arm unit 5031 according to the operational input. Through this control, endoscope 5001, located at the distal end of arm unit 5031, can be moved from any position to any position, and then, after movement, is fixedly supported in a specific position. It should be noted that arm unit 5031 can be operated in a so-called master-slave manner. In this case, the user can remotely operate arm unit 5031 via input device 5047 installed at a location far from the operating room.

[0067] Furthermore, in the case of force control, the arm control device 5045 can receive external forces from the user and perform so-called power assist control on the actuators driving joints 5033a to 5033c, so that the arm unit 5031 moves smoothly according to the external force. With this configuration, when the user moves the arm unit 5031 while directly touching it, the arm unit 5031 can be moved with relatively light force. Therefore, the endoscope 5001 can be moved more intuitively with simpler operation, and user convenience can be improved.

[0068] Here, in endoscopic surgery, the endoscope 5001 is typically supported by a physician known as a scopist. Therefore, by using the support arm device 5027, the position of the endoscope 5001 can be more reliably fixed without the human hand, thereby allowing for stable image acquisition of the surgical site and smooth execution of the surgical procedure.

[0069] It should be noted that the arm control device 5045 does not necessarily need to be located in the trolley 5037. Furthermore, the arm control device 5045 does not necessarily need to be a single device. For example, the arm control device 5045 may be located at each joint 5033a to 5033c of the arm unit 5031 of the support arm device 5027, or the drive control of the arm unit 5031 may be achieved by multiple arm control devices 5045 cooperating with each other.

[0070] (Light source equipment)

[0071] The light source device 5043 supplies illumination light to the endoscope 5001 at the moment of capturing the surgical site. For example, the light source device 5043 may be configured using a white light source composed of LEDs, a laser light source, or a combination thereof. In this case, where the white light source is composed of a combination of RGB laser light sources, the output intensity and output time of each color (each wavelength) can be controlled with high precision, and thereby the white balance of the image captured by the light source device 5043 can be adjusted. Furthermore, in this case, images corresponding to each RGB can be captured in a time-division manner by illuminating the object of observation with laser light from each RGB laser light source and controlling the driving of the imaging element of the camera head 5005 in sync with the illumination time. According to this method, color images can be obtained without providing color filters in the imaging element.

[0072] Furthermore, the drive of the light source device 5043 can be controlled to change the intensity of the light output at predetermined intervals. The drive of the imaging element of the camera head 5005 is controlled synchronously with the change in light intensity to acquire images in a time-division manner, and so-called high dynamic range images that are free from crushed blacks and bleed whites can be generated by combining the images.

[0073] Furthermore, the light source device 5043 can be configured to supply light at a predetermined wavelength band compatible with special light observation. For example, in special light observation, light within a narrow band is emitted by utilizing the wavelength dependence of light absorption in body tissue and comparing it with the illumination light during normal observation (in other words, white light), thereby performing so-called narrow-band imaging (NBI), in which predetermined tissues such as blood vessels in the mucosal surface are captured with higher contrast. Alternatively, in special light observation, fluorescence observation can also be performed, which uses fluorescence light generated by emitting excitation light to obtain an image. In fluorescence observation, body tissue can be irradiated with excitation light and fluorescence light from the body tissue can be observed (autofluorescence observation), such as by locally injecting a reagent such as indocyanine green (ICG) into body tissue and irradiating the body tissue with excitation light corresponding to the fluorescence wavelength of the reagent to obtain a fluorescence image, etc. The light source device 5043 can be configured to supply narrow-band light and / or excitation light corresponding to this special light observation.

[0074] (Camera lens and CCU)

[0075] Reference Figure 2 The functions of the camera head 5005 and CCU 5039 of the endoscope 5001 are described in more detail. Figure 2 It is shown Figure 1 A block diagram illustrating an example of the functional configuration of camera head 5005 and CCU 5039 is shown.

[0076] refer to Figure 2 The camera head 5005 includes a lens unit 5007, an imaging unit 5009, a drive unit 5011, a communication unit 5013, and a camera head control unit 5015. Furthermore, the CCU 5039 includes a communication unit 5059, an image processing unit 5061, and a control unit 5063. The camera head 5005 is connected to the CCU 5039 to enable bidirectional communication via a transmission cable 5065.

[0077] First, the functional configuration of the camera head 5005 will be described. The lens unit 5007 is an optical system disposed at the connection portion with the lens barrel 5003. Observational light acquired from the distal end of the lens barrel 5003 is guided to the camera head 5005 and incident on the lens unit 5007. The lens unit 5007 is configured by combining multiple lenses, including a zoom lens and a focusing lens. The optical characteristics of the lens unit 5007 are adjusted so that the observational light is focused onto the light-receiving surface of the imaging element of the imaging unit 5009. Furthermore, the zoom lens and the focusing lens are configured such that their positions on the optical axis can be moved to adjust the magnification and focal length of the captured image.

[0078] Imaging unit 5009 comprises an imaging element and is arranged in the subsequent stage of lens unit 5007. It focuses the observation light passing through lens unit 5007 onto the light-receiving surface of the imaging element and generates an image signal corresponding to the observed object through photoelectric conversion. The image signal generated by imaging unit 5009 is provided to communication unit 5013.

[0079] For example, as the imaging element constituting the imaging unit 5009, a complementary metal-oxide-semiconductor (CMOS) type image sensor capable of color capture with a Bayer arrangement can be used. It should be noted that, for example, an imaging element compatible with capturing high-resolution images of 4K or higher can be used. Because a high-resolution image of the surgical site can be obtained, the operator 5067 can have a more detailed understanding of the surgical site and can perform surgery more smoothly.

[0080] Furthermore, the imaging elements constituting the imaging unit 5009 are configured to have a pair of imaging elements compatible with 3D display to acquire image signals for the right and left eyes respectively. When performing 3D display, the operator 5067 can more accurately determine the depth of living tissue in the surgical site. It should be noted that when the imaging unit 5009 is configured as a multi-lens type, multiple lens units 5007 are set to correspond to the respective imaging elements.

[0081] Furthermore, the imaging unit 5009 need not be located in the camera head 5005. For example, the imaging unit 5009 can be located in the lens barrel 5003, which is only located behind the objective lens.

[0082] The actuator-configured drive unit 5011, under the control of the camera head control unit 5015, moves the zoom lens and focusing lens in the lens unit 5007 a predetermined distance along the optical axis. This movement allows for appropriate adjustment of the magnification and focal length of the image captured by the imaging unit 5009.

[0083] The communication unit 5013 is configured using communication equipment to send various types of information to and receive various types of information from the CCU 5039. The communication unit 5013 transmits the image signal obtained from the imaging unit 5009 as raw data to the CCU 5039 via a transmission cable 5065. In this case, it is preferable to transmit the image signal via optical communication, thereby displaying the captured image of the surgical site with low latency. During surgery, the operator 5067 performs the surgery while observing the state of the infected site through the captured image, and therefore, it is necessary to display the moving image of the surgical site as real-time as possible for safer and more reliable surgery. In the case of optical communication, a photoelectric conversion module that converts electrical signals into optical signals is provided in the communication unit 5013. The image signal is converted into an optical signal by the photoelectric conversion module and then transmitted to the CCU 5039 via the transmission cable 5065.

[0084] Furthermore, the communication unit 5013 receives control signals from the CCU 5039 to control the camera head 5005. For example, the control signals include information about imaging conditions, such as information specifying the frame rate of the captured image, information specifying the exposure value during imaging, and / or information specifying the magnification and focal length of the captured image. The communication unit 5013 provides the received control signals to the camera head control unit 5015. It should be noted that control signals from the CCU 5039 can also be transmitted via optical communication. In this case, the communication unit 5013 is equipped with a photoelectric conversion module that converts optical signals into electrical signals, and the control signals are converted into electrical signals via the photoelectric conversion module and then provided to the camera head control unit 5015.

[0085] It should be noted that the control unit 5063 in CCU 5039 automatically sets imaging conditions such as the frame rate, exposure value, magnification, and focal length based on the acquired image signal. That is, the endoscope 5001 is equipped with so-called automatic exposure (AE) function, automatic focus (AF) function, and automatic white balance (AWB) function.

[0086] The camera head control unit 5015 controls the driving of the camera head 5005 based on control signals received from the CCU 5039 via the communication unit 5013. For example, the camera head control unit 5015 controls the driving of the imaging element in the imaging unit 5009 based on information specifying the frame rate of the captured image and / or information specifying the exposure during imaging. Further, for example, the camera head control unit 5015 appropriately moves the zoom lens and focusing lens of the lens unit 5007 via the drive unit 5011 based on information specifying the magnification and focal length of the captured image.

[0087] Furthermore, the camera head control unit 5015 may have the function of storing information that identifies the lens barrel 5003 and the camera head 5005.

[0088] It should be noted that by arranging the lens unit 5007, imaging unit 5009, etc. in a sealed structure with high airtightness and waterproofness, the camera head 5005 can withstand high-pressure sterilization.

[0089] Next, the functional configuration of CCU 5039 will be described. Communication unit 5059 is configured using a communication device to transmit various types of information to and receive various types of information from camera head 5005. Communication unit 5059 receives image signals transmitted from camera head 5005 via transmission cable 5065. As described above, in this case, image signals can be appropriately transmitted via optical communication. In this case, communication unit 5059 is provided with a photoelectric conversion module that converts optical signals into electrical signals to ensure compatibility with optical communication. Communication unit 5059 provides the image signals converted into electrical signals to image processing unit 5061.

[0090] Furthermore, the communication unit 5059 sends control signals to the camera head 5005 to control its drive. Control signals can also be sent via optical communication.

[0091] The image processing unit 5061 performs various types of image processing on the image signal, which is the raw data sent from the camera head 5005. For example, image processing includes various types of known signal processing, such as development processing, image quality improvement processing (e.g., bandwidth enhancement processing, super-resolution processing, noise reduction (NR) processing, and / or camera shake correction processing), and / or magnification processing (electronic scaling processing). Furthermore, the image processing unit 5061 performs detection processing on the image signal to perform AE, AF, and AWB.

[0092] The image processing unit 5061 is configured using processors such as CPUs and GPUs, and when the processor operates according to a predetermined program, it is able to perform the aforementioned image processing and detection processes. It should be noted that when the image processing unit 5061 is composed of multiple GPUs, the image processing unit 5061 appropriately segments information about the image signal and the image processing is performed in parallel by the multiple GPUs.

[0093] The control unit 5063 uses the endoscope 5001 that captures the images and the display to perform various types of control regarding imaging of the surgical site. For example, the control unit 5063 generates control signals to control the drive of the camera head 5005. In this case, if the imaging conditions are input by the user, the control unit 5063 generates the control signals based on the user's input. Alternatively, if the endoscope 5001 is equipped with AE, AF, and AWB functions, the control unit 5063 appropriately calculates the optimal exposure value, focal length, and white balance to generate control signals based on the detection and processing results of the image processing unit 5061.

[0094] Furthermore, the control unit 5063 enables the display device 5041 to display an image of the surgical site based on the image signal processed by the image processing unit 5061.

[0095] At this time, the control unit 5063 uses various image recognition technologies to identify various objects in the image of the surgical site. For example, the control unit 5063 detects the edge shape, color, etc., of objects included in the image of the surgical site, and thereby can identify surgical instruments such as forceps, specific living parts, bleeding, and haze when using the energy therapy tool 5021. When the display device 5041 displays the image of the surgical site, the control unit 5063 uses the recognition results to overlay and display various types of surgical support information about the surgical site image. Because the surgical support information is overlaid and displayed and presented to the operator 5067, surgery can be performed more safely and reliably.

[0096] The transmission cable 5065 connecting the camera head 5005 and the CCU 5039 is an electrical signal cable compatible with electrical signal communication, an optical fiber compatible with optical communication, or a composite cable thereof.

[0097] Here, in the example shown, communication is performed via a wired connection using a transmission cable 5065. However, communication between the camera head 5005 and the CCU 5039 can be performed wirelessly. When communication is performed wirelessly, it is not necessary to lay the transmission cable 5065 in the operating room, thus resolving the issue of the transmission cable 5065 obstructing the movement of medical personnel in the operating room.

[0098] Examples of endoscopic surgical systems 5000 capable of applying the technology according to this disclosure have been described above. It should be noted that while the endoscopic surgical system 5000 has been described as an example herein, systems capable of applying the technology according to this disclosure are not limited to this example. For instance, the technology according to this disclosure can be applied to flexible endoscopic systems or microsurgical systems used for examinations.

[0099] Alternatively, aspects of this disclosure can be applied to medical robot systems, including master-slave medical robot systems. In a medical robot system, a user (such as a physician 5067) operates a master device (surgical console) to send operating commands to slave devices (excluding trolleys) via wired or wireless communication devices, and remotely operates the slave devices. The medical robot system may also include a separate trolley containing supporting hardware and software components, such as an electrosurgical unit (ESU), a suction / irrigation pump, and a light source for an endoscope / microscope.

[0100] Figure 3 An example of the use of the main device 60 according to this disclosure is shown. Figure 3 The system provides two main devices 60R and 60L for the right and left hands, respectively. The surgeon places both arms or elbows on the support base 50 and grasps the operating units 100R and 100L using both hands. In this position, the surgeon operates the operating units 100R and 100L while viewing a monitor 210 displaying the surgical site. The surgeon can shift the position and orientation of the corresponding operating units 100R and 100L to remotely operate the position or orientation of surgical instruments attached to various subordinate devices (not shown), or to perform grasping operations using the respective surgical instruments.

[0101] The above has already referred to the contents of this disclosure. Figures 1 to 3 A basic configuration of an exemplary surgical system applicable to embodiments of this disclosure has been described. Specific embodiments of this disclosure will now be described.

[0102] <Controlling image capture equipment during surgical procedures>

[0103] As described above, it is desirable to provide an apparatus that can optimize the viewpoint of a computer-aided camera system during surgery without interrupting the surgical procedure. Therefore, according to embodiments of this disclosure, an apparatus, method, and computer program product for controlling an image capture device during surgical procedures are provided.

[0104] The apparatus for controlling an image capture device during surgery will now be described with reference to an exemplary surgical scenario. However, it should be understood that this disclosure is not specifically limited to this particular example and can be applied as needed to any such surgical scenario.

[0105] Example scenario:

[0106] Figure 4 Exemplary surgical scenarios in which embodiments of this disclosure can be applied are illustrated.

[0107] In this example, a surgical scenario 800 (such as an operating room) is shown. Surgeon 804 performs surgery on patient 802. This could be a surgical procedure requiring the surgeon to perform surgery on a target area 808 of the patient. In this example, the surgery being performed by the surgeon is laparoscopic surgery; however, this application is not specifically limited in this respect. During laparoscopic surgery, the surgeon uses one or more surgical instruments and an endoscope (i.e., a scope attached to a camera head). These surgical instruments and the endoscope are passed through a cannula (such as those described in this disclosure). Figure 1 (As described) is inserted into the patient's body cavity to enable the surgeon to perform laparoscopic surgery on the patient.

[0108] Now, in this example, a computer-aided surgical system, including a computer-aided camera system 806, assists the surgeon 804 during surgery. For example, the computer-aided surgical system may be such as those described in this disclosure. Figures 1 to 3 The systems described are systems of these.

[0109] In this example, the computer-assisted camera system 806 includes a medical image capture device (such as an endoscope system including a scope and a camera head) that captures images of scene 800 and provides the images to a display (not shown). The surgeon 804 is then able to view the images acquired by the computer-assisted camera system 806 while performing surgery on the patient 802.

[0110] As described above, during a surgical procedure, surgeon 804 performs treatment on target area 808 of patient 802. To perform the treatment, surgeon 804 may introduce one or more surgical tools 810 and 812 into the scene. In this specific example, surgical tool 810 may be a scalpel, and surgical tool 812 may be a suction device. Because the surgeon is performing surgery on target area 808, the computer-aided camera system is configured such that an image capture device in the computer-aided camera system captures an image of target area 808 of patient 802. That is, the computer-aided camera system is configured such that target area 808 falls within the field of view of the image capture device (in this example, the area enclosed by line 814 shows the field of view of the image capture device).

[0111] During the surgical procedure, one or more medical support personnel and / or assistants 816 also assist the surgeon 804. Importantly, these medical support personnel and / or assistants 816 are located close to the patient 802 and the surgeon 804 to enable them to provide necessary support and assistance to the surgeon 804 during the surgical procedure. For example, the surgeon 804 may require the medical assistant 816 to pass specific tools to the surgeon or perform specific tasks at a given stage during the surgical procedure.

[0112] Additional medical equipment 818 can also be located in the surgical setting. This equipment may include items such as anesthesia machines, instrument tables, and patient monitors. Importantly, this equipment is positioned close to the patient 802 and the surgeon 808 so that the surgeon (or other surgical professionals in the surgical environment, such as the anesthesiologist) can easily access the equipment as needed during the surgical procedure.

[0113] In some examples, such as endoscopic surgery, surgeon 808 may not be able to directly observe the target area 808 of patient 802. That is, computer-assisted camera system 806 can provide the surgeon with the only available field of view of the target area. Moreover, even in cases where the surgeon is thus able to directly observe the target area 808, the computer-assisted camera system can provide enhanced views of the target area 808 that the surgeon relies on to perform the surgery (such as magnified views of the target area).

[0114] Accordingly, it is important that computer-assisted camera systems provide surgeons with a clear and / or unobstructed view of the target area. Therefore, the initial configuration of a computer-assisted camera system may require considerable care.

[0115] However, as the surgical procedure proceeds, dynamic elements within the surgical environment may obstruct images acquired by the computer-aided camera system, resulting in a degraded field of view of the scene presented to the surgeon 804. In other words, from the perspective of the computer-aided camera system (i.e., from the position where the image capture device of the computer-aided camera system captures an image of the target area 808), introducing one or more additional surgical tools into the surgical environment during the surgical procedure may at least partially obstruct the target area.

[0116] Alternatively, the movement of the surgeon 804 and / or support personnel and assistants 816 may impede the ability of the image capturing device in a computer-aided camera system to capture a clear image of the scene.

[0117] Figure 5 This shows an example of an image captured by an image capture device from a first-person perspective.

[0118] exist Figure 5 The image shown is an image 900 of the target region 808 of the patient 802 captured by the image capturing device in the computer-assisted camera system 806. Surgical instruments 810 are also seen in this image captured by the image capturing device. Now, when surgery begins, the image capturing device captures a clear image of the target region 808. However, at this time (i.e., at the time corresponding to the current image captured by the image capturing device), the field of view of the scene captured by the image capturing device has deteriorated.

[0119] Specifically, in this example, because significant glare and reflective points 902 have appeared on the tissue surface of the target area, the surgeon can no longer obtain a clear view of the target area. These glare and / or reflective points 902 appear due to changes in the target area and / or the surgical environment, preventing the surgeon from obtaining a clear view of the target area.

[0120] However, the surgeon 804 may not be aware of a better position or viewpoint for the image capture device of a computer-aided camera system. Moreover, due to the delay in surgical procedures caused by the repositioning of the image capture device, the surgeon 804 is unwilling to try other viewpoints to see if they reduce glare and reflections.

[0121] Accordingly, an apparatus for controlling an image capture device during surgical procedures is provided according to embodiments of the present disclosure.

[0122] Device:

[0123] Figure 6 This disclosure illustrates an apparatus or system for controlling an image capture device (such as a medical image capture device) during surgical procedures according to embodiments thereof.

[0124] The apparatus 1000 includes: a first receiving unit 1002 configured to receive additional information about the scene and a first image of the surgical scene captured from a first viewpoint by a medical image capturing device; a determining unit 1004 configured to determine one or more candidate viewpoints for the medical image capturing device based on the additional information and previous viewpoint information of the surgical scene, from which the image of the surgical scene is obtained; a providing unit 1006 configured to provide a simulated image of the surgical scene from each of the one or more candidate viewpoints based on the first image of the surgical scene; and a control unit 1008 configured to control the medical image capturing device to obtain the image of the surgical scene from a candidate viewpoint corresponding to the selection of one of the one or more simulated images of the surgical scene.

[0125] Return to this disclosure Figure 4In an exemplary scenario, device 1000 can be connected to an arm control device (such as reference 1000). Figure 1 The described arm control device 5045 is used to control the movement of the image capture device. Alternatively, the device 1000 may be connected to or form part of a central processing unit. Reference will now be made to the present disclosure. Figure 4 The features of the device 1000 are described using exemplary surgical scenarios. However, it should be understood that the device can be applied to any such surgical scenario as needed.

[0126] First receiving unit:

[0127] <First Image Data>

[0128] As described above, during surgery, the image capture device of the computer-assisted camera system 1000 captures images of the surgical scene. The first receiving unit 1002 in the device 1000 is configured to receive the image captured by the image capture device as a first image (or image data). Thus, when the image is captured by the image capture device, the first image provides the device 1000 with information about the appearance of the surgical scene. Therefore, in this example, the first image is the same image displayed on a display device (such as display device 5041) to a user (such as a surgeon). That is, the first image displays the current appearance of the surgical scene. Therefore, in this example, the first image may be the same image displayed by the present disclosure. Figure 5 Image 900 is shown in the image.

[0129] It should be understood that the receiving unit is not specifically limited in the manner in which it receives the first image data. For example, the receiving unit can receive image data from the image capture device via any suitable wired or wireless means. Moreover, the actual form of the image data depends on the type of image capture device used to capture the image data. In this example, the image capture device may be an endoscope, a telescopic device, a microscope, or an exoscope. Therefore, in this example, the image data acquired by the acquisition unit may be a high-definition image of the scene, a 4K image, or an 8K image, etc. That is, any medical imaging device according to the embodiments of this disclosure can be used as needed.

[0130] <Type of Additional Information>

[0131] Furthermore, the first receiving unit 1002 of the device 1000 is further configured to receive additional information about the scenario. The form of this additional information is not specifically limited and varies depending on the application of the embodiments of this disclosure. Moreover, it should be understood that the device 1000 may receive additional information from multiple different sources depending on the type of additional information received. However, it should be understood that, regardless of form, the additional information is contextual information that provides the device 1000 with a better understanding of the surgical procedure performed by the surgeon 804.

[0132] In a specific example, additional information about the scene may include at least one of surgical information and / or environmental information of the surgical scene.

[0133] In some examples, environmental information may include information about the surgeon's work area. This information may include, for example, the surgeon's location and orientation relative to the patient's target area, the surgeon's surrounding work area, obstacles within the area surrounding the surgeon (such as the surgical environment), lighting conditions (such as lighting type and lighting control information), and the orientation of the operating table relative to the image capture equipment.

[0134] In some examples, surgical information may include surgical tool information, providing device 1000 with a detailed understanding of the surgical tools used by the surgeon and their respective locations within the surgical scene. Specifically, in the examples, additional information may include surgical tool information such as: the type of tool located in the surgical scene; the location of the tool within the surgical scene; the usage status of the tool (e.g., whether a tool such as an energy device is activated); information about how the surgeon manipulates the tool (e.g., whether the surgeon holds the tool with both hands or whether it is held by a supported surgeon); tool spatial and motion information (including speed, trajectory, degree of tool activity (i.e., movement per minute), and end-actuator separation between multiple tools); the number of times a tool is changed within a specific time period; tools to be used (e.g., tools that the assistant surgeon prepares to use in the next stage of the surgery), etc.

[0135] In some examples, the received surgical information may include information about the appearance and / or nature of the surgical tissue the surgeon is about to operate on. For example, it may include information about the site of the patient the surgeon is about to operate on (such as, for example, the heart or lungs).

[0136] In some examples, surgical information may include procedural information relating to the status of the surgery (such as the progress of the surgery) and the specific type of surgery performed by the surgeon (such as a standardized workflow for a given type of surgery). This information may also include the stages of the surgical procedure performed by the surgeon.

[0137] In some examples, surgical information may include information about the patient's medical condition. This may include information such as the patient's blood pressure, oxygen saturation level, and abdominal pressure.

[0138] <Source of Additional Information>

[0139] Now, as described above, the receiving unit 1002 can receive additional information from one or more sources depending on the situation. In this example, additional information can be received from one or more sensors in the surgical environment. That is, additional information can be received from one or more sensors located within the tools used by the surgeon. Alternatively, position or movement data can be received from orientation information measured by one or more sensors in a computer-aided camera system.

[0140] Alternatively, this additional information can be received from the analysis of images (which may include images of the patient, surgeon, or other features in the operating room) or video streams within or outside the patient's surgical environment. Machine vision systems can extract information about material categories (to identify tissue types and / or tissue properties), object identification (e.g., tool or organ type), and motion recognition (tool movement, tool activity, etc.).

[0141] Alternatively, additional information can be extracted from one or more device and / or system interfaces (such as lighting systems, suction devices, operating room cameras, etc.). Alternatively, the receiving unit 1002 in device 1000 can interface with the operating room management unit to obtain relevant external patient data.

[0142] Alternatively, the first receiving unit 1002 in device 1000 can extract additional information (such as dialogue between the surgeon and assistant during surgery) from the audio stream captured in the operating room. For example, the first receiving unit 1002 can utilize speech recognition technology to enable the device to monitor the dialogue of surgical personnel and extract relevant information. Speech recognition technology enables device 1000 to detect specific instructions given by the surgeon for the next stage of surgery, extract basic keywords from the dialogue, and / or apply natural language processing to the entire dialogue to obtain all relevant contextual data.

[0143] Alternatively, this additional information can be received via manual input from the surgeon, medical assistant, or support staff. This can include an interface that allows the surgeon and / or medical assistant / support staff to instruct on relevant information such as the next surgical stage and / or manually mark items such as tools, organs, and other features in the camera vision feed. The surgical stage can then be used to retrieve information from a centralized database (using lookup tables, etc.) detailing the typical surgical workflow, stages, associated procedures, and tools used at each stage of the surgical procedure.

[0144] Once the receiving unit 1002 receives the additional information and the first image, it transmits the additional information to the determining unit 1004 in the device 1000. In some examples, the receiving unit 1002 may transmit the information directly to the determining unit 1004. In other examples, the first receiving unit 1002 may store the additional information in a memory or storage device accessible to the determining unit 1004.

[0145] Determine the unit:

[0146] The determining unit 1004 in the device 1000 is configured to determine one or more candidate viewpoints for the image capturing device based on additional information and previous viewpoint information of the surgical scene, from which the image of the surgical scene is obtained.

[0147] These candidate viewpoints are suggested viewpoints within a surgical setting where the image capture device can be used to provide a clear image of the scene. According to embodiments of this disclosure, these candidate viewpoints are determined based on viewpoints already used in previous surgical procedures. Therefore, viewpoint information may include location and / or orientation information of the image capture device (i.e., the location and / or orientation information of the image capture device used in previous surgical procedures).

[0148] That is, as described above, the additional information received by the first receiving unit 1002 is information that enables the device 1000 to determine information about the surgical procedure performed by the surgical procedure 804.

[0149] Accordingly, in the example, the determining unit 1004 can use this information to query a lookup table providing information on candidate viewpoints for a surgical procedure. The table providing information on candidate viewpoints for a surgical procedure can be constructed based on the operational history of the computer-aided camera system (i.e., viewpoints used by the image capturing device in previous surgeries related to the surgical procedure).

[0150] Refer to the contents of this disclosure Figure 7 An exemplary lookup table that can be used to determine candidate viewpoints is shown.

[0151] The lookup table 1100 can be stored in the internal storage of the device 1000, or alternatively, it can be stored in external storage accessible to the device 1000 (such as an external server). In this specific example, the first column 1102 defines information about the surgical procedure (this may also include different items for different stages of the same surgical procedure, such as the initial, intermediate, and final stages of the surgical procedure). The determining unit can query the lookup table 1100 based on the surgical procedure determined from the additional information to determine the item corresponding to the current surgical procedure (or the query can be performed based on the additional information itself). Once the item corresponding to the current surgical procedure is identified, the determining unit 1004 in the device 1000 can read candidate viewpoint information about the surgical procedure from the corresponding rows of subsequent columns 1104, 1106, and 1108.

[0152] That is, each column 1104, 1106, and 1108 can store information about the viewpoint used by the image capture device in a previous surgical procedure that matches the current procedure.

[0153] Therefore, the determining unit can identify one or more candidate viewpoints for the current surgical procedure from the table.

[0154] That is, in this example, querying lookup table 1100 enables determination unit 1004 to extract candidate viewpoints from the autonomous operation history of the computer-aided camera system in relation to the current surgical scene. In some examples, candidate viewpoints may be extracted based on previous viewpoints used in comparable surgical procedures (e.g., this could include viewpoints used at different stages of the same surgical procedure).

[0155] As described above, the query lookup table 1100 can be constructed based on the viewpoint used by the computer-aided camera system in previous surgical scenarios. However, the query lookup table can be further constructed based on the viewpoint used by the computer-aided camera system in one or more realistic simulations of the surgical procedure. Alternatively or additionally, the table can be constructed based on the viewpoint used by other surgeons (human or robotic) performing the surgical procedure.

[0156] Thus, the lookup table enables the determining unit 1004 to identify candidate viewpoints for image capture devices that the surgeon 804 has not yet conceived. Therefore, the candidate viewpoints may be surprising or unexpected for the surgeon 804, thereby providing the surgeon with viewpoints that they had not previously conceived.

[0157] Now, it should be recognized that Figure 7The example in the diagram is merely one example of the determination of candidate viewpoints that the determination unit 1004 can perform. The apparatus 1000 may use any such processing as needed, i.e., it may enable the determination unit 1004 to determine one or more candidate viewpoints based on prior viewpoint preferences related to additional information acquired by the first acquisition unit 1002.

[0158] Thus, the determining unit 1004 organizes the previous viewpoints of the surgery into one or more candidate viewpoints for the surgical scene.

[0159] In some examples, the determining unit 1004 is configured to analyze candidate viewpoints according to predetermined metrics and display the top N candidate viewpoints (e.g., the top three candidates) to the surgeon for selection. That is, the determining unit can use one or more evaluation algorithms to evaluate candidate viewpoints related to the current viewpoint and select a subgroup of candidate viewpoints that provide a relevant viewpoint advantage to the surgeon. This enables the determining unit 1004 to select multiple candidate viewpoints that provide, or can provide, a viewpoint advantage over the current operation for the surgeon 804.

[0160] The surgeon’s viewpoint advantages may include viewpoints derived from previous surgical procedures, i.e., an expanded viewpoint that provides a specific area of ​​tissue, an expanded viewpoint of the tools used by the surgeon, improved identification of key features (such as features of the target area, including subcutaneous veins or tumors removed from the target area), and / or improved lighting conditions (such as fewer shadows or less reflection from the tissue surface), etc.

[0161] The selection of N candidate viewpoints can also be performed based on a comparison between the viewpoint and the viewpoint preference of surgeon 804. For example, this enables the determining unit to identify favorable candidate viewpoints that surgeon 804 is unlikely to consider.

[0162] The evaluation is based on the viewpoint information itself (such as information about candidate viewpoints that have been extracted from the lookup table).

[0163] Furthermore, in some examples, the favorable evaluation unit can be configured to assess candidate viewpoints based on predetermined metrics, and to display at least a subset of candidate viewpoints on the evaluation control display. As described above, for example, the predetermined metrics can be based on a comparison of candidate viewpoints with one or more viewpoint preferences of the surgeon. Thus, only a subset of the generated alternative candidate viewpoints is displayed to the surgeon for selection.

[0164] Now, return to this disclosure Figure 4For example, one or more candidate viewpoints may include information about candidate locations where the image capture device is capable of capturing an image of a target region 808 of the patient 802. However, candidate viewpoints may also include information about the candidate image capture properties of the image capture device. For example, this may include candidate imaging types used by the image capture device. For instance, one candidate viewpoint may be one where hyperspectral imaging utilizing spectroscopy is used to measure changes in the interaction between light and radiation within the body.

[0165] Within the patient's body cavity, another candidate viewpoint can be obtained using optical imaging with visible light illumination. Image capture properties, such as the zoom level or image aperture used by the image capture device, may also be included within the candidate viewpoints determined by the determining unit 1004.

[0166] Therefore, in a specific example, the imaging properties of the image capture device may include at least one of the optical system conditions of the medical image capture device and / or the image processing conditions of the captured image. For example, optical system conditions may include factors such as optical image scaling, image focus, image aperture, image contrast, image brightness, and / or imaging type of the image capture device. Conversely, image processing conditions of the captured image may include factors such as digital image scaling applied to the image and / or factors related to image processing (such as image brightness, contrast, saturation, hue, etc.).

[0167] Furthermore, in some examples, candidate viewpoints may include static viewpoints and dynamic viewpoints (i.e., viewpoints from a single location or viewpoints that move or display two or more locations of a surgical scene).

[0168] Once the determining unit 1004 determines a list of one or more candidate viewpoints, the candidate viewpoints are passed to the providing unit 1006 in the device 1000 for processing.

[0169] Provided Unit:

[0170] The providing unit 1006 in the device 1000 is configured to provide a simulated image of the surgical scene from one or more candidate viewpoints based on a first image of the surgical scene.

[0171] That is, in the example, the providing unit receives one or more candidate viewpoints from the determining unit 1004 and a first image from the first receiving unit 1002, and uses this information to generate simulated images of the surgical scene for each candidate viewpoint. These simulated images provide a predicted appearance of how the scene will look from the candidate viewpoints (and are obtained without actually changing the image capture properties of the image capture device at this stage). These generated images are then provided for selection.

[0172] Moreover, in other examples, it should be recognized that the providing unit in device 1000 can be configured to receive (via an external computing device) simulated images of previously generated scenes and provide these simulated images directly to the surgeon for selection.

[0173] Reconsider this disclosure Figure 4 An exemplary scenario is described below. In this example, device 1000 has already received image 900 (refer to the present disclosure). Figure 5 (As shown) serves as the first image of the scene. This first image of the scene is hampered by multiple reflections from the surface of the tissue, thus hindering the surgeon 804 from obtaining a clear image of the target area 808. Moreover, in this example, the determining unit 1004 has determined, from the additional information of the scene received by the first receiving unit 1002, the selection of three candidate viewpoints for the surgical procedure corresponding to the surgical procedure performed by the surgeon 804. That is, the advantage is that the amount of glare or reflection from the tissue surface is reduced, as learned from previous surgical procedures.

[0174] Accordingly, in this example, as predicted, providing unit 1006 generates simulated images of the surgical scene as if the scene had been displayed from each determined candidate viewpoint. These images are generated based on a first image of scene 900 received by the first receiving unit. It should be understood that providing unit 1006 generates simulated images with the aim of reproducing the favorable robot viewpoint within the context of the current surgical scene as much as possible.

[0175] Figure 8 An exemplary example image of a simulated viewpoint is shown in the figure.

[0176] In this example, simulated image 1200 is a simulated image of a first candidate viewpoint determined by determining unit 1004. This first candidate viewpoint refers to a viewpoint that uses hyperspectral imaging to reduce reflections from the tissue surface. Accordingly, simulated image 1200 shows a prediction of how the target region 808 of the patient would appear when using this hyperspectral imaging.

[0177] The simulated image 1202 is a simulated image from a second candidate viewpoint determined by the determining unit 1004. The second candidate viewpoint refers to a viewpoint where the image capturing device captures an image from a second physical location within the surgical environment (a physical location different from the current physical location of the image capturing device).

[0178] Accordingly, the simulated image 1202 shows a prediction of how the patient's target area 808 would look when the image is captured from the second physical location within the surgical environment.

[0179] Finally, the simulated image 1204 is a simulated image from a third candidate viewpoint determined by the determining unit 1004. This third candidate viewpoint refers to a viewpoint where the image capturing device captures an image from a third physical location (different from the current physical location and the physical location of the second candidate viewpoint). Accordingly, the simulated image 1204 shows a prediction of how the patient's target region 808 will appear when the image is captured from this third physical location within the surgical environment.

[0180] For all three simulated images 1200, 1202, and 1204, the glare and reflectance from the patient's tissue are less than those presented in the current image of scene 900 (refer to this disclosure). Figure 5 (As shown).

[0181] In some examples, when generating a simulated image of the scene, the providing unit 1006 may also utilize additional information received by the first receiving unit 1002 in the device 1000.

[0182] For example, when generating a simulated image of a scene from candidate viewpoints, information about the surgical environment, such as the corresponding orientation of components within the surgical scene, can be used.

[0183] Now, in embodiments of this disclosure, the ability of an artificial intelligence system to simulate unseen viewpoints of a scene is used to generate simulated images of the scene from a first image of the scene based on determined candidate viewpoints. That is, it is known that an artificial intelligence system can see a scene from a specific first angle (in this example, corresponding to the viewpoint of the first image 900) and predict what the same scene will look like from another unobserved angle (in this example, corresponding to simulated images 1200, 1202, and 1204).

[0184] In a specific example, this can be achieved using a machine learning system trained on previous viewpoints of a surgical scene; this can include previous viewpoints of the surgical scene used in previous surgical procedures and can also include one or more viewpoints used when simulating a surgical scene.

[0185] In specific situations, deep learning models (as an example of machine learning systems) can be used to generate simulated images of a scene. These deep learning models are built using neural networks. These neural networks consist of an input layer and an output layer. Multiple hidden layers lie between the input and output layers. Each layer contains multiple independent nodes. Nodes in the input layer connect to nodes in the first hidden layer. Nodes in the first hidden layer (and each subsequent hidden layer) connect to nodes in subsequent hidden layers. Nodes in the final hidden layer connect to nodes in the output layer.

[0186] In other words, each node in a layer is connected back to all nodes in the previous layer of the neural network.

[0187] Of course, it should be recognized that the number of hidden layers used in the model and the number of nodes in each layer can vary depending on the size of the training data and the various requirements of the simulated images of the scene.

[0188] Now, each node receives multiple inputs and generates an output. The inputs provided to the nodes (through connections to previous layers in the neural network) have weighting factors applied to them.

[0189] In a neural network, the input layer receives multiple inputs (which may include a first image of the scene). These inputs are then processed in hidden layers using weights that are adjusted during training. Finally, the output layer generates predictions from the neural network.

[0190] Specifically, during training, the training data can be split into input and target. Input data refers to all data except for the target (an image of the scene the neural network is trying to predict).

[0191] Then, during training, the input data is analyzed through the neural network to adjust the weights between the corresponding nodes. In the example, this adjustment of weights can be achieved during training using a linear regression model. However, in other examples, a non-linear method can be used to adjust the weights between nodes to train the neural network.

[0192] In practice, during training, the weighting factors applied to the nodes in the neural network are adjusted for the provided input data to determine the values ​​of the weighting factors that best match the target data. That is, during training, input and target output are provided. The network then processes the input and compares the generated output with the target data. The difference between the output and the target data is then propagated back through the neural network, causing the network to adjust the weights of the corresponding nodes in the network.

[0193] Of course, the number of training cycles (or periods) used to train the model can vary depending on the situation. In some examples, the model can be continuously trained based on the training data until it generates output within a predetermined threshold of the target data.

[0194] Once trained, new input data can be fed into the input layer of the neural network, causing the model to generate a predicted output (based on the weights applied to each node in the neural network during training) for the given input data.

[0195] Of course, it should be recognized that this implementation is not specifically limited to deep learning models (such as neural networks) and any such machine learning algorithm according to the implementation of this disclosure can be used as appropriate.

[0196] In some examples, a Generative Query Network (GQN) can be used to generate simulated images of the scene. In this example, the network acquires images from viewpoints within the scene. That is, the GQN acquires images of the surgical scene from the initial location (i.e., the first image of the scene). However, in other examples, additional images of the scene describing how it looks from other angles can be obtained from other image capture devices within the surgical environment.

[0197] Alternatively, during initial calibration before the start of the surgical procedure, additional images of the scene can be acquired by a first image capture device. As the camera moves to its initial position to capture an image of the patient's target area 808, the image capture device can capture images of the surgical scene from slightly different angles (i.e., as the image capture device moves to its initial position). These images can be stored to assist in subsequent viewpoint generation. For example, depending on the data storage capacity of the surgical facility, the range of stored images can range from a few frames to a complete motion playback. Thus, images of the scene from multiple viewpoints can be obtained. In a particular example, the device 1000 can be further configured to use this information to generate a map of the surgical environment as it moves to a certain position. This can be achieved using a Simultaneous Localization and Mapping (SLAM) algorithm.

[0198] Now, the initial images or images acquired by the image capture device during the initial calibration constitute the observation set of GON. Each additional observation (i.e., each additional image of the scene from a different viewpoint) enables GQN to further accumulate evidence about the content of the scene.

[0199] Then, the GQN trained on the surgical scene is able to generate simulated images of the scene from one or more candidate viewpoints determined by the determination unit 1004 in the device 1000.

[0200] However, it should be recognized that GQN is merely one example of an artificial intelligence imaging system used to generate simulated images of a scene according to embodiments of this disclosure. Any other type of artificial intelligence system can be used as needed to generate simulated images of candidate viewpoints for a scene.

[0201] Reconsider referencing this disclosure Figure 4 The described exemplary scenario. In this example, once unit 1006 generates three simulated images of a surgical scene ( Figure 8 The image provided (images 1200, 1202, and 1204) shown in the figure are transmitted to the surgeon 804 by the providing unit for display.

[0202] In the example, unit 1006 may provide an interface (“user interface”) for the surgeon 804 to interact with the simulation of candidate viewpoints. In this disclosure Figure 9 An exemplary diagram of the user interface 1300 is shown. The user interface 1300 can be displayed on a display screen present in the operating room (such as a display screen used by a surgeon to perform surgical procedures (i.e., a display image showing a first image of the scene)). That is, once simulated images of the scene from candidate viewpoints are generated (showing how the predicted scene will look from these candidate viewpoints), the device 1000 is configured to provide the simulated images to the surgeon for review.

[0203] In this example, the user interface 1300 provided to surgeon 804 includes a first area displaying the current field of view of scene 900 (i.e., a first image captured by an image capture device). This is the viewpoint used by surgeon 804 to currently perform surgical procedures on the patient. A second area of ​​the user interface is also provided, which displays simulations 1200, 1202, and 1204 of candidate viewpoints generated by providing unit 1006 in device 1000.

[0204] Therefore, surgeon 804 can view simulated images of candidate viewpoints generated by device 1000 from the user interface and can assess whether these viewpoints provide a favorable reduction in glare and reflections currently experienced by the tissue of target region 808 (as seen in image 900). This allows surgeon 804 to assess whether a better view of the patient's target region 808 can be achieved via the image capture device without any delay in the surgical procedure (because the image capture device is still in the initial image capture position when the simulated images of candidate viewpoints are generated).

[0205] In some examples, when it is determined that the surgeon can gain an advantage from a candidate viewpoint, the device 1000 can use the user interface 1300 to autonomously suggest candidate viewpoints to the surgeon.

[0206] Alternatively, in other examples, the user interface can integrate call / request functionality, allowing the surgeon to instruct the system to generate and provide one or more candidate viewpoints for display.

[0207] For example, this can be especially useful when notifying a surgeon that an image provided by an image capture device has degraded.

[0208] For each candidate viewpoint presented to the surgeon, the providing unit 1006 in device 1000 may also provide one or more other pieces of information about the candidate viewpoint. This other information may include information about the relationship between the current viewpoint and the candidate viewpoint (this could be through schematic communication indicating the path taken by the image capture device from the current viewpoint to the candidate viewpoint and / or a digital description of the path taken by the image capture device from the current viewpoint to the candidate viewpoint), and the purpose of generating the candidate viewpoint (primarily the advantages gained by employing the candidate viewpoint (e.g., this could include a numerical improvement in expected image quality)).

[0209] Of course, when referring to the contents of this disclosure Figure 9 While an exemplary user interface is shown, the embodiments of this disclosure are not intended to be specifically limited to this aspect.

[0210] Alternatively, candidate viewpoints can be presented to the surgeon via, for example, a picture-in-picture (PinP) function integrated with the surgical camera display or via a separate display screen or method.

[0211] In fact, any method according to embodiments of this disclosure that enables surgeons to see simulated images generated by device 1000 can be used.

[0212] Thus, the providing unit 1006 provides a realistic visualization of the viewpoint to simulate the appearance of the scene from one or more candidate viewpoints determined by the determining unit 1004.

[0213] Control unit:

[0214] At this stage, the image capture device in the computer-aided camera system remains in its initial position (i.e., a still captured image from the initial viewpoint of the scene); a simulated image has been generated based on a prediction of how the scene would look from that candidate position without moving the camera. However, upon receiving a selection of one of the simulated images of the surgical scene, the control unit 1008 in the device 1000 is configured to control the image capture device to acquire an image of the surgical scene from the candidate viewpoint corresponding to the selection of one of the simulated images of the surgical scene.

[0215] The manner in which the selection of one of one or more simulated images of a surgical scene provided by the providing unit 1006 is not specifically limited.

[0216] In the example, the control unit 1008 is configured to receive a selection of one of one or more simulated images of a surgical scene from a surgeon, medical assistant, or personnel.

[0217] That is, in the example, the surgeon is able to interact with the user interface to select a simulated image of a candidate viewpoint. This simulated image may be a simulated image of the candidate viewpoint that the surgeon wants the image capture device to move to (so that an actual image of the scene can be obtained from the candidate viewpoint).

[0218] That is, the surgeon 804 can use the user interface to accept or select simulated images of candidate viewpoints suggested by the system (“preferred viewpoints”).

[0219] Optionally, the surgeon 804 can select multiple preferred nodes, which the system can save and apply upon request by the surgeon. That is, the surgeon can indicate that they wish to store viewpoints for use in subsequent surgical procedures. Alternatively, the surgeon can indicate that they wish to use a first candidate viewpoint in a first time period, and then a second candidate viewpoint in a later stage of the procedure.

[0220] In some examples, the control unit may be configured to accept touch input on the user interface 1300 as the surgeon 804's selection of simulated images of candidate viewpoints. In other examples, the surgeon may be able to provide voice input as the selection of one or more simulated images of candidate viewpoints (e.g., "select simulated image number one").

[0221] In fact, any such configuration according to the embodiments of this disclosure, which enables the control unit to receive the surgeon's selection of one or more simulated images of a surgical scene, can be used as needed.

[0222] In a specific example, the control unit is configured to determine a candidate image corresponding to a simulated image selected by the surgeon, and to perform one or more operations to control the image capture device of the computer-aided camera system such that the image capture device is reconfigured to capture an image of the patient's target region 808 using a candidate viewpoint corresponding to the simulated image selected by the surgeon.

[0223] In a specific example, the control unit may perform camera actuation processing to physically move the image capture device to a position corresponding to a selected candidate viewpoint. The image capture device then captures subsequent images of the scene from this actual, real-world position (corresponding to the candidate position selected by the surgeon). As part of the camera actuation processing, the surgeon or support personnel may manually move the image capture device after navigation guidance is provided by device 1000. In this case, the navigation guidance can be communicated to the surgeon or support personnel via user interface 1300. Alternatively, the image capture device may be moved autonomously by a surgical robot after the surgeon has verified the expected movement (as needed). That is, in some examples, the control unit may be configured to control the position and / or orientation of the articulated arm supporting the image capture device to control the image capture device to acquire images of the surgical scene from a candidate viewpoint corresponding to the selection of one of one or more simulated images of the surgical scene.

[0224] In other examples, the control unit may perform camera modulation processing to reconfigure one or more image capture properties of the image capture device (such as zoom level), so that the image capture device may subsequently use the real-world reconfiguration to capture subsequent images of the scene.

[0225] Consider again the reference in this disclosure Figure 4 The exemplary scenario described herein. Here, after viewing three simulated images 1200, 1202, and 1204 generated by device 1000, surgeon 804 selects candidate viewpoint 1202 as the viewpoint from which he / she wishes the image capture device to capture subsequent images of the scene. Accordingly, control unit 1008 controls the image capture device of the computer-aided camera system to capture subsequent images of the target region 808 from the selected candidate viewpoint.

[0226] exist Figure 10 An exemplary diagram is shown of a real image 1400 captured by an image capture device after a candidate viewpoint is selected (i.e., the simulated image 1202 corresponding to candidate viewpoint 2 is selected).

[0227] That is, in contrast to the simulated image 1202 (generated by the providing unit 1006 without actuating the image capturing device) which constitutes a prediction of how the target image will appear from the third candidate viewpoint, image 1400 shows the image actually captured by the image capturing device after the viewpoint has been moved to the third candidate viewpoint. Accordingly, because it relates to the actual image of the patient's target area, the surgeon 804 is able to use the actual image 1400 to perform surgical procedures on the patient.

[0228] In image 1400, the target region 808 of patient 802 is shown. However, unlike the first image of scene 900 (i.e., the image of the target region 808 captured from the initial position of the image capturing device), image 1400 provides the surgeon with a clear image of the patient's target region 808. That is, compared to image 900, the amount of glare and reflection received from the tissue of the target region in image 1400 is significantly reduced.

[0229] Thus, the control unit in device 1000 controls the image capture device so that the image capture device captures a real image of the scene corresponding to the selected simulated image.

[0230] Beneficial effects:

[0231] According to embodiments of the present disclosure, the apparatus for controlling an image capture device during surgery enables the surgeon to consider multiple alternative viewpoints of the computer-aided camera system during surgery without repositioning the camera to consider alternative viewpoints, thereby enabling the optimization of the viewpoint strategy of the computer-aided camera system without causing unnecessary delays in the surgical procedure.

[0232] Furthermore, candidate viewpoints that the surgeon himself could not have conceived of can be presented to the surgeon. Therefore, these candidate viewpoints can provide surprising benefits that the surgeon had not previously considered, such as improved surgical performance or reduced surgical duration. Specifically, embodiments of this disclosure enable human surgeons to benefit from viewpoint strategies developed by other human or robotic surgeons.

[0233] Of course, this disclosure is not specifically limited to these advantageous technical effects, and other effects may become apparent to those skilled in the art when reading this disclosure.

[0234] Additional deformations:

[0235] Although reference has been made to this disclosure above Figures 4 to 10 The configuration of the device 1000 has been described; however, it should be understood that embodiments of this disclosure are not limited to this specific example. For example, embodiments of this disclosure can be applied to image capture devices, such as endoscopic image capture devices, telescopic image capture devices, microscopic image capture devices, etc., depending on the surgical procedure being performed.

[0236] Furthermore, several additional variations in the configuration of the device are described below. Figure 11 An apparatus 1000 for controlling an image capture device during surgical procedures is shown according to these embodiments of the present disclosure.

[0237] <Strengths Assessment Unit>

[0238] In some optional examples, the device 1000 may be further configured to include a dominance evaluation unit 1010. The dominance evaluation unit 1010 may be configured to evaluate one or more quantifiable features of a simulated image of candidate viewpoints and arrange candidate viewpoints based on the evaluation results. For example, candidate viewpoints for which the dominance evaluation unit evaluates a more favorable viewpoint for the surgeon may be arranged in a more prominent position on the display.

[0239] That is, the providing unit 1006 can be configured to additionally provide simulated images of candidate viewpoints to the advantage evaluation unit 1010, so that the advantage evaluation unit can arrange the candidate viewpoints corresponding to these simulated images on the display based on the quantifiable benefits generated by the surgeon. Once the advantage evaluation unit 1010 evaluates the candidate viewpoints, it can return this information to the providing unit 1006, so that the providing unit can provide the surgeon with information about the beneficial effects of each candidate viewpoint. Alternatively, or additionally, the providing unit 1006 can use information from the advantage evaluation unit 1010 when determining which candidate viewpoints to provide to the surgeon. Alternatively, or additionally, the providing unit 1006 can use information from the advantage evaluation unit 1010 when determining the order in which simulated images corresponding to the candidate viewpoints are provided to the surgeon.

[0240] In the example, the advantage evaluation unit 1010 can determine the advantage of each viewpoint in relation to the first image received by the first receiving unit 1002 (i.e., in relation to the current image of the scene obtained by the image capture device).

[0241] In the example, the advantage evaluation unit 1010 can evaluate candidate viewpoints based on scores of quantifiable features assigned to simulated images of a surgical scene. These features may include, for example, a percentage increase in visibility of the surgeon's work area or critical tissue areas; a percentage decrease in light reflection or glare; a percentage increase in image contrast and / or sharpness; a percentage increase in the available range / degree of movement of one or more surgical tools within the surgical scene; a decrease in the probability of collisions between image capture devices and one or more tools within the surgical scene, etc. Each of these features can be weighted according to the situation, and the simulated image with the highest cumulative score is evaluated by the advantage evaluation unit 1010 as the surgeon's most favorable candidate viewpoint. The advantage evaluation unit 1010 can evaluate these features using any suitable image processing techniques as needed.

[0242] Alternatively, in the example, the unpredictability of candidate viewpoints can be calculated in the evaluation performed by the advantage evaluation unit 1010. That is, one or more candidate viewpoints determined by the determining unit 1004 (whose simulated images have been generated by the providing unit 1006) can be compared with the surgeon's viewpoint preferences and / or the surgeon's specific viewpoint history (indicating the image-capture viewpoints that the surgeon typically prefers to use at a given stage of a given surgical procedure). Because these viewpoints may provide the surgeon with the most surprising benefit (advantageous viewpoints that the surgeon had not previously conceived of for the surgical procedure), the advantage evaluation unit 1010 may rank advantageous viewpoints with higher contrast to viewpoints typically chosen by the surgeon as the highest.

[0243] Furthermore, for a given stage of the surgical procedure, candidate viewpoints can be compared with a database of viewpoints commonly used by human surgeons worldwide, so that the advantage evaluation unit 1010 can identify a viewpoint that surprises or is unexpected by most human surgeons (and not just the surgeon currently performing the surgical procedure) at the same time that is known to the computer-aided surgical system (such as a robotic surgeon).

[0244] In the example, the advantages identified by the advantage evaluation unit 1010 and actually communicated to the surgeon by the providing unit 1006 may vary depending on the surgeon's level of experience and / or training. Novice surgeons who need help finding a good viewpoint in a surgical scene may be particularly concerned with the interaction between the image capture device and surgical instruments and may therefore require more workspace. Therefore, when scoring candidate viewpoints in such situations, a higher weighting factor can be applied to the workspace by the advantage evaluation unit.

[0245] Alternatively, the surgeon may use a computer-aided surgical device with greater degrees of freedom in the image capture device than the computer-aided surgical system the surgeon has previously experienced, and therefore, the surgeon may not be aware of the additional advantageous viewpoints that can increase the range of motion; these additional advantageous viewpoints can preferably be communicated to the surgeon. That is, therefore, when scoring candidate viewpoints in such a situation, a higher weighted factor viewpoint utilizing the enhanced degrees of freedom of the image capture device can be applied by the advantage evaluation unit.

[0246] <Viewpoint Adjustment Unit>

[0247] In some optional examples, the device 1000 may be further configured to include a viewpoint adjustment unit 1012. The viewpoint adjustment unit 1012 may be configured to receive information from the providing unit 1006 about simulated images of candidate viewpoints that have been provided to the user.

[0248] The viewpoint adjustment unit is configured to enable the surgeon to modify one or more properties of the selected candidate viewpoint before the image capture device moves to the new viewpoint.

[0249] In some examples, the viewpoint adjustment unit 1012 may be configured to receive an interaction with a simulated image of a surgical scene and update one or more properties of the corresponding candidate viewpoint based on the interaction.

[0250] Consider again the reference in this disclosure Figure 4 The exemplary scenario described herein. In this example, a user interface 1300 (as described in this disclosure) will be displayed on the screen. Figure 9 (As shown) is provided to the surgeon so that the surgeon can perform the selection of a simulated image of a candidate viewpoint as the viewpoint for obtaining the actual image of the target region 808.

[0251] In this example, when the surgeon selects a candidate viewpoint, the viewpoint adjustment unit 1012 can be configured to cooperate with the providing unit 1006 to generate another user interface for the surgeon. This other user interface enables the surgeon to update one or more properties of the corresponding candidate viewpoint.

[0252] exist Figure 12 An example of this other user interface 1600 is shown in the image.

[0253] Here, at the top of the user interface 1600, a current image (first image) of scene 900 is provided to the surgeon. It is important to continue providing the surgeon with a current image of the scene so that the surgeon can focus on patient safety and the efficiency of the surgical procedure. In addition to the first image 900, the user interface 1600 also provides the surgeon with an enhanced field of view of a simulated image (selected by the surgeon) generated by the providing unit. In this specific example, the surgeon selects simulated image 1202 as a candidate viewpoint of interest.

[0254] Furthermore, the user interface 1600 provides one or more candidate viewpoint adjustment tools 1602 to the surgeon. These candidate viewpoint adjustment tools 1602 enable the surgeon to manipulate simulated images of candidate viewpoints generated by the providing unit 1006. For example, the surgeon can use a candidate viewpoint adjustment tool to zoom in on a target area. In this case, the viewpoint adjustment unit is configured to simulate the candidate viewpoints presented to the user and update one or more properties of the candidate viewpoints (in this specific example, the zoom level used in the candidate viewpoint). Other properties of the candidate viewpoints may include the position of the candidate viewpoint, the aperture of the candidate viewpoint, the image modality of the candidate viewpoint, etc.

[0255] In some embodiments, the providing unit in device 1000 generates a simulated image of the scene using updated properties of candidate viewpoints and provides it to the surgeon. That is, in a particular example, the circuitry is configured to receive interaction with a simulated image of the surgical scene and update one or more properties and / or the simulated image of the corresponding candidate viewpoint of the surgical scene based on the interaction.

[0256] Accordingly, once the surgeon confirms the selection, the control unit 1008 is configured to control the image capture device to capture an image from the selected candidate viewpoint adjusted by the surgeon. Specifically, in this example, the control unit controls the image capture device to capture an image from a second candidate viewpoint (corresponding to the analog image 1202) with an enhanced zoom level (corresponding to the adjustment performed by the surgeon).

[0257] In other words, the viewpoint adjustment unit 1012 enables the surgeon to manually adjust the selected candidate viewpoint according to their specific preferences. This ensures that the viewpoint provided by the image capture device is comfortable for the surgeon's operation, while allowing the surgeon to benefit from the candidate viewpoint.

[0258] <Compatibility Assessment Unit>

[0259] In some optional examples, the device 1000 may be further configured to include a compatibility evaluation unit 1014. For example, the compatibility evaluation unit may receive a list of candidate viewpoints determined by the determination unit 1004.

[0260] In a specific example, the compatibility evaluation unit 1014 can be configured to determine the image capture device's ability to implement the candidate viewpoints generated by the determining unit and to exclude candidate viewpoints unsuitable for the image capture device. That is, due to limitations of the workspace around the image capture device, the compatibility evaluation unit 1014 can determine that the image capture device cannot implement a given candidate viewpoint in a specific surgical scenario. Then, before generating an image simulation of the scene obtained from the candidate viewpoints, the compatibility evaluation unit 1014 can remove candidate viewpoints that the image capture device cannot implement from the list of candidate viewpoints. Thus, processing resources for generating simulated images that the image capture device cannot implement are not used.

[0261] In other examples, the compatibility evaluation unit 1014 may be configured to evaluate the capability of candidate viewpoints used by the surgeon and exclude those that are unsuitable for use by the surgeon in a surgical scenario. That is, a particular candidate viewpoint advantageous to a computer-aided surgical system (such as a robotic surgeon) may be too complex for a human surgeon to understand. This may be the case, for example, if the viewpoint is a dynamic viewpoint where the scene changes rapidly. Thus, the compatibility evaluation unit 1014 can remove viewpoints impractical for human use from the list of candidate units generated by the determining unit in device 1000.

[0262] In some examples, the compatibility evaluation unit 1014 can be configured to identify specific candidate viewpoints that are incompatible with human surgeons in their form, and these viewpoints can be adjusted through one or more modifications to make them compatible with human surgeons. For example, the compatibility evaluation unit 1014 can adapt specific dynamic robot viewpoints to make the dynamic viewpoints practically usable for human use. For example, the compatibility evaluation unit 1014 can achieve this by reducing the movement rate of the image capture device, thereby reducing the number of unequal viewpoints used and / or minimizing the switching frequency between different visual modalities.

[0263] In this way, while still providing human surgeons with comparable benefits related to candidate viewpoints, viewpoints optimized for computer-aided surgical devices can be adapted to improve the usability of viewpoints for human surgeons.

[0264] Example settings:

[0265] Refer to the contents of this disclosure Figure 13 An exemplary setup of a computer-aided surgical system according to embodiments of the present disclosure is shown. This system can be used in endoscopic surgical situations (as described in the present disclosure). Figure 1 (as described), or can be used alternatively in primary and secondary surgical situations (as described in this disclosure). Figure 3 The exemplary setup described herein, or alternatively used in the use of a microscope or exoskeleton, may be employed.

[0266] This exemplary setup for controlling an image capture device during surgical procedures, according to embodiments of this disclosure, can be used.

[0267] In this example, a scene evaluation system (such as a first receiving unit 1002) receives context information and first image information from a surgical scene 1702.

[0268] The scenario evaluation system is configured to use the information received from the surgical scenario 1702 to determine the surgical stage (i.e., the surgical procedure performed by the surgeon, and the stage of the surgical procedure (such as the initial, intermediate, or final stage of the surgical procedure)).

[0269] Then, the scene evaluation system provides information about the surgical phase to an alternative viewpoint generation system (e.g., such as determining unit 1004 and providing unit 1006).

[0270] Then, the alternative viewpoint generation system 1704 receives robot viewpoints from the robot viewpoint database 1710. These are viewpoints used by the robotic surgical system (in the form of a computer-aided surgical system) in a prior surgical procedure corresponding to the surgery performed by the surgeon. The robot viewpoint generation algorithm 1706 then uses these viewpoints to generate a simulated image of multiple robot viewpoints (i.e., a simulated image of how the surgical scene looks from a specific robot viewpoint retrieved from the robot viewpoint database).

[0271] Optionally, these simulated images are passed to a surprise viewpoint selection algorithm 1708 configured to select from multiple most surprising viewpoints among viewpoint candidates provided to the surgeon.

[0272] The selected candidate viewpoints are then provided to the surgeon using the user interface 1712. Thus, the surgeon is able to observe how the images from the image capture device appear from these selected candidate viewpoints without moving the image capture device and without interrupting the surgical procedure.

[0273] Once the surgeon selects one or more preferred viewpoints from the viewpoints displayed on the user interface, the camera actuation processing 1714 is configured to control the image capture device of the computer-aided surgical system such that the image capture device is configured to capture subsequent images of the scene from a real-world viewpoint corresponding to the virtual candidate viewpoint selected by the surgeon.

[0274] In this way, surgeons can consider multiple alternative viewpoints of the computer-aided camera system during surgery without having to repeatedly reposition the camera to consider alternative viewpoints, thereby enabling the optimization of the viewpoint strategy of the computer-aided camera system without causing unnecessary delays in the surgical procedure.

[0275] method:

[0276] According to embodiments of this disclosure, a method for controlling a medical image capture device during surgical procedures is provided. Reference is made to this disclosure. Figure 14A method for controlling a medical image capture device is shown.

[0277] The method begins at step S1800 and proceeds to step S1802.

[0278] In step S1802, the method includes: receiving a first image of a surgical scene captured from a first viewpoint by a medical image capture device, along with additional information about the scene.

[0279] Once the image and additional information are received, the method proceeds to step S1804.

[0280] In step S1804, the method includes: determining one or more candidate viewpoints for a medical image capture device based on additional information and previous viewpoint information of the surgical scene, the medical image capture device obtaining an image of the surgical scene from the one or more candidate viewpoints.

[0281] Once candidate viewpoints are determined, the method proceeds to step S1806.

[0282] In step S1806, the method includes: providing a simulated image of the surgical scene from each of one or more candidate viewpoints, based on a first image of the surgical scene.

[0283] Once a simulated image of the surgical scene is provided, the method proceeds to step S1808.

[0284] In step S1808, the method includes: controlling a medical image capture device to obtain an image of the surgical scene from a candidate viewpoint corresponding to the selection of one of one or more simulated images of the surgical scene.

[0285] Then, the method proceeds to step S1810 and ends at step S1810.

[0286] It should be recognized that in some cases, once step S1810 is completed, the method returns to step 1802. Thus, the desired image capture properties of the image capture device are continuously and periodically evaluated and updated as needed.

[0287] Computer equipment:

[0288] For reference Figure 15 The diagram illustrates a computing device 1900 according to an embodiment of the present disclosure. The computing device 1900 may be a computing device for controlling an image capture device during surgical procedures. Typically, the computing device may be a device such as a personal computer or terminal connected to a server. Indeed, in this embodiment, the computing device may also be a server. The computing device 1900 is controlled using a microprocessor or other processing circuitry 1902.

[0289] The processing circuit 1902 may be a microprocessor that executes computer instructions or an application-specific integrated circuit. The computer instructions are stored on a storage medium 1904, which may be a magnetically readable medium, an optically readable medium, or a solid-state circuit.

[0290] Storage medium 1904 may be integrated into computing device 1900 (as shown) or may be detached from computing device 1900 and connected to it using wired or wireless connections.

[0291] The computer instructions may encompass computer software containing computer-readable code, which, when loaded into the processor circuitry 1902, configures the processor circuitry 1902 of the computing device 1900 to perform a method for controlling an image capture device during surgical procedures according to embodiments of the present disclosure. Furthermore, connected to the processor circuitry 1902 is a user input (not shown). The user input may be a touchscreen or a mouse or stylus-type input device. The user input may also be a keyboard or any combination of these devices.

[0292] Network connection 1906 is also coupled to processor circuitry 1902. Network connection 1906 can be a connection to a local area network (LAN) or a wide area network (WAN) such as the Internet or a virtual private network (VPN). Network connection 1906 can connect to medical device infrastructure to allow processor circuitry 1902 to communicate with other medical devices to obtain or provide relevant data to other medical devices. Network connection 1906 can be located behind a firewall or some other form of network security.

[0293] Furthermore, coupled to the processing circuitry 1902 is a display device 1908. Although shown as integrated into the computing device 1900, the display device 1908 may be additionally separate from the computing device 1900 and may be a monitor or other device that allows users to visualize the operation of the system. Additionally, the display device 1908 may be a printer or other device that allows users or third parties (such as medical support personnel) to view relevant information generated by the computing device 1900.

[0294] Although described for the foreshadowing with reference to "master-slave" robotic systems, this disclosure is not limited thereto. In some instances, surgical robots can operate independently of a human surgeon with the supervision of that surgeon. Furthermore, for endoscopy or laparoscopy, the endoscopist can be a robot, and the human surgeon guides the robot. In embodiments, the robotic system can be a multi-robot surgical system, in which the primary surgeon uses a robotic surgeon and an assistant surgeon remotely manipulates an auxiliary robotic arm. The robotic system can also be a single-person surgical system consisting of a pair of jointly operating and autonomous robotic arms that hold surgical instruments. In this case, the human surgeon can use a master-slave arrangement.

[0295] <Exemplary System>

[0296] Figure 16 An example of a computer-aided surgical system 11260 to which this technology can be applied is schematically illustrated. The computer-aided surgical system is a master-slave system integrating an autonomous arm 11000 and one or more surgeon-controlled arms 11010. The autonomous arm holds an imaging device 11020 (e.g., a medical endoscope such as an endoscope, microscope, or external speculum). Each of the one or more surgeon-controlled arms 11010 holds a surgical device 11030 (e.g., a cutting tool, etc.). The imaging device of the autonomous arm outputs images of the surgical scene to an electronic display 11100 observable by the surgeon. Although the surgeon performs surgery using one or more surgeon-controlled arms, the autonomous arm autonomously adjusts the field of view of the imaging device to provide the surgeon with an appropriate view of the surgical scene in real time.

[0297] A surgeon uses a main console 11040 to control one or more surgeon-controlled arms 11010. The main console includes a main controller 11050. The main controller 11050 includes one or more force sensors 11060 (e.g., torque sensors), one or more rotation sensors 11070 (e.g., encoders), and one or more actuators 11080. The main console includes an arm (not shown) comprising one or more joints and an operating part. The surgeon can grasp and move the operating part, causing the arm to move about one or more joints. One or more force sensors 11060 detect the force applied by the surgeon to the operating part of the arm about one or more joints. One or more rotation sensors detect the rotation angle of one or more joints of the arm. The actuators 11080 drive the arm about one or more joints to allow the arm to provide tactile feedback to the surgeon. The main console includes Natural User Interface (NUI) inputs / outputs for receiving input information from the surgeon and / or providing output information to the surgeon. The NUI inputs / outputs include the arm (the surgeon moves the arm to provide input information and the arm provides tactile feedback as output information to the surgeon). NUI input can also include voice input, gaze input, and / or gesture input.

[0298] The main control console includes an electronic display 11100 for outputting images captured by the imaging device 11020.

[0299] The main console 11040 communicates with each of the autonomous arm 11000 and one or more surgeon-controlled arms 11010 via the robot control system 11110. The robot control system is connected to the main console 11040, the autonomous arm 11000, and one or more surgeon-controlled arms 11010 via wired or wireless connections 11230, 11240, and 11250. Connections 11230, 11240, and 11250 allow the exchange of wired or wireless signals between the main console, the autonomous arm, and one or more surgeon-controlled arms.

[0300] The robot control system includes a control processor 11120 and a database 11130. The control processor 11120 processes signals received from one or more force sensors 11060 and one or more rotation sensors 11070 and outputs control signals that drive one or more surgeon-controlled arms 11010 in response to the one or more force sensors 11060. Thus, movement of the operating part of the main console 11040 causes a corresponding movement of one or more surgeon-controlled arms.

[0301] The control processor 11120 also outputs control signals to drive the autonomous arm 11000 in response to one or more actuators 11160. The control processor 11120 determines the control signals output to the autonomous arm in response to signals received from one or more master consoles 11040, one or more surgeon-controlled arms 11010, the autonomous arm 11000, and any other signal sources (not shown). The received signals are signals indicative of the appropriate position of the autonomous arm so that the imaging device 11020 can capture images from the appropriate angle. The database 11130 stores the values ​​of the received signals and the corresponding positions of the autonomous arm.

[0302] For example, for a given combination of signal values ​​received from one or more force sensors 11060 and rotation sensors 11070 in the main controller (which in turn indicate the corresponding movement of one or more surgeon-controlled arms 11010), the corresponding position of the autonomous arm 11000 is set such that the image captured by the imaging device 11020 is not obstructed by one or more surgeon-controlled arms 11010.

[0303] As another example, if a signal output by one or more force sensors 11170 (e.g., torque sensors) in the autonomous arm indicates that the autonomous arm is experiencing resistance (e.g., due to an obstacle in the path of the autonomous arm), the corresponding position of the autonomous arm is set such that the imaging device 11020 captures an image from an alternative field of view (e.g., a field of view that allows the autonomous arm to move along an alternative path that does not involve obstacles).

[0304] It should be recognized that other types of received signals may exist to indicate the proper position of the autonomous arm.

[0305] The control processor 11120 queries the database 11130 for the value of the received signal and retrieves information indicating the corresponding position of the autonomous arm 11000. This information is then processed to generate additional signals that, in response to the actuator 11160 of the autonomous arm, cause the autonomous arm to move to the indicated position.

[0306] Each of the autonomous arm 11000 and one or more surgeon-controlled arms 11010 includes an arm unit 11140. The arm unit includes an arm (not shown), a control unit 11150, one or more actuators 11160, and one or more force sensors 11170 (e.g., torque sensors). The arm includes one or more links and joints that allow movement of the arm. The control unit 11150 transmits signals to and receives signals from the robot control system 11110.

[0307] In response to signals received from the robot control system, the control unit 11150 controls one or more actuators 11160 to move one or more joint drive arms to the appropriate position.

[0308] For one or more surgeon-controlled arms 11010, the received signal is generated by the robot control system based on signals received from the main console 11040 (e.g., by the surgeon controlling the arm on the main console). For the autonomous arm 11000, the received signal is generated by the robot control system by querying the appropriate autonomous arm position information in the database 11130.

[0309] In response to signals output by one or more force sensors 11170 surrounding one or more joints, control unit 11150 outputs a signal to the robot control system. For example, this allows the robot control system to transmit a signal indicating the resistance experienced by one or more surgeon-controlled arms 11010 to the main console 11040 to provide corresponding tactile feedback to the surgeon (e.g., to cause the resistance experienced by one or more surgeon-controlled arms to generate corresponding resistance in the arm of the main console actuator 11080). As another example, this allows the robot control system to query database 11130 for suitable autonomous arm position information (e.g., to identify alternative positions for the autonomous arm if one or more force sensors 11170 indicate an obstacle in its path).

[0310] The imaging device 11020 of the autonomous arm 11000 includes a camera control unit 11180 and an imaging unit 11190. The camera control unit controls the imaging unit to capture images and controls various parameters of the captured images, such as zoom level, exposure value, white balance, etc.

[0311] The imaging unit captures images of the surgical scene. The imaging unit includes all the components necessary for image capture, including one or more lenses and an image sensor (not shown). The field of view of the surgical scene in the captured images depends on the position of the autonomous arm.

[0312] A surgical device 11030 with one or more surgeon-controlled arms includes a device control unit 11200, a manipulator 11210 (e.g., including one or more motors and / or actuators), and one or more force sensors 11220 (e.g., torque sensors).

[0313] The device control unit 11200 controls the manipulator to perform physical actions (e.g., cutting actions when the surgical device 11030 is a cutting tool) in response to signals received from the robot control system 11110. Signals are generated by the robot control system in response to signals received from the main console 11040, i.e., signals generated by the surgeon who inputs information to the NUI input / output 11090 to control the surgical device. For example, the NUI input / output includes one or more buttons or joysticks that are part of the operating section of the arm of the main console, i.e., operated by the surgeon to cause the surgical device to perform predetermined actions (e.g., turning the motorized blades on or off when the surgical device is a cutting tool).

[0314] The device control unit 11200 also receives signals from one or more force sensors 11220. In response to the received signals, the device control unit provides a corresponding signal to the robot control system 11110, which in turn provides a corresponding signal to the main console 11040. The main console provides tactile feedback to the surgeon via NUI input / output 11090. Thus, the surgeon receives tactile feedback from the surgical device 11030 and from one or more surgeon-controlled arms 11010. For example, when the surgical device is a cutting tool, the tactile feedback involves buttons or joysticks that, when signals from one or more force sensors 11220 indicate a greater force on the cutting tool (occurring when cutting through harder materials, such as bone), cause the cutting tool to operate with greater resistance, and when signals from one or more force sensors 11220 indicate a lesser force on the cutting tool (occurring when cutting through softer materials, such as muscle), cause less resistance. NUI input / output 11090 includes one or more suitable motors, actuators, etc., that provide tactile feedback in response to signals received from robot control system 11110.

[0315] Figure 17 Another example of a computer-aided surgical system 12090 to which the present technology can be applied is illustrated schematically. The computer-aided surgical system 12090 refers to a surgical system in which a surgeon performs tasks via a master-slave system 11260 and a computerized surgical device 12000 performs tasks autonomously.

[0316] Master-slave system 11260 and Figure 16 The same as that in the previous description is not included. However, in alternative implementations, the system can be the same as... Figure 16 Different systems, or they can be omitted together (in which case, although the surgeon performs routine surgical procedures, system 12090 operates autonomously).

[0317] Computerized surgical apparatus 12000 includes a robot control system 12010 and a tool holding arm device 12100. The tool holding arm device 12100 includes an arm unit 12040 and a surgical instrument 12080. The arm unit includes an arm (not shown), a control unit 12050, one or more actuators 12060, and one or more force sensors 12070 (e.g., torque sensors). The arm includes one or more joints that allow arm movement. The tool holding arm device 12100 transmits signals to and receives signals from the robot control system 12010 via a wired or wireless connection 12110. The robot control system 12010 includes a control processor 12020 and a database 12030. Although shown as a separate robot control system, the robot control system 12010 and the robot control system 12010 can be one and the same. The surgical instrument 12080 has the same components as the surgical instrument 11030. Figure 17 Not shown in the image.

[0318] In response to control signals received from the robot control system 12010, the control unit 12050 controls one or more actuators 12060 to move one or more joint-driven arms to appropriate positions. Operation of the surgical device 12080 is also controlled via control signals received from the robot control system 12010. Control signals are generated by the control processor 12020 in response to signals received from one or more arm units 12040, the surgical device 12080, and any other signal sources (not shown). Other signal sources may include imaging devices that capture images of the surgical scene (e.g., imaging device 11020 of the master-slave system 11260). The values ​​of the signals received by the control processor 12020 are compared with signal values ​​stored in the database 12030 and corresponding arm position and / or surgical device operation status information. The control processor 12020 retrieves position and / or surgical device operation status information associated with the received signal values ​​from the database 12030. Then, the control processor 12020 uses the retrieved arm position and / or surgical device operation status information to generate control signals that are sent to the control unit 12050 and the surgical device 12080.

[0319] For example, if a signal received from an imaging device capturing an image of a surgical scene indicates a predetermined surgical scenario (e.g., via a neural network image classification process, etc.), the predetermined surgical scenario is queried in database 12030, and arm position information and / or surgical device operating status information associated with the predetermined surgical scenario is retrieved from the database. As another example, if a signal indicates a resistance value measured by one or more force sensors 12070 around one or more joints of arm unit 12040, the resistance value is queried in database 12030, and arm position information and / or surgical device operating status information associated with the resistance value is retrieved from the database (e.g., if the increased resistance corresponds to an obstacle in the arm path, it allows the arm position to be changed to an alternative position). Therefore, in any case, the control processor 12020 transmits a signal to the control unit 12050 to control one or more actuators 12060 to change the position of the arm to the position indicated by the retrieved arm position information, and / or transmits a signal to the surgical device 12080 to control the surgical device 12080 to enter the operating state indicated by the retrieved operating state information (e.g., if the surgical device 12080 is a cutting tool, then the electric blade is in an "on" or "off" state).

[0320] Figure 18 Another example of a computer-aided surgical system 13000 to which this technology can be applied is schematically illustrated. The computer-aided surgical system 13000 refers to a computer-aided medical endoscope system in which an autonomous arm 11000 holds an imaging device 11020 (e.g., a medical endoscope such as an endoscope, microscope, or external speculum). The imaging device of the autonomous arm outputs images of the surgical scene to an electronic display (not shown) observable by the surgeon. As the surgeon performs surgery, the autonomous arm autonomously adjusts the field of view of the imaging device to provide the surgeon with an appropriate view of the surgical scene in real time. The autonomous arm 11000 and Figure 16 The same applies to that described herein and therefore not. However, in this case, the autonomous arm is configured as part of a separate computer-assisted medical endoscope system 13000, rather than... Figure 16 It is part of the master-slave system 11260. Therefore, for example, the autonomous arm 11000 can be used in multiple different surgical settings, including laparoscopic surgery (where the endoscope is an endoscope) and open surgery.

[0321] The computer-aided medical endoscope system 13000 also includes a robot control system 13020 for controlling the autonomous arm 11000. The robot control system 13020 includes a control processor 13030 and a database 13040. Wired or wireless signals are exchanged between the robot control system 13020 and the autonomous arm 11000 via a connection 13010.

[0322] In response to a control signal received from the robot control system 13020, the control unit 11150 controls one or more actuators 11160 to drive the autonomous arm 11000 to move it to an appropriate position so that the imaging device 11020 can capture an image from the appropriate field of view. The control processor 13030 generates control signals in response to signals received from one or more arm units 11140, the imaging device 11020, and any other signal sources (not shown). The value of the signal received by the control processor 13030 is compared with signal values ​​and corresponding arm position information stored in the database 13040. The control processor 13030 retrieves the arm position information associated with the received signal value from the database 13040. The control processor 13030 then uses the retrieved arm position information to generate the control signal sent to the control unit 11150.

[0323] For example, if a signal received from imaging device 11020 indicates a predetermined surgical scenario (e.g., via a neural network image classification process, etc.), the predetermined surgical scenario is queried in database 13040, and arm position information associated with the predetermined surgical scenario is retrieved from the database. As another example, if a signal indicates a resistance value measured by one or more force sensors 11170 in arm unit 11140, the resistance value is queried in database 12030, and arm position information associated with the resistance value is retrieved from the database (e.g., if the increased resistance corresponds to an obstacle in the arm path, it allows the arm position to be changed to an alternative position). Therefore, in any case, control processor 13030 transmits a signal to control unit 11150 to control one or more actuators 1116 to change the arm position to the position indicated by the retrieved arm position information.

[0324] Figure 19Another example of a computer-aided surgical system 14000 to which the present technology can be applied is schematically illustrated. The system includes one or more autonomous arms 11000 having an imaging unit 11020, and one or more autonomous arms 12100 having a surgical instrument 12100. The one or more autonomous arms 11000 and one or more autonomous arms 12100 are the same as the previously described autonomous arms. Each autonomous arm 11000 and 12100 is controlled by a robot control system 14080 including a control processor 14090 and a database 14100. Wired or wireless signals are transmitted between the robot control system 14080 and the respective autonomous arms 11000 and 12100 via connections 14110 and 14120, respectively. The robot control system 14080 performs the functions of the previously described robot control systems 11110 and / or 13020 for controlling the respective autonomous arms 11000 and performs the functions of the previously described robot control system 12010 for controlling the respective autonomous arms 12100.

[0325] Autonomous arms 11000 and 12100 perform at least a portion of the surgical procedure entirely autonomously (e.g., when system 14000 is an open surgical system). Robotic control system 14080 controls autonomous arms 11000 and 12100 to perform predetermined actions during the surgical procedure based on input information indicating the current stage of the surgery and / or events occurring during the surgery. For example, the input information includes images captured by image capture device 11000. The input information may also include sound captured by a microphone (not shown), detections of the surgical instruments in use based on motion sensors included in the surgical instruments (not shown), and / or any other suitable input information.

[0326] The input information is analyzed using a suitable machine learning (ML) algorithm (e.g., a suitable artificial neural network) implemented by the machine learning-based surgical planning device 14020. The planning device 14020 includes a machine learning processor 14030, a machine learning database 14040, and a trainer 14050.

[0327] Machine learning database 14040 includes information indicating the classification of surgical stages (e.g., resection, organ removal, or suturing) and / or surgical events (e.g., blood or patient parameters falling outside a predetermined range) as well as prior-known input information corresponding to these classifications (e.g., one or more images captured by imaging device 11020 during each surgical stage and / or surgical event). Machine learning database 14040 is populated during the training phase by providing the information indicating each classification and the corresponding input information to trainer 14050. Trainer 14050 then uses this information to train a machine learning algorithm (e.g., to determine appropriate artificial neural network parameters using the information). The machine learning algorithm is implemented by machine learning processor 14030.

[0328] Once trained, the machine learning algorithm can classify previously unseen input information (e.g., images of newly captured surgical scenes) to determine the surgical stage and / or surgical event associated with that input. The machine learning database also includes action information instructing the respective autonomous arms 11000 and 12100 to take actions in response to the various surgical stages and / or surgical events stored in the machine learning database (e.g., controlling autonomous arm 12100 to perform a resection at a relevant location in the surgical stage “perform a resection” and controlling autonomous arm 12100 to perform appropriate cauterization for the surgical event “bleeding”). Therefore, the machine learning-based surgical planner 14020 can determine the relevant actions taken by autonomous arms 11000 and / or 12100 in response to the surgical stage and / or surgical event classification output by the machine learning algorithm. Information instructing the relevant actions is provided to the robot control system 14080, which in turn signals the autonomous arms 11000 and / or 12100 to perform the relevant actions.

[0329] The planning device 14020 may be included within a control unit 14010 having a robot control system 14080, thereby allowing direct electronic communication between the planning device 14020 and the robot control system 14080. Alternatively or additionally, the robot control system 14080 may receive signals from other devices 14070 via a communication network 14050 (e.g., the Internet). This allows for remote control of the autonomous arms 11000 and 12100 based on processing performed by these other devices 14070. In the example, device 14070 is a cloud server with sufficient processing power to quickly implement complex machine learning algorithms, thereby achieving more reliable classification of surgical stages and / or surgical events. Different machine learning algorithms may be implemented by different corresponding devices 14070 using the same training data stored in an external (e.g., cloud-based) machine learning database 14060 accessible to each device. Therefore, each device 14070 does not require its own machine learning database (such as machine learning database 14040 in planning device 14020) and can update training data for centralized use by all devices 14070. Each device 14070 still includes a trainer (such as trainer 14050) and a machine learning processor (such as machine learning processor 14030) to implement its corresponding machine learning algorithm.

[0330] Figure 20 An example of arm unit 11140 is shown. Arm unit 12040 is configured in the same manner. In this example, arm unit 11140 supports an endoscope as an imaging device 11020. However, in another example, it supports a different imaging device 11020 or a surgical device 11030 (in the case of arm unit 11140) or 12080 (in the case of arm unit 12040).

[0331] Arm unit 11140 includes a base 7100 and an arm 7200 extending from the base 7100. Arm 7200 includes a plurality of movable joints 721a to 721f and supports an endoscope 11020 at its distal end. Links 722a to 722f are generally rod-shaped members. Each of the plurality of links 722a to 722f is connected to each other via movable joints 721a to 721f, a passive sliding mechanism 7240, and a passive joint 7260. The base unit 7100 serves as a fulcrum such that the arm shape extends from the base 7100.

[0332] The position and orientation of the endoscope 11020 are controlled by driving and controlling actuators disposed in the movable joints 721a to 721f of the arm 7200. According to this example, the distal end of the endoscope 11020 is caused to enter the patient's body cavity (i.e., the treatment site) and an image of the treatment site is captured. However, the endoscope 11020 may alternatively be another device such as another imaging device or surgical instrument. More generally, a device held at the end of the arm 7200 is referred to as a distal unit or distal device.

[0333] Here, by Figure 20 The coordinate axes shown in the figure define the arm unit 7200 as follows.

[0334] Furthermore, the vertical, longitudinal, and horizontal directions are defined according to the coordinate axes. In other words, the vertical direction relative to the base 7100 mounted on the floor surface is defined as the z-axis direction and the vertical direction. Further, the direction orthogonal to the z-axis, i.e., the direction in which the arm 7200 extends from the base 7100 (in other words, the direction in which the endoscope 11020 is positioned relative to the base 7100), is defined as the y-axis direction and the longitudinal direction. Moreover, the direction orthogonal to both the y-axis and z-axis is defined as the x-axis direction and the horizontal direction.

[0335] Movable joints 721a to 721f connect the links to each other and allow them to rotate. Each movable joint 721a to 721f has an actuator and a respective rotation mechanism that is driven by the actuator to rotate about a predetermined rotation axis. For example, by controlling the rotational drive of each movable joint 721a to 721f, the drive of the arm 7200 can be controlled, causing the arm unit 7200 to extend or retract (fold).

[0336] The passive sliding mechanism 7240 is one aspect of the passive form change structure and connects links 722c and 722d to each other for forward and backward movement in a predetermined direction. For example, the passive sliding mechanism 7240 is operated by the user to move forward and backward, and the distance between the movable joint 721c located at one end of link 722c and the passive joint 7260 is variable. With this configuration, the entire form of the arm unit 7200 can be changed.

[0337] The passive joint 7360 is one aspect of the passive configuration that allows links 722d and 722e to rotate by connecting them. For example, the passive joint 7260 can be rotated by user operation, and the angle formed between links 722d and 722e is variable. By utilizing this configuration, the entire configuration of the arm unit 7200 can be altered.

[0338] In this embodiment, the arm unit 11140 has six movable joints 721a to 721f and achieves six degrees of freedom for driving the arm 7200. That is, although the drive control of the arm unit 11140 is achieved through the drive control of the six movable joints 721a to 721f, the passive sliding mechanism 7260 and the passive joints 7260 are not the objects of drive control.

[0339] Specifically, such as Figure 20 As shown, the movable joints 721a, 721d, and 721f are each configured such that the major axis direction of the connecting rods 722a and 722e is the same as the capture direction of the endoscope 11020 as the rotation axis direction. The movable joints 721b, 721c, and 721e are configured such that the x-axis direction is the rotation axis direction, that is, the direction in which the connection angle between each connecting rod 722a to 722c, 722e, and 722f and the endoscope 11020 changes in the yz plane (the plane defined by the y-axis and z-axis). Thus, the movable joints 721a, 721d, and 721f have a so-called yaw function, and the movable joints 721b, 721c, and 721e have a so-called pitch function.

[0340] Because the drive of the arm 7200 in the arm unit 11140 achieves six degrees of freedom, the endoscope 11020 can move freely within the movable range of the arm 7200. Figure 14 An example of a hemisphere as the movable range of endoscope 11020 is shown. With the capture center of endoscope 11020 fixed at the center point of the hemisphere, assuming that the center point RCM (remote motion center) of the hemisphere is the capture center of the treatment site captured by endoscope 11020, the treatment site can be captured from various angles by moving endoscope 11020 on the spherical surface of the hemisphere.

[0341] The embodiments of this disclosure are further defined by the following numbered clauses:

[0342] (1) A system for controlling a medical image capture device during surgery, the system comprising: circuitry configured to receive a first image of a surgical scene captured by the medical image capture device from a first viewpoint and additional information about the scene; determine one or more candidate viewpoints from which the medical image capture device thereby obtains an image of the surgical scene based on the additional information and previous viewpoint information of the surgical scene; provide each of the one or more candidate viewpoints with a simulated image of the surgical scene from the candidate viewpoint based on the first image of the surgical scene; and control the medical image capture device to obtain an image of the surgical scene from a candidate viewpoint corresponding to the selection of one of the one or more simulated images of the surgical scene.

[0343] (2) The system according to paragraph 1, wherein the circuit is further configured to: perform an evaluation of the ability of the candidate viewpoints used by the user and exclude such candidate viewpoints that are unsuitable for use by the user in a surgical scenario.

[0344] (3) The system according to any one of the preceding clauses, wherein the circuitry is further configured to: provide one or more simulated images of a surgical scene for display to a user; and receive from the user a selection of one of the one or more simulated images of the surgical scene.

[0345] (4) The system according to any one of the preceding clauses, wherein the circuitry is further configured to: control the position and / or orientation of the articulated arm supporting the medical image capture device to control the medical image capture device to acquire an image of the surgical scene from a candidate viewpoint corresponding to the selection of one of one or more simulated images of the surgical scene.

[0346] (5) The system according to any one of the preceding clauses, wherein the circuit is configured to: analyze candidate viewpoints according to a predetermined metric and display the top N candidate viewpoints to the user for selection.

[0347] (6) The system according to paragraph 5, wherein the circuitry is configured to analyze a candidate viewpoint as a predetermined metric based on a comparison of the candidate viewpoint with one or more viewpoint preferences of the user.

[0348] (7) The system according to any one of the preceding clauses, wherein the circuitry is configured to: evaluate candidate viewpoints according to a predetermined metric, and control the display to display at least a subset of candidate viewpoints based on the evaluation.

[0349] (8) The system according to paragraph 5, paragraph 6 or paragraph 7, wherein the circuit is configured to: evaluate one or more quantifiable features of the analog image and arrange candidate viewpoints as predetermined metrics based on the evaluation results.

[0350] (9) The system according to any one of the preceding clauses, wherein the circuit is configured to: determine the ability of the image capture device to realize candidate viewpoints and exclude such candidate viewpoints that are not suitable for the image capture device.

[0351] (10) The system according to any one of the preceding clauses, wherein the additional information received by the circuit includes surgical data and / or environmental data of the surgical scene.

[0352] (11) The system according to paragraph 10, wherein the surgical data and / or environmental data of the surgical scene includes at least one of the following: surgical information indicating the state of the surgery; location data of objects in the surgical environment; movement data of objects in the surgical environment; information about the type of surgical tools used by the user; lighting information about the surgical environment; and patient information indicating the state of the patient.

[0353] (12) The system according to any one of the preceding clauses, wherein the circuitry is configured to: receive interaction with a simulated image of a surgical scene, and update one or more properties and / or simulated images of a corresponding candidate viewpoint of the surgical scene based on the interaction.

[0354] (13) The system according to any one of the preceding clauses, wherein the circuitry is configured to determine viewpoint information according to at least one of the following: a previous viewpoint selected by the device for the surgical scenario corresponding to the additional information and a previous viewpoint used by other users for the surgical scenario corresponding to the additional information.

[0355] (14) The system according to paragraph 12, wherein the viewpoint information includes the location information and / or orientation information of the image capturing device.

[0356] (15) The system according to any one of the preceding clauses, wherein the circuit is configured to generate a simulated image of a candidate viewpoint using a machine learning system trained on a previous viewpoint of a surgical scene.

[0357] (16) The system according to any one of the preceding clauses, wherein the circuit is configured to: control the image capture device to acquire images from multiple discrete predetermined locations within the surgical scene as initial calibration to obtain a previous viewpoint of the surgical scene.

[0358] (17) The system according to any one of the preceding clauses, wherein the candidate viewpoint includes at least one of the following: candidate location of the image capture device and / or candidate imaging properties.

[0359] (18) The system according to paragraph 17, wherein the imaging properties include at least one of the following: image scaling of the image capturing device, image focus, image aperture, image contrast, image brightness, and / or imaging type.

[0360] (19) The system according to any one of the preceding clauses, wherein the circuitry is configured to receive at least one of touch input, keyboard input, or voice input as a selection of one of one or more simulated images of a surgical scene.

[0361] (20) A method for controlling a medical image capture device during surgical procedures, the method comprising:

[0362] Receive a first image of a surgical scene captured from a first viewpoint by a medical image capture device, along with additional information about the scene;

[0363] Based on additional information and previous viewpoint information of the surgical scene, the medical image capture device thereby obtains one or more candidate viewpoints for the image of the surgical scene.

[0364] Based on a first image of the surgical scene, provide each of one or more candidate viewpoints with a simulated image of the surgical scene from the candidate viewpoint;

[0365] The medical image capture device is controlled to acquire an image of a surgical scene from a candidate viewpoint corresponding to the selection of one of a number of simulated images of a surgical scene.

[0366] (21) A computer program product including instructions, which, when executed by a computer, cause the computer to perform a method of controlling a medical image capture device during surgical procedures, the method comprising:

[0367] Receive a first image of a surgical scene captured from a first viewpoint by a medical image capture device, along with additional information about the scene;

[0368] Based on additional information and previous viewpoint information of the surgical scene, the medical image capture device thereby obtains one or more candidate viewpoints for the image of the surgical scene.

[0369] Provide a simulated image of the surgical scene from each of one or more candidate viewpoints, based on a first image of the surgical scene.

[0370] The medical image capture device is controlled to acquire an image of a surgical scene from a candidate viewpoint corresponding to the selection of one of a number of simulated images of a surgical scene.

[0371] It is obvious that various modifications and variations of this disclosure are possible based on the foregoing teachings. Therefore, it should be understood that this disclosure can be practiced in ways different from those specifically described herein, within the scope of the appended claims.

[0372] In embodiments of this disclosure that have so far been described as being implemented by data processing apparatuses at least in part by software control, it should be recognized that non-volatile machine-readable media carrying the software, such as optical discs, magnetic disks, semiconductor memories, etc., are also considered to represent embodiments of this disclosure.

[0373] It should be understood that, for clarity, the above description of the embodiments has been made with reference to different functional units, circuits, and / or processors. However, it should be understood that any suitable functional distribution among the different functional units, circuits, and / or processors can be used without departing from the embodiments.

[0374] The described implementations can be implemented in any suitable form, including hardware, software, firmware, or any combination thereof. Optionally, the described implementations can be at least partially implemented as computer software running on one or more data processors and / or digital signal processors. Elements and components of any implementation can be physically, functionally, and logically implemented in any suitable manner. Indeed, functionality can be implemented in a single unit, multiple units, or as part of other functional units. Therefore, the disclosed implementations can be physically and functionally distributed in a single unit or among different units, circuits, and / or processors.

[0375] Although this disclosure has been described in conjunction with some embodiments, it is not intended to be limited to the specific forms set forth herein. Furthermore, while features may appear to have been described in conjunction with specific embodiments, those skilled in the art will recognize that various features of the described embodiments can be combined in any manner suitable for implementing the technology.

Claims

1. A system for controlling a medical image capture device during surgical procedures, the system comprising: The circuit is configured as follows: Receive additional information about the surgical scene and a first image of the surgical scene captured from a first viewpoint by the medical image capture device; One or more candidate viewpoints are determined for the medical image capture device based on the additional information and previous viewpoint information of the surgical scene, wherein the previous viewpoint of the surgical scene is a viewpoint that has been used in a previous surgical procedure, and a second image of the surgical scene is obtained from one or more of the candidate viewpoints. For each of the one or more candidate viewpoints, a simulated image of the surgical scene from the candidate viewpoint is provided based on the first image of the surgical scene; The medical image capture device is controlled to obtain a second image of the surgical scene from a candidate viewpoint corresponding to the selection of one of the simulated images of the surgical scene from one of the simulated images of the surgical scene.

2. The system according to claim 1, wherein, The circuit is further configured to: perform an evaluation of the user's ability to use candidate viewpoints, and exclude candidate viewpoints that are unsuitable for the user to use in the surgical scenario.

3. The system according to claim 1, wherein, The circuit is further configured as follows: One or more of the simulated images of the surgical scene to be displayed are provided to the user; The user receives a selection of one of the simulated images of the surgical scene from one or more of the simulated images.

4. The system according to claim 1, wherein, The circuit is further configured to control the position and / or orientation of the articulated arm supporting the medical image capture device to control the medical image capture device to obtain a second image of the surgical scene from a candidate viewpoint corresponding to the selection of one of the simulated images of one or more of the simulated images of the surgical scene.

5. The system according to claim 1, wherein, The circuit is configured to analyze the candidate viewpoints according to a predetermined metric and display the top N candidate viewpoints to the user for selection.

6. The system according to claim 5, wherein, The circuit is configured to analyze the candidate viewpoints based on a comparison between the candidate viewpoints as a predetermined metric and one or more viewpoint preferences of the user.

7. The system according to claim 1, wherein, The circuit is configured to evaluate a plurality of candidate viewpoints according to a predetermined metric, and based on the evaluation, control the display to show at least a subset of the plurality of candidate viewpoints.

8. The system according to claim 5, 6, or 7, wherein, The circuit is configured to evaluate one or more quantifiable features of the simulated image and arrange the candidate viewpoints according to the evaluation results as a predetermined metric.

9. The system according to claim 1, wherein, The circuit is configured to: determine the ability of the medical image capture device to achieve the candidate viewpoints and exclude candidate viewpoints that are not suitable for the medical image capture device.

10. The system according to claim 1, wherein, The additional information received by the circuit includes surgical data and / or environmental data of the surgical scenario.

11. The system according to claim 10, wherein, The surgical data and / or environmental data of the surgical scenario include at least one of the following: surgical information indicating the status of the surgery; location data of objects in the surgical environment; movement data of objects in the surgical environment; information about the type of surgical tools used by the user; lighting information about the surgical environment; and patient information indicating the patient's status.

12. The system according to claim 1, wherein, The circuit is configured to: receive an interaction with a simulated image of the surgical scene, and update one or more properties of a corresponding candidate viewpoint of the surgical scene and / or the simulated image based on the interaction.

13. The system according to claim 1, wherein, The circuit is configured to determine the previous viewpoint information based on at least one of the following: a previous viewpoint selected by the device for a surgical scenario corresponding to the additional information, and a previous viewpoint used by other users for a surgical scenario corresponding to the additional information.

14. The system according to claim 12, wherein, The viewpoint information includes the location information and / or orientation information of the image capture device.

15. The system according to claim 1, wherein, The circuit is configured to use a machine learning system trained on a previous viewpoint of the surgical scene to generate the simulated image of the candidate viewpoint.

16. The system according to claim 1, wherein, The circuit is configured to, as an initial calibration, control the image capture device to acquire second images from multiple discrete predetermined locations within the surgical scene in order to obtain a previous viewpoint of the surgical scene.

17. The system according to claim 1, wherein, The candidate viewpoints include at least one of the following: the candidate location of the image capture device and the candidate imaging properties.

18. The system according to claim 17, wherein, The imaging properties include at least one of the following: image scaling, image focus, image aperture, image contrast, image brightness, and imaging type of the image capture device.

19. The system according to claim 1, wherein, The circuit is configured to receive at least one of touch input, keyboard input, and voice input as the selection of one of one or more simulated images of the surgical scene.

20. A computer-readable storage medium having a program stored thereon, the program, when executed by a computer, causing the computer to perform a method of controlling a medical image capture device during surgical procedures, the method comprising: Receive additional information about the surgical scene and a first image of the surgical scene captured from a first viewpoint by the medical image capture device; The medical image capture device determines one or more candidate viewpoints based on the additional information and previous viewpoint information of the surgical scene, wherein the previous viewpoint of the surgical scene is a viewpoint that has been used in a previous surgical procedure, and the second image of the surgical scene is obtained from the one or more candidate viewpoints. For each of the one or more candidate viewpoints, a simulated image of the surgical scene from the candidate viewpoint is provided based on the first image of the surgical scene; The medical image capture device is controlled to obtain a second image of the surgical scene from a candidate viewpoint corresponding to the selection of one of the simulated images of the surgical scene from one of the simulated images.

Images