Computer-assisted surgery system, surgical control device, surgical control method, program, and non-transitory storage medium

Abstract

1. A computer-assisted surgery system comprising: an image capture device; a display; a user interface; and circuitry configured to receive information indicating a surgical scenario and a surgical process associated with the surgical scenario; acquire an artificial image of the surgical scenario; output the artificial image for display on the display; and receive, via the user interface, permission information indicating whether permission is granted to perform the surgical process if it is determined that the surgical scenario will occur.

Description

[Technical Field]

[0001] The present disclosure relates to a computer-assisted surgery system, a surgical control device, and 、 surgical control method , program and non-transitory storage medium Regarding. [Background technology]

[0002] The "Background" discussion provided herein is intended to generally present the context for the present disclosure. To the extent described in the Background section, the inventors' work, as well as aspects of the description that may not be admitted as prior art at the time of filing, are not admitted expressly or implicitly as prior art to the present disclosure.

[0003] Some computer-assisted surgery systems allow a computerized surgical device (e.g., a surgical robot) to automatically make decisions based on images captured during surgery. These decisions can then trigger a predetermined process, such as clamping or cauterizing a blood vessel if the computerized surgical system determines there is bleeding, or moving a surgical camera or medical scope used by a human during surgery if the imagery indicates an obstruction. Examples of computer-assisted surgery systems include computer-assisted medical scope systems (in which a computerized surgical device holds and positions a medical scope system (also known as a medical visual scope), such as a medical endoscope, surgical microscope, or surgical exoscope, while a human surgeon uses the images from the medical scope to perform the surgery), master-slave systems (in which a master device used by the surgeon controls a robotic slave device), and open surgery systems in which both the surgeon and the computerized surgical device autonomously perform tasks during surgery.

[0004] The problem with these computer-assisted surgery systems is that it can be difficult to know what the computerized surgical device is looking for when making a decision. This is particularly true when using artificial neural networks to classify images captured during surgery to make decisions. While neural networks can be trained with a large number of training images to increase the likelihood that new images (i.e., those captured during an actual surgical procedure) will be correctly classified, it is not possible to guarantee that all new images will be correctly classified. Therefore, it is impossible to guarantee that all automated decisions made by computerized surgical devices will be correct.

[0005] As a result, decisions made by a computerized surgical device typically require a human user to grant permission before the decision is finalized and a predetermined process associated with the decision is executed. This is inconvenient and time-consuming during surgery for both the human surgeon and the computerized surgical device. This is particularly undesirable in emergency scenarios (e.g., if a major hemorrhage occurs, time that could be spent by the computerized surgical device clamping or cauterizing a blood vessel to stop the bleeding is wasted seeking permission from the human surgeon). Summary of the Invention [Problem to be solved by the invention]

[0006] However, it is also undesirable to allow a computerized surgical device to make automated decisions without the permission of a human surgeon when the captured image is poorly classified and the automated decisions would therefore be incorrect. Thus, a solution to this problem is needed. [Means for solving the problem]

[0007] According to the present disclosure, there is provided a computer-assisted surgery system comprising an image capture device, a display, a user interface, a storage medium, and circuitry, wherein the circuitry receives information indicative of a surgical scenario and a surgical process associated with the surgical scenario from the storage medium, obtains an artificial image of the surgical scenario from an artificial neural network, outputs the artificial image for display on the display, and receives permission information via the user interface indicative of whether permission is granted to perform the surgical process when it is determined that the surgical scenario will occur. receiving an actual image captured by the image capturing device, determining whether the actual image indicates the occurrence of the surgical scenario, determining whether permission to perform the surgical process is available if the actual image indicates the occurrence of the surgical scenario, and controlling a surgical robot to perform the surgical process if permission to perform the surgical process is available. It is structured as follows.

[0008] Non-limiting embodiments and advantages of the present disclosure will be best understood by referring to the following detailed description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 shows a schematic diagram of a computer-assisted surgery system. [Figure 2] FIG. 2 shows a schematic diagram of a surgical control device. [Figure 3A] FIG. 3A illustrates schematically the generation of an artificial image of a given surgical scenario for display to a human. [Figure 3B] FIG. 3B illustrates schematically the generation of an artificial image of a given surgical scenario for display to a human. [Figure 3C] FIG. 3C illustrates schematically the generation of an artificial image of a given surgical scenario for display to a human. [Figure 4A] FIG. 4A shows a schematic representation of a proposal for adjusting the field of view of an imaging device for display to a human. [Figure 4B] FIG. 4B shows a schematic representation of a proposal for adjusting the field of view of the imaging device for display to a human. [Figure 5] FIG. 5 shows a look-up table that stores the permissions associated with each given surgical scenario. [Figure 6] FIG. 6 shows a surgical control method. [Figure 7]FIG. 7 shows a schematic diagram of a first example of a computer-assisted surgery system to which the present technology can be applied. [Figure 8] FIG. 8 shows a schematic diagram of a second example of a computer-assisted surgery system to which the present technology can be applied. [Figure 9] FIG. 9 shows a schematic diagram of a third example of a computer-assisted surgery system to which the present technology can be applied. [Figure 10] FIG. 10 schematically illustrates a fourth example of a computer-assisted surgery system to which the present technology can be applied. [Figure 11] FIG. 11 shows a schematic diagram of an example of an arm unit. [Figure 12] FIG. 12 shows a schematic diagram of an example of a master console.

[0010] Like reference numbers indicate the same or corresponding parts throughout the drawings. DETAILED DESCRIPTION OF THE INVENTION

[0011] 1 illustrates surgery on a patient 106 using an open surgical system. The patient 106 lies on an operating table 105, and a human surgeon 104 and a computerized surgical device 103 perform the surgery together.

[0012] Each of the human surgeon and the computerized surgical device monitors one or more parameters of the surgery, such as patient data collected from one or more patient data collection devices (e.g., electrocardiogram (ECG) data from an ECG monitor, blood pressure data from a blood pressure monitor, etc.; patient data collection devices are known in the art and will not be shown or described in detail), and one or more parameters determined by analyzing images of the surgery (captured by the surgeon's eyes or the computerized surgical device's camera 109) or audio of the surgery (captured by the surgeon's ear or the computerized surgical device's microphone 113). Each of the human surgeon and the computerized surgical device performs respective tasks during the surgery (e.g., some tasks performed only by the surgeon, some tasks performed only by the computerized surgical device, and some tasks performed by both the surgeon and the computerized surgical device) and makes decisions about how to perform those tasks using the monitored one or more surgical parameters.

[0013] It can be difficult to know why a computerized surgical device made a particular decision. For example, based on image analysis using an artificial neural network, the computerized surgical device may determine that a patient has experienced unexpected bleeding and that measures should be taken to stop the bleeding. However, there is no guarantee that the image classification and resulting decision to stop the bleeding are correct. Therefore, before actions to stop the bleeding are performed by the computerized surgical device, the surgeon must be presented with and confirm the decision. This is inconvenient and time-consuming for the surgeon and the computerized surgical device. However, if this is not done and the image classification and resulting decision made by the computerized surgical device are incorrect, the computerized surgical device may take measures to stop nonexistent bleeding, thereby unnecessarily delaying the surgery or risking harm to the patient.

[0014] The present technology helps to meet this need through the ability of artificial neural networks to generate artificial images based on image classifications that the artificial neural network is configured to output. A neural network (e.g., implemented as software on a computer) is made up of many individual neurons, each of which activates under a set of conditions when it recognizes the input it is looking for. When a sufficient number of these neurons activate (e.g., neurons that look for different features of a cat, such as whiskers, fur texture, etc.), the object (e.g., a cat) associated with those neurons is identified by the system.

[0015] Early examples of these recognition systems suffered from a lack of interpretability, which makes it difficult to trace back an output (such as an object classification or recognition event that associates one of several predetermined classifications with an input image) to the input that caused it. This problem has recently begun to be addressed in the field of AI interpretability, where different techniques can be used to trace the decision path of a neural network from input to output.

[0016] One such known technique is feature visualization, which allows artificial generation of visual (or other data type, if input into a neural network appropriately trained to classify other types of data) features that are most likely to cause activation of a particular output, thereby showing a human what stimuli a particular part of the network is looking for.

[0017] In general, there are trade-offs in feature visualization, and the generated features that neurons are looking for may be: Optimized. The generated output of the feature visualization process is an image that maximizes the activation confidence of the selected neural network layer or neuron. Diversity. The range of features that activate selected neural network layers or neurons can be illustrated by the generated images.

[0018] These techniques have different advantages and disadvantages, but in combination they allow someone examining a neural network to check which input features cause neuron activation and therefore a particular classification output.

[0019] Feature visualization is used in this technology to allow a human surgeon (or other human involved in a surgery) to view artificial images that represent what the computerized surgical device's neural network is looking for when making a particular decision. The human can view the artificial images and determine how well they represent the real images of the scene relevant to the decision. If the artificial images appear sufficiently realistic in the context of the decision being made (e.g., the decision is to automatically clamp or cauterize a blood vessel to stop bleeding, and the artificial images sufficiently resemble the bleeding of the blood vessel to be clamped or cauterized), the human will give permission for the decision to be made when the computerized surgical device makes a decision based on real images captured during surgery. The decision can then be automatically executed without further human input during surgery, thereby preventing unnecessary human interruptions and delays to the surgery. On the other hand, if the images do not appear sufficiently realistic (e.g., they contain unnatural artifacts that reduce the human's confidence in the neural network's accurate determination of whether a bleeding blood vessel has occurred), the human will not give such permission. The decision will not be automatically executed during surgery. Instead, if a decision is made, the human is presented with the decision during surgery and asked to give their approval at that time. Therefore, decisions that are likely to be incorrect (due to a reduced ability of the neural network to correctly classify images and make decisions) are not approved in advance, thereby preventing surgical problems resulting from incorrect decisions. This technology therefore provides more automated decision-making during surgery (thereby reducing the frequency of unnecessary interruptions to human surgeons and reducing surgical delays) while keeping surgery safer for patients.

[0020] While FIG. 1 illustrates an open surgical system, the present technology is applicable to other computer-assisted surgical systems in which a computerized surgical device (e.g., a device holding a medical scope in a computer-assisted surgical scope system or a slave device in a master-slave system) can make decisions. Thus, a computerized surgical device is a surgical device equipped with a computer that can make decisions about a surgery using captured images of the surgery. As a non-limiting example, the computerized surgical device 103 in FIG. 1 is a surgical robot that can make decisions and perform autonomous operations based on images captured by a camera 109.

[0021] The robot 103 comprises a controller 110 (a surgical control device) and one or more surgical tools 107 (e.g., a movable scalpel, clamps, or robotic hands). The controller 110 is connected to a camera 109 for taking images of the surgery, a microphone 113 for capturing an audio feed of the surgery, a movable camera arm 112 for holding and adjusting the position of the camera 109 (the movable camera arm comprises a suitable mechanism including one or more electric motors (not shown) controllable by the controller to move the movable camera arm, and hence the camera 109), and an electronic display 102 (e.g., a liquid crystal display) held on a stand 101 so that the electronic display 102 can be viewed by the surgeon 104 during surgery.

[0022] FIG. 2 shows some components of the controller 110.

[0023] The control device 110 includes a processor 201 for processing electronic instructions, a memory 202 for storing the processed electronic instructions and input / output data associated with the electronic instructions, a storage medium 203 (e.g., hard disk drive, solid state drive, etc.) for long-term storage of electronic information, a tool interface 204 for transmitting and / or receiving electronic information to and from one or more surgical tools 107 of the robot 103 to control the one or more surgical tools, and a camera interface 204 for receiving electronic information representing images of the surgical scene captured by the camera 109 and transmitting and / or receiving electronic information to and / or from the camera 109 and the movable camera arm 112 to control the camera. The surgical system includes a camera interface 205 for controlling the operation of the tool interface 109 and the movement of the movable camera arm 112, a display interface 202 for transmitting electronic information representing information displayed on the electronic display 102, a microphone interface 207 for receiving electrical signals representing an audio feed of the surgical scene captured by the microphone 113, a user interface 208 (e.g., with a touch screen, physical buttons, a voice control system, etc.), and a network interface 209 for transmitting and / or receiving electronic information to one or more other devices via a network (e.g., the Internet). Each of the processor 201, memory 202, storage medium 203, tool interface 204, camera interface 205, display interface 206, microphone interface 207, user interface 208, and network interface 209 is implemented, for example, using suitable circuitry. The processor 201 controls the operation of each of the memory 202 , the storage medium 203 , the tool interface 204 , the camera interface 205 , the display interface 206 , the microphone interface 207 , the user interface 208 and the network interface 209 .

[0024] In an embodiment, the artificial neural network used for feature visualization and image classification according to the present technology is hosted on the controller 110 itself (i.e., as computer code stored in memory 202 and / or storage medium 203 for execution by processor 201). Alternatively, the artificial neural network is hosted on an external server (not shown). Information input to the neural network is sent to the external server, and information output from the neural network is received from the external server via network interface 209.

[0025] FIG. 3A shows a surgical scene captured by camera 109. The scene includes a patient's liver 300 and blood vessels 301. The surgeon 104 uses user interface 209 to provide tasks to robot 103 before proceeding to the next stage of surgery. In this case, the selected tasks are (1) applying suction while the surgeon performs a human incision (section labeled "1") and (2) clamping a blood vessel (section labeled "2"). For example, if the user interface includes a touchscreen display, the surgeon selects the location within the surgical scene where each task should be performed by selecting the task from a visually interactive menu provided by the user interface and selecting the corresponding location on a displayed image of the scene captured by camera 109. In this example, electronic display 102 is a touchscreen display, and thus the user interface is configured as part of electronic display 102.

[0026] 3B illustrates a predetermined surgical scenario that may occur during the next stage of the surgical procedure. In this scenario, a blood vessel rupture occurs at location 302, requiring rapid clamping or cauterization by the robot 103 (e.g., using an appropriate tool 107). The robot 103 can detect such a scenario and perform clamping or cauterization by classifying images of the surgical scene captured by the camera 109 when the scenario occurs. This is possible because such images contain information indicating that the scenario has occurred (i.e., blood vessel rupture or bleeding is visually detectable in the image), and the artificial neural network used by the robot 103 for classification uses this information to classify the image as being an image of a blood vessel rupture requiring clamping or a blood vessel rupture requiring cauterization. Thus, in this case, there are two possible predetermined surgical scenarios that may occur during the next stage of the surgery and that are detectable by the robot based on images captured by the camera 109. One is a ruptured blood vessel that requires clamping (applicable when the blood vessel is in the process of rupturing or has recently ruptured), and the other is a blood vessel that requires cauterization (applicable when the blood vessel has already ruptured and is bleeding).

[0027] The problem, however, is that due to the nature of artificial neural network classification, the surgeon 104 does not know what images the robot 103 is looking for to detect the occurrence of these predetermined scenarios. Thus, the surgeon does not know how accurate the robot's determination that one of the predetermined scenarios has occurred is, and therefore traditionally had to give permission for the robot to perform a clamp or cauterize if and when the relevant predetermined scenario was detected by the robot.

[0028] Therefore, before proceeding to the next stage of surgery, feature visualization is performed using the image classification output by the artificial neural network to indicate the occurrence of predetermined scenarios. An image generated using feature visualization is shown in Figure 3C. The image is displayed on the electronic display 102. This allows the surgeon to review the image and determine whether it is a sufficiently realistic depiction of what the surgical scene would look like if each predetermined scenario (i.e., vessel rupture requiring clamping and vessel rupture requiring cauterization) were to occur.

[0029] For clarity, the image in Figure 3C is not an image of the scene captured by camera 109, which is still capturing the scene shown in Figure 3A because the next stage of the surgery has not yet begun. Rather, the image in Figure 3C is an artificial image of the scene generated using artificial neural network feature visualization based on classifications given to real images depicting the surgical scene when each of the predetermined scenarios occurs (classification is possible by previously training the artificial neural network with an appropriate set of training images).

[0030] Each of the artificial images in FIG. 3C exhibits visual features that, if detected in a future real image captured by camera 109, would result in the classification of that future real image as indicating that a predetermined scenario associated with the artificial image (i.e., a vessel rupture requiring clamping or a vessel rupture requiring cauterization) has occurred, and therefore, that robot 103 should perform a predetermined process associated with that classification (i.e., clamping or cauterization). In particular, first set of artificial images 304 shows a rupture 301A of a vessel 301 occurring in a first direction and a rupture 301B of a vessel 301 occurring in a second direction. These artificial images correspond to the predetermined scenario of a vessel rupture requiring clamping. Thus, the predetermined process associated with these images is for robot 103 to perform clamping. Second set of artificial images 305 shows bleeding 301C of a vessel 301 having a first shape and bleeding 301D of a vessel 301 having a second shape. These artificial images correspond to the predetermined scenario of a vessel rupture requiring cauterization. In both sets of images, a graphic 303 is displayed showing the location within the image of features of interest, allowing the surgeon to easily determine the visual features within the image that are likely to result in a particular classification. The location of the graphic 303 is determined, for example, based on the image features associated with the activation of the highest level neural network layers or neurons during the image visualization process.

[0031] It will be appreciated that more or fewer artificial images may be generated for each set. For example, for a more “diverse” image set, more images are generated (showing a more diverse range of possible classifications for image features, but with lower confidence in individual image features), and for a more “optimized” image set, fewer images are generated (showing a less diverse range of possible classifications for image features, but with higher confidence in individual image features). In one example, the number of artificial images generated using feature visualization is adjusted based on the expected visual diversity of image features indicative of a particular given scenario. Thus, a more “diverse” artificial image set may be used for visual features that may be more visually diverse in different instances of a given scenario, and a more “optimized” artificial image set may be used for visual features that may be less visually diverse in different instances of a given scenario.

[0032] If, after reviewing the set of artificial images of FIG. 3C , the surgeon determines that the set is a sufficiently accurate representation of how the surgical scene would look in the predetermined scenario associated with the set, the surgeon may authorize the robot 103 to perform the associated predetermined process (clamping in the case of image set 304 or cauterization in the case of image set 305) without further authorization. This would therefore occur automatically if future images captured by camera 109 during the next stage of the surgical procedure are classified as indicating that the predetermined scenario has occurred. Thus, the surgeon is not hindered by the robot 103 requesting authorization during the surgical procedure, and the time delay for the robot performing the predetermined process is also reduced. On the other hand, if, after reviewing the set of artificial images of FIG. 3C , the surgeon determines that the set is not a sufficiently accurate representation of how the surgical scene would look in the predetermined scenario associated with the set, the surgeon may not grant the robot 103 that authorization. In this case, if a future image captured by camera 109 during the next stage of the surgical procedure is classified as indicating that the predetermined scenario associated with that set has occurred, the robot will still seek permission from the surgeon before performing the associated predetermined process (i.e., clamping in the case of image set 304, or cauterizing in the case of image set 305). This helps ensure patient safety and reduce surgical delays by reducing the likelihood that the robot 103 will make an incorrect decision and therefore perform the associated predetermined process unnecessarily.

[0033] Permission (or lack thereof) is granted by the surgeon via user interface 209. In the example of FIG. 3C, textual information 308 indicating the predetermined process associated with each set of artificial images is displayed with each image set, along with virtual buttons 306A and 306B indicating whether permission is granted (yes) or not (no), respectively. The surgeon indicates permission or not by touching the associated virtual button. The button last touched by the surgeon is highlighted (in this case, the surgeon is satisfied with granting permission for both image sets, so the "yes" button 306A is highlighted in both image sets). Once the surgeon is satisfied with his or her selection, he or she touches the "continue" virtual button 307. This indicates to the robot 103 that the next stage of the surgery is to begin, and that the images captured by camera 109 should be classified and that the predetermined process according to the classified images should be carried out in accordance with the permissions selected by the surgeon.

[0034] In one embodiment, for any predetermined process for which permission was not previously given (e.g., if the "No" button 306B for that predetermined process in FIG. 3C was selected), permission from the surgeon is still sought during the next stage of the surgery using the electronic display 102. In this case, the electronic display displays only text information 308 indicating the proposed predetermined process (optionally using an image captured by the camera 109 having the classification that generated the suggestion) and "Yes" and "No" buttons 306A, 306B. If the surgeon selects the "Yes" button, the robot 103 proceeds to perform the predetermined process. If the surgeon selects the "No" button, the robot 103 does not perform the predetermined process and the surgery continues as planned.

[0035] In one embodiment, the textual information 308 indicating a predetermined process performed by the robot 103 may be replaced with other visual information, such as an appropriate graphic overlaid on the image (artificial or real) to which the predetermined process pertains. For example, for the predetermined process associated with artificial image set 304 of FIG. 3C , “clamping a blood vessel to prevent rupture,” a graphic of a clamp may be overlaid on the appropriate portion of each image in the set. For the predetermined process associated with artificial image set 305 of FIG. 3C , “cauterizing to prevent bleeding,” a graph illustrating cauterization may be overlaid on the appropriate portion of each image in the set. Similar overlay graphics may be used on real images captured by camera 109 if prior authorization is not given and, therefore, a predetermined scenario occurs during the next stage of the surgical procedure and authorization from the surgeon 104 is sought.

[0036] In one embodiment, a surgical procedure is divided into predetermined surgical phases, and each surgical phase is associated with one or more predetermined surgical scenarios. Each of the one or more predetermined surgical scenarios associated with each surgical phase is associated with an image classification of an artificial neural network, and a newly captured image of the surgical scene given the image classification by the artificial neural network is determined to be an image of the surgical scene when the predetermined surgical scenario is occurring. Each of the one or more predetermined surgical scenarios is also associated with one or more respective predetermined processes to be executed by the robot 103 when the image classification indicates that the predetermined surgical scenario is occurring.

[0037] The storage medium 203 stores information indicating one or more predetermined surgical scenarios associated with each surgical stage and information indicating one or more predetermined processes associated with each of those predetermined scenarios. Thus, when the robot 103 is notified of the current predetermined surgical stage, it can retrieve the information indicating the one or more predetermined surgical scenarios and one or more predetermined processes associated with that stage and use this information to obtain authorization (e.g., as in FIG. 3C ) and, if necessary, perform the one or more predetermined processes.

[0038] The robot 104 can learn the current predetermined surgical stage using any suitable method. For example, the surgeon 104 may notify the robot 103 of the predetermined surgical stage in advance (e.g., using a visually interactive menu system provided by the user interface 208), or the surgeon 104 may manually notify the robot 103 each time a new surgical stage is to be entered (e.g., by selecting a predetermined virtual button provided by the user interface 208). Alternatively, the robot 103 may determine the current surgical stage based on tasks assigned by the surgeon. For example, the robot may determine that the current surgical stage includes tasks (1) and (2) based on tasks (1) and (2) provided to the robot in FIG. 3A. In this case, the information indicating each surgical stage may include information indicating the combination of tasks associated with that stage, so that the robot can determine the current surgical stage by comparing the tasks assigned to the robot with the tasks associated with each surgical stage and selecting the surgical stage with the most matching tasks. Alternatively, the robot 103 may automatically determine the current stage based on images of the surgical scene captured by the camera 109, an audio feed of the surgery captured by the microphone 113, and / or information (e.g., position, movement, motion, or measurements) about one or more robotic tools 107, each of which tends to have characteristics unique to a given surgical stage. In one example, these characteristics may be determined using a suitable machine learning algorithm (e.g., another artificial neural network) trained using images, audio, and / or tool information from several previous instances of the surgical procedure.

[0039] 3A-3C, the predetermined process is for the robot 103 to automatically perform a direct surgical action (i.e., clamp or cauterize), but the predetermined process may take the form of any other decision that a robot given appropriate permissions can make automatically. For example, the predetermined process may relate to a change in schedule (e.g., a change in the planned incision path) or a change in the position of the camera 109 (e.g., if the predetermined surgical scenario includes blood splatter that may obstruct the camera's view). Several other embodiments are described below.

[0040] In one embodiment, a predetermined process performed by the robot 103 is to move the camera 109 (via control of the movable camera arm 112) to maintain a view of the active tool 107 within the surgical scene when blood splatter (or other bodily fluid splatter) may obstruct the camera's view.

[0041] 1. One of the given surgical scenarios for the current surgical stage is one in which blood may splash onto the camera 109, thereby affecting the camera's ability to image the scene.

[0042] 2. An artificial image of a given surgical scenario is generated and displayed along with information indicating the given process that the robot will perform if the scenario occurs. For example,

[0043] a. An artificial image of the initial scenario or just before its occurrence (e.g., a blood vessel being cut with a scalpel and wide-angle blood splatter) is displayed with an overlay graphic (e.g., a directional arrow) indicating that the robot 103 is lowering the angle of incidence of the camera 109 onto the surgical scene to avoid collision with the blood splatter while maintaining a view of the scene.

[0044] b. An artificial image of an initial scenario or just before its occurrence (e.g., a blood vessel dissection with a scalpel and wide-angle blood splatter) is displayed along with an additional image of the same scenario in which the viewpoint of the image moves to correspond to the planned movement of the camera 109. This may be achieved, for example, by mapping the artificial image onto a 3D model of the surgical scene and moving the viewpoint within the 3D model of the surgical scene to match the viewpoint of the actual camera in the actual surgical scene (if a predetermined scenario indicating potential blood splatter occurs). Alternatively, the camera 109 itself may be temporarily moved to a proposed new position and an actual image captured by the camera 109 in the new position may be displayed (thereby allowing the surgeon 104 to view the proposed different viewpoint and decide whether it is acceptable).

[0045] In one embodiment, the predetermined process performed by the robot 103 is to move the camera 109 (via control of the movable camera arm 112) to obtain the best camera angle and view for the current surgical stage.

[0046] 1. One of the given surgical scenarios of the current surgical stage is that there are changes in the surgical scene during the surgical stage, making different camera view strategies more beneficial. Examples of changes include: a. Surgeon 104 switches tools b. Introduction of new tools c. Retreating or removing the tool from the scene d. A transition of the surgical phase, such as the revealing of a particular organ or structure indicating that the surgery will proceed to the next phase. In this case, the given surgical scenario is that the surgery will proceed to the next surgical phase.

[0047] 2. An artificial image of a predetermined surgical scenario is generated and displayed along with information indicating the predetermined process the robot will perform if the scenario occurs. This may include overlaying an appropriate graphic on the artificial image indicating the direction of camera movement, or changing the viewpoint of the artificial or real image as described above. In one example, if a particular organ or structure is revealed that indicates a surgical phase transition (see d), the predetermined process may be to move the camera 109 closer to the organ or structure so that more precise actions can be performed on the organ or structure.

[0048] In one embodiment, a predetermined process performed by the robot 103 is to move the camera 109 (via control of the movable camera arm 112) so that if a mistake is made by the surgeon 104 (e.g., by dropping a tool, etc.), one or more features of the surgical scene always remain within the camera's field of view.

[0049] 1. One of the given surgical scenarios for the current surgical stage is that a visually identifiable error is made by the surgeon 104. Examples of errors include: a. Dropping of grasped organs b. Dropping of held tool

[0050] 2. An artificial image of a predetermined surgical scenario is generated and displayed along with information indicating the predetermined process the robot would perform if the scenario were to occur. This may involve overlaying an appropriate graphic onto the artificial image to indicate the direction of camera movement, or changing the viewpoint of the artificial or real image as described above. In one example, the camera position is adjusted so that the dropped item and the surgeon's hand that dropped the item are always kept within the camera's field of view.

[0051] In one embodiment, the predetermined process performed by the robot 103 is to move the camera 109 (via control of the movable camera arm 112) when bleeding is seen within the camera's field of view but is from a source that is not within the field of view.

[0052] 1. One of the given surgical scenarios at the current stage of surgery is that there is bleeding from an unseen source.

[0053] 2. An artificial image of a predetermined surgical scenario is generated and displayed along with information indicating the predetermined process the robot would perform if the scenario were to occur. This may involve overlaying an appropriate graphic onto the artificial image indicating the direction of camera movement, or changing the viewpoint of the artificial or real image as described above. In one example, the camera 109 is moved to a higher position to widen the field of view to include the source of bleeding and the original camera focus.

[0054] In one embodiment, the predetermined process performed by the robot 103 is to move the camera 109 (via control of the movable camera arm 112) to provide a better view for performing the incision.

[0055] 1. One of the given surgical scenarios for the current surgical stage is that an incision is about to be made.

[0056] 2. An artificial image of a predetermined surgical scenario is generated and displayed along with information indicating the predetermined process the robot would perform if the scenario were to occur. This may involve overlaying an appropriate graphic onto the artificial image to indicate the direction of camera movement, or changing the viewpoint of the artificial or real image as described above. In one example, the camera 109 is moved directly above the patient 106 to provide a view of the incision with less occlusion of the tools.

[0057] In one embodiment, a predetermined process performed by the robot 103 is to move the camera 109 (via control of the movable camera arm 112) to get a better view of the incision when the incision is detected to be deviating from the planned incision path.

[0058] 1. One of the given surgical scenarios at the current surgical stage is that the incision deviates from the planned incision path.

[0059] 2. An artificial image of a predetermined surgical scenario is generated and displayed along with information indicating the predetermined process the robot will perform if the scenario occurs. This may include overlaying an appropriate graphic on the artificial image indicating the direction of camera movement, or changing the viewpoint of the artificial or real image as described above. In one example, the camera may be moved to compensate for insufficient depth resolution (or another imaging characteristic) that caused a deviation from the planned incision path. For example, the camera may be moved to have a field of view that emphasizes the spatial dimensions of the deviation, thereby allowing the deviation to be more easily assessed by the surgeon.

[0060] In one embodiment, the predetermined process performed by the robot 103 is to move the camera 109 (via control of the movable camera arm 112) to avoid occlusions (e.g., by tools) in the camera's field of view.

[0061] 1. One of the given surgical scenarios in the current surgical stage is that a tool blocks the camera's view.

[0062] 2. An artificial image of a predetermined surgical scenario is generated and displayed along with information indicating the predetermined process the robot would perform if the scenario were to occur. This may involve overlaying an appropriate graphic on the artificial image indicating the direction of camera movement, or changing the viewpoint of the artificial or real image as described above. In one example, the camera is moved in an arc while keeping a predetermined object (e.g., an incision) within its field of view to avoid occlusion by the tool.

[0063] In one embodiment, the predetermined process performed by the robot 103 is to move the camera 109 (via control of the movable camera arm 112) to adjust the camera's field of view as the surgeon's work area (e.g., as indicated by the position of a tool used by the surgeon) moves towards the boundaries of the camera's field of view.

[0064] 1. One of the given surgical scenarios for the current surgical stage is that the surgeon's working area approaches the boundary of the camera's current field of view.

[0065] 2. An artificial image of a predetermined surgical scenario is generated and displayed, along with information indicating the predetermined process the robot will perform if the scenario occurs. This may include overlaying an appropriate graphic onto the artificial image indicating the direction of camera movement, or changing the viewpoint of the artificial or real image as described above. In one example, the camera is moved to shift the field of view so that the surgeon's working area is at the center of the field of view, or the field of view of the camera is expanded (e.g., by moving the camera further away or activating the camera's optical or digital zoom-out function) to keep the surgeon's working area in view (along with objects originally in view).

[0066] In one embodiment, the predetermined process performed by the robot 103 is to move the camera 109 (via control of the movable camera arm 112) to avoid a collision between the camera 109 and another object (e.g., a tool held by a surgeon).

[0067] 1. One of the given surgical scenarios in the current surgical stage is that the camera may collide with another object.

[0068] 2. An artificial image of a predetermined surgical scenario is generated and displayed along with information indicating the predetermined process the robot would perform if the scenario were to occur. This may involve overlaying an appropriate graphic onto the artificial image indicating the direction of camera movement, or changing the viewpoint of the artificial or real image as described above. In one example, camera movement can be compensated for by performing a digital zoom on an appropriate area of the camera's new field of view to approximate the camera's field of view before the movement (this is possible if the camera's previous and new fields of view have an appropriate overlap area).

[0069] In one embodiment, the predetermined process performed by the robot 103 is to move the camera 109 (via control of the movable camera arm 112) away from a predetermined object and towards a new event (e.g., bleeding) that occurs within the camera's field of view.

[0070] 1. One of the given surgical scenarios for the current surgical stage is that while the camera is focused on a given object, a new event occurs within the camera's field of view.

[0071] 2. An artificial image of a predetermined surgical scenario is generated and displayed along with information indicating the predetermined process the robot would perform if the scenario were to occur. This may include overlaying an appropriate graphic onto the artificial image to indicate the direction of camera movement, or changing the viewpoint of the artificial or real image as described above. In one example, as part of the task assigned to the robot, a camera tracks the position of a needle during suturing. If bleeding becomes visible within the camera's field of view, the camera stops tracking the needle and moves to focus on the bleeding.

[0072] It will be appreciated that the above-described embodiments do not necessarily require a change in the position of the camera 109. Rather, what is important is an appropriate change in the camera's field of view. A change in the camera's field of view may or may not require a change in the camera's position. For example, a change in the camera's field of view may be obtained by activating an optical or digital zoom function of the camera, which changes the field of view without requiring a physical change in the camera's position. It will also be appreciated that the above-described embodiments may be applied to any other suitable movable and / or zoomable image capture device, such as a medical scope.

[0073] 4A and 4B show an example of a graphic overlay or modified image perspective that may be displayed on the display 102 when a given process for which permission is requested involves changing the camera's field of view. This example relates to an embodiment in which the camera's field of view is changed because a tool obstructs the camera's 109 field of view. However, a similar arrangement may be provided for other given surgical scenarios that require a change in the camera's field of view. The display screens of FIGS. 4A and 4B may be shown, for example, prior to the start of a given surgical phase associated with the given surgical scenario.

[0074] FIG. 4A shows an example of a graphic overlay 400 of a camera-obstructing tool 401 on an artificial image 402 associated with a predetermined surgical scenario. The overlay 400 indicates that the predetermined process for which permission is being sought is to rotate the camera's field of view 180 degrees while keeping the patient's liver 300 in view. The surgeon is also notified of this by textual information 308. The surgeon reviews the artificial image 402 and determines whether it is a sufficient representation of what the surgical scene will look like in the predetermined surgical scenario. In this case, the surgeon believes it is a sufficient representation. Therefore, the surgeon selects the virtual "Yes" button 306A and then the virtual "Continue" button 307. Thus, future classification of a real image of the camera-obstructing tool captured by the camera during the next surgical phase that shows the predetermined surgical scenario will automatically result in the camera position being automatically rotated 180 degrees while keeping the patient's liver 300 in view. Therefore, the surgeon is not hindered from providing permission during the surgical procedure, and the tool's obstruction of the camera's field of view is quickly alleviated.

[0075] FIG. 4B illustrates an example of an altered image perspective associated with a predetermined surgical scenario in which a tool 401 obstructs the camera's view. The predetermined process for which permission is sought is the same as in FIG. 4A: rotating the camera's view by 180 degrees while keeping the patient's liver 300 within view. However, instead of a graphic overlay on the artificial image 402, an additional image 403 is displayed. The perspective of the additional image 403 is the camera's perspective when rotated 180 degrees according to the predetermined process. Image 403 may be another artificial image (e.g., obtained by mapping the artificial image 402 to a 3D model of the surgical scene and rotating the view within the 3D model by 180 degrees according to the predetermined process). Alternatively, image 403 may be an actual image captured by temporarily rotating the camera 180 degrees according to a predetermined process so that the surgeon can see the camera's actual view when the camera is in this alternative position. For example, the camera may be rotated to the proposed position long enough to capture image 403 and then rotated back to its original position. The surgeon is also reminded of the proposed camera movement by textual information 308. The surgeon can then review the artificial image 402, in this case by reselecting the virtual "Yes" button 306A and the virtual "Continue" button 307 in the same manner as described for Figure 4A.

[0076] In one embodiment, each predetermined process for which permission is sought is assigned information indicating how invasive the predetermined process is to a human patient, referred to as an “invasiveness score.” More invasive predetermined processes (e.g., cauterization, clamping, or incision performed by the robot 103) are assigned a higher invasiveness score than less invasive procedures (e.g., changing the camera view). A particular predetermined surgical scenario may be associated with multiple predetermined processes (e.g., changing the camera view, incision, and cauterization) that require permission. To reduce the time it takes for the surgeon to grant permission for each predetermined process, once the surgeon grants permission for a predetermined process with a higher invasiveness score, permission is automatically granted for all other predetermined processes with the same or lower invasiveness scores. Thus, for example, if incision has the highest invasiveness score, followed by cauterization and then changing the camera view, granting permission for incision automatically grants permission for cauterization and changing the camera view. Granting permission for cauterization automatically grants permission for changing the camera view (but not for incision, which has a higher invasiveness score). Giving permission to change the camera view does not automatically give permission to cauterize or incise (as these have lower invasiveness scores than cauterize and incise).

[0077] In one embodiment, after classification of a real image captured by camera 109 as indicating a predetermined surgical scenario has occurred, the real image is first compared to the artificial image previously used to determine authorization for one or more predetermined processes associated with the predetermined surgical scenario. The comparison of the real image with the artificial image is performed using any suitable image comparison algorithm (e.g., a pixel-by-pixel comparison using appropriately determined parameters and tolerances) that outputs a score indicating the similarity of the two images (similarity score). The previously authorized one or more predetermined processes are then automatically executed only if the similarity score exceeds a predetermined threshold. This helps reduce the risk that an improper classification of the real image by the artificial neural network will result in the execution of one or more authorized predetermined processes. Such an improper classification may occur, for example, if the real image contains unexpected image features (e.g., lens artifacts) for which the artificial neural network was not trained. Even if the real image does not resemble the image used to train the artificial neural network to output the associated classification, the unexpected image features may cause the artificial neural network to output that classification. Therefore, performing an image comparison before performing one or more permitted predetermined processes associated with a classification also reduces the risk of improper performance of one or more permitted predetermined processes (which may adversely affect surgical efficiency and / or patient safety).

[0078] Once permission is granted (or not granted) for each predetermined surgical scenario associated with a particular predetermined surgical stage, information indicating each predetermined surgical scenario, the one or more predetermined processes associated with that predetermined surgical scenario, and whether permission has been granted is stored in memory 202 and / or storage medium 203 for reference during the predetermined surgical stage. For example, the information may be stored as a lookup table such as that shown in FIG. 5. The table in FIG. 5 also stores an invasiveness score (in this example, "high," "medium," or "low") for each predetermined process. When an actual image captured by a camera is classified by an artificial neural network (ANN) as representing a predetermined surgical scenario, processor 201 references one or more predetermined processes and their permissions associated with that predetermined surgical scenario. Processor 201 then controls robot 103 to automatically execute predetermined processes for which permission has been granted (i.e., "yes" in the permission field). For processes for which permission has not been granted (i.e., "no" in the permission field), permission is separately requested during surgery, and robot 103 will not execute the process unless permission is granted. 5 is for a predetermined surgical phase that involves a surgeon opening a patient's liver 300 along a predetermined path. Different predetermined surgical phases may have different predetermined surgical scenarios and different predetermined processes associated with them, which is reflected in each lookup table.

[0079] Although the above description considers a surgeon, the present technology is applicable to any human supervisor in the operating room (e.g., anesthesiologist, nurse, etc.) who must seek permission before the robot 103 automatically executes a predetermined process in a detected predetermined surgical scenario.

[0080] Thus, this technique allows a supervisor of a computer-assisted surgery system to grant permission for a computerized surgical device (e.g., robot 103) to perform an action before permission is requested. This allows permission requests to be grouped during surgery at times convenient for the supervisor (e.g., before surgery or before each predetermined stage of surgery when time pressure is less). This also allows the computerized surgical device to take action more quickly (because time is not wasted requesting permission when action needs to be taken) and allows the computerized surgical device to address a wider range of situations requiring prompt action (where the process of requesting permission would normally prevent the computerized surgical device from addressing the situation). The permission requests provided are also more meaningful (because the artificial images more closely represent possible options for the actual stimuli that trigger the computerized surgical device to make a decision). The review effort by the human supervisor is also reduced for certain surgical scenarios that are likely to occur (and therefore traditionally require permission to be given several times during surgery) and for which communication with a human during surgery is difficult (e.g., when a decision needs to be made quickly or when the decision requires lengthy communication to the surgeon). Closer collaboration with the human surgeon is possible when the required permission makes it easier for the computerized surgical device to communicate what it perceives as a likely surgical scenario to the human surgeon.

[0081] FIG. 6 shows a flowchart illustrating a method performed by the controller 110 according to one embodiment.

[0082] The method begins at step 600 .

[0083] In step 601, an artificial image of a surgical scene during a predetermined surgical scenario is obtained using a visualization feature of an artificial neural network configured to output information indicative of the predetermined surgical scenario when an actual image of the surgical scene captured by camera 109 during the predetermined surgical scenario is input to the artificial neural network.

[0084] In step 602, the display interface outputs the artificial image for display on the electronic display 102.

[0085] In step 603, the user interface 208 receives permission information indicating whether a human has given permission for a predetermined process to be performed in response to the artificial neural network outputting information indicative of a predetermined surgical scenario when an actual image captured by the camera 109 is input to the artificial neural network.

[0086] In step 604 , the camera interface 205 receives the actual image captured by the camera 109 .

[0087] In step 605, a real image is input to the artificial neural network.

[0088] In step 606, it is determined whether the artificial neural network outputs information indicative of a predetermined surgical scenario. If not, the method ends in step 609. If yes, the method proceeds to step 607.

[0089] In step 607, it is determined whether a human has authorized the execution of the given process. If so, the method ends in step 609. If so, the method proceeds to step 608.

[0090] In step 608, the controller causes a predetermined process to be executed.

[0091] The process ends at step 609.

[0092] FIG. 7 schematically illustrates an example of a computer-assisted surgery system 1126 to which the present technology can be applied. The computer-assisted surgery system is a master-slave system incorporating an autonomous arm 1100 and one or more surgeon-controlled arms 1101. The autonomous arm holds an imaging device 1102 (e.g., a surgical camera or a medical visual scope such as a medical endoscope, surgical microscope, or surgical exoscope). Each of the one or more surgeon-controlled arms 1101 holds a surgical instrument 1103 (e.g., a cutting tool). The imaging device of the autonomous arm outputs an image of the surgical scene to an electronic display 1110 that can be viewed by the surgeon. The autonomous arm autonomously adjusts the view of the imaging device while the surgeon is performing surgery using the one or more surgeon-controlled arms to provide the surgeon with an appropriate view of the surgical scene in real time.

[0093] The surgeon controls one or more surgeon-controlled arms 1101 using a master console 1104. The master console includes a master controller 1105. The master controller 1105 includes one or more force sensors 1106 (e.g., torque sensors), one or more rotation sensors 1107 (e.g., encoders), and one or more actuators 1108. The master console includes an arm (not shown) that includes one or more joints and a manipulator. The manipulator is grasped by the surgeon and can be moved to move the arm about one or more joints. One or more force sensors 1106 detect forces applied by the surgeon to the manipulator of the arm about one or more joints. One or more rotation sensors detect the angle of rotation of one or more joints of the arm. The actuator 1108 drives the arm about one or more joints, enabling the arm to provide tactile feedback to the surgeon. The master console includes a natural user interface (NUI) input / output for receiving input information from the surgeon and providing output information to the surgeon. The NUI input / output includes an arm (which the surgeon moves to provide input information and provides haptic feedback to the surgeon as output information). The NUI input / output may also include, for example, voice input, eye gaze input, and / or gesture input. The master console includes an electronic display 1110 for outputting images captured by the imaging device 1102.

[0094] The master console 1104 communicates with the autonomous arm 1100 and each of the one or more surgeon-controlled arms 1101 via a robotic control system 1111. The robotic control system is connected to the master console 1104, the autonomous arm 1100, and the one or more surgeon-controlled arms 1101 by wired or wireless connections 1123, 1124, and 1125. The connections 1123, 1124, and 1125 allow for the exchange of wired or wireless signals between the master console, the autonomous arm, and the one or more surgeon-controlled arms.

[0095] The robotic control system includes a control processor 1112 and a database 1113. The control processor 1112 processes signals received from one or more force sensors 1106 and one or more rotation sensors 1107 and outputs control signals for one or more actuators 1116 to drive one or more surgeon-controlled arms 1101. In this manner, movement of the controls on the master console 1104 causes corresponding movement of one or more surgeon-controlled arms.

[0096] The control processor 1112 also outputs control signals for one or more actuators 1116 to drive one or more autonomous arms 1100. The control signals output to the autonomous arms are determined by the control processor 1112 in response to signals received from one or more of the master console 1104, one or more surgeon-controlled arms 1101, the autonomous arms 1100, and any other signal sources (not shown). The received signals are signals that indicate the appropriate position of the autonomous arm for an appropriate view of image to be captured by the imaging device 1102. The database 1113 stores the values of the received signals and the corresponding autonomous arm positions.

[0097] For example, for a given combination of values of signals received from one or more force sensors 1106 and rotation sensors 1107 of the master controller (which signals in turn indicate corresponding movements of one or more surgeon-controlled arms 1101), the corresponding position of the autonomous arm 1100 is set so that the images captured by the imaging device 1102 are not obstructed by the one or more surgeon-controlled arms 1101.

[0098] As another example, if the signal output by one or more force sensors 1117 (e.g., torque sensors) of the autonomous arm indicates that the autonomous arm is experiencing resistance (e.g., due to an obstacle in the path of the autonomous arm), a corresponding position of the autonomous arm is set so that the imaging device 1102 captures an image from an alternative view (e.g., one that allows the autonomous arm to move along an alternative path that does not include the obstacle).

[0099] It will be appreciated that there may be other types of received signals that indicate the approximate position of the autonomous arm.

[0100] The control processor 1112 looks up the values of the received signals in the database 1112 and obtains information indicative of the corresponding position of the autonomous arm 1100. This information is then processed to generate further signals that cause the autonomous arm's actuators 1116 to move the autonomous arm to the indicated position.

[0101] The autonomous arm 1100 and one or more surgeon-controlled arms 1101 each include an arm unit 1114. The arm unit includes an arm (not shown), a control unit 1115, one or more actuators 1116, and one or more force sensors 1117 (e.g., torque sensors). The arm includes one or more links and joints that enable movement of the arm. The control unit 1115 sends and receives signals to and from the robot control system 1111.

[0102] In response to signals received from the robotic control system, the control unit 1115 controls one or more actuators 1116 to drive the arm about one or more joints to move the arm to the appropriate position. For one or more surgeon-controlled arms 1101, the received signals are generated by the robotic control system based on signals received from the master console 1104 (e.g., by the surgeon manipulating the arm on the master console). For the autonomous arm 1100, the received signals are generated by the robotic control system looking up appropriate autonomous arm position information in a database 1113.

[0103] The control unit 1115 outputs signals to the robotic control system in response to signals output by one or more force sensors 1117 around one or more joints. This enables, for example, the robotic control system to send signals to the master console 1104 indicative of the resistance experienced by one or more surgeon-controlled arms 1101 and provide corresponding haptic feedback to the surgeon (e.g., resistance experienced by one or more surgeon-controlled arms causes the master console's actuators 1108 to produce corresponding resistance in the master console's arms). As another example, this enables the robotic control system to reference appropriate autonomous arm position information in the database 1113 (e.g., to find an alternative position for the autonomous arm if one or more force sensors 1117 indicate an obstacle is in the autonomous arm's path).

[0104] The imaging device 1102 of the autonomous arm 1100 includes a camera control unit 1118 and an imaging unit 1119. The camera control unit controls the imaging unit to capture images and controls various parameters of the captured images, such as zoom level, exposure value, white balance, etc. The imaging unit captures images of the surgical scene. The imaging unit includes all components necessary to capture images, including one or more lenses and an image sensor (not shown). The view of the surgical scene from which an image is captured depends on the position of the autonomous arm.

[0105] The surgical instrument 1103 of one or more surgeon control arms includes a device control unit 1120, a manipulator 1121 (e.g., including one or more motors and / or actuators), and one or more force sensors 1122 (e.g., torque sensors).

[0106] The device control unit 1120 controls the manipulator to perform a physical action (e.g., a cutting action if the surgical instrument 1103 is a cutting tool) in response to signals received from the robotic control system 1111. The signals are generated by the robotic control system in response to signals received from the master console 1104, which are generated by the surgeon entering information into the NUI input / output 1109 to control the surgical instrument. For example, the NUI input / output includes one or more buttons or levers configured as part of the controls on the arm of the master console that the surgeon can operate to cause the surgical instrument to perform a predetermined action (e.g., turning on or off an electric blade if the surgical instrument is a cutting tool).

[0107] The device control unit 1120 also receives signals from one or more force sensors 1122. In response to the received signals, the device control unit provides corresponding signals to the robotic control system 1111, which in turn provides corresponding signals to the master console 1104. The master console provides tactile feedback to the surgeon via the NUI input / output 1109. Thus, the surgeon receives tactile feedback from the surgical instrument 1103 and one or more surgeon-controlled arms 1101. For example, if the surgical instrument is a cutting tool, the tactile feedback may include a button or lever that operates the cutting tool to provide more resistance to movement when the signal from the one or more force sensors 1122 indicates a greater force on the cutting tool (e.g., as occurs when cutting harder material such as bone), and less resistance to movement when the signal from the one or more force sensors 1122 indicates a lesser force on the cutting tool (e.g., as occurs when cutting softer material such as muscle). The NUI input / output 1109 includes one or more suitable motors, actuators, etc. to provide haptic feedback in response to signals received from the robot control system 1111.

[0108] Figure 8 shows a schematic diagram of another example of a computer-assisted surgery system 1209 to which the present technology can be applied. The computer-assisted surgery system 1209 is a surgical system in which a surgeon performs tasks via a master-slave system 1126 and a computerized surgical device 1200 performs the tasks autonomously.

[0109] The master-slave system 1126 is the same as in Figure 7 and will not be described, however, the master-slave system may be a different system than that of Figure 7 in alternative embodiments, or may be omitted entirely (in which case the system 1209 operates autonomously and the surgeon performs the traditional surgery).

[0110] The computerized surgical apparatus 1200 includes a robotic control system 1201 and a tool-holding arm apparatus 1210. The tool-holding arm apparatus 1210 includes an arm unit 1204 and a surgical instrument 1208. The arm unit includes an arm (not shown), a control unit 1205, one or more actuators 1206, and one or more force sensors 1207 (e.g., torque sensors). The arm includes one or more joints that enable movement of the arm. The tool-holding arm apparatus 1210 sends and receives signals to the robotic control system 1201 via a wired or wireless connection 1211. The robotic control system 1201 includes a control processor 1202 and a database 1203. Although shown as separate robotic control systems, the robotic control system 1201 and the robotic control system 1111 may be the same. The surgical instrument 1208 has a configuration similar to the surgical instrument 1103; these are not shown in FIG. 8 .

[0111] In response to control signals received from the robotic control system 1201, the control unit 1205 controls one or more actuators 1206 to drive the arm about one or more joints to move the arm to the appropriate position. The operation of the surgical instrument 1208 is also controlled by control signals received from the robotic control system 1201. The control signals are generated by the control processor 1202 in response to signals received from one or more of the arm unit 1204, the surgical instrument 1208, and any other signal sources (not shown). The other signal sources may include an imaging device (e.g., the imaging device 1102 of the master-slave system 1126) that captures images of the surgical scene. The signal values received by the control processor 1202 are compared with signal values stored in a database 1203 along with corresponding arm position and / or surgical instrument operating state information. The control processor 1202 retrieves the arm position and / or surgical instrument operating state information associated with the received signal values from the database 1203. The control processor 1202 then uses the retrieved arm position and / or surgical tool motion state information to generate control signals that are sent to the control unit 1205 and the surgical tool 1208 .

[0112] For example, if a signal received from an imaging device capturing images of a surgical scene indicates a predetermined surgical scenario (e.g., via a neural network image classification process, etc.), the predetermined surgical scenario is looked up in database 1203, and arm position information and / or surgical tool motion state information associated with the predetermined surgical scenario is retrieved from the database. As another example, if the signal indicates a resistance value measured by one or more force sensors 1207 around one or more joints of arm unit 1204, the resistance value is looked up in database 1203, and arm position information and / or surgical tool motion state information associated with the resistance value is retrieved from the database (e.g., so that the position of the arm can be changed to an alternate position if the increased resistance corresponds to an obstacle in the path of the arm). In either case, the control processor 1202 then sends a signal to the control unit 1205 to control one or more actuators 1206 to change the arm position to the position indicated by the retrieved arm position information, and / or sends a signal to the surgical instrument 1208 to control the surgical instrument 1208 to enter the operating state indicated by the retrieved operating state information (e.g., if the surgical instrument 1208 is a cutting tool, to turn the electric blade to the "on" state or the "off" state).

[0113] FIG. 9 schematically illustrates another example of a computer-assisted surgery system 1300 to which the present technology can be applied. The computer-assisted surgery system 1300 is a computer-assisted medical scope system in which an autonomous arm 1100 holds an imaging device 1102 (e.g., a medical scope such as an endoscope, microscope, or exoscope). The imaging device of the autonomous arm outputs an image of the surgical scene to an electronic display (not shown) that the surgeon can view. The autonomous arm autonomously adjusts the view of the imaging device while the surgeon is performing the surgery to provide the surgeon with an appropriate view of the surgical scene in real time. The autonomous arm 1100 is the same as that shown in FIG. 7 and will not be described further. However, in this case, the autonomous arm is provided as part of a standalone computer-assisted medical scope system 1300, rather than as part of the master-slave system 1126 of FIG. 7. Therefore, the autonomous arm 1100 can be used in many different surgical setups, including laparoscopic surgery (where the medical scope is an endoscope) and open surgery.

[0114] The computer-aided medical scope system 1300 also includes a robotic control system 1302 for controlling the autonomous arm 1100. The robotic control system 1302 includes a control processor 1303 and a database 1304. Wired or wireless signals are exchanged between the robotic control system 1302 and the autonomous arm 1100 via connection 1301.

[0115] In response to control signals received from the robot control system 1302, the control unit 1115 controls one or more actuators 1116 to drive the autonomous arm 1100 to move the arm to an appropriate position for an appropriate view of the image to be captured by the imaging device 1102. The control signals are generated by the control processor 1303 in response to signals received from one or more of the arm unit 1114, the imaging device 1102, and any other signal sources (not shown). The signal values received by the control processor 1303 are compared with signal values stored in the database 1304 along with corresponding arm position information. The control processor 1303 retrieves the arm position information associated with the received signal value from the database 1304. The control processor 1303 then generates a control signal to send to the control unit 1115 using the retrieved arm position information.

[0116] For example, if the signals received from the imaging device 1102 indicate a predetermined surgical scenario (e.g., via a neural network image classification process, etc.), the predetermined surgical scenario is looked up in database 1304, and arm position information associated with the predetermined surgical scenario is retrieved from the database. As another example, if the signals indicate a value of resistance measured by one or more force sensors 1117 of the arm unit 1114, the value of the resistance is looked up in database 1203, and arm position information associated with the value of the resistance is retrieved from the database (e.g., so that the position of the arm can be changed to an alternate position if the increased resistance corresponds to an obstacle in the path of the arm). In either case, the control processor 1303 then sends a signal to the control unit 1115 to control one or more actuators 1116 to change the position of the arm to the position indicated by the retrieved arm position information.

[0117] 10 schematically illustrates another example of a computer-assisted surgery system 1400 to which the present technology can be applied. The system includes one or more autonomous arms 1100 having an imaging unit 1102 and one or more autonomous arms 1210 having a surgical instrument 1210. The one or more autonomous arms 1100 and the one or more autonomous arms 1210 are the same as those described above. Each of the autonomous arms 1100 and 1210 is controlled by a robotic control system 1408 including a control processor 1409 and a database 1410. Wired or wireless signals are transmitted between the robotic control system 1408 and each of the autonomous arms 1100 and 1210 via connections 1411 and 1412, respectively. The robotic control system 1408 performs the functions of the robotic control systems 1111 and / or 1302 described above for controlling each of the autonomous arms 1100, and performs the functions of the robotic control system 1201 described above for controlling each of the autonomous arms 1210.

[0118] The autonomous arms 1100 and 1210 perform at least a portion of the surgery fully autonomously (e.g., if the system 1400 is an open surgery system). The robotic control system 1408 controls the autonomous arms 1100 and 1210 to perform predetermined actions during the surgery based on input information indicative of the current stage of the surgery and / or events occurring in the surgery. For example, the input information includes images captured by the imaging device 1102. The input information may also include audio captured by a microphone (not shown), detection of a surgical instrument in use based on motion sensors included in the surgical instrument (not shown), and / or any other suitable input information.

[0119] The input information is analyzed using a suitable machine learning (ML) algorithm (e.g., a suitable artificial neural network) executed by a machine learning based surgical planning device 1402. The planning device 1402 includes a machine learning processor 1403, a machine learning database 1404, and a trainer 1405.

[0120] The machine learning database 1404 includes information indicative of classifications of surgical steps (e.g., making an incision, removing an organ, or applying a suture) and / or surgical events (e.g., bleeding or patient parameters outside of a predetermined range) and previously known input information corresponding to those classifications (e.g., one or more images captured by the imaging device 1102 during each classified surgical step and / or surgical event). The machine learning database 1404 is populated during a training phase by providing information indicative of each classification and the corresponding input information to the trainer 1405. The trainer 1405 then uses this information to train a machine learning algorithm (e.g., by using the information to determine appropriate artificial neural network parameters). The machine learning algorithm is executed by the machine learning processor 1403.

[0121] Once trained, the machine learning algorithm can classify previously unseen input information (e.g., newly captured images of a surgical scene) to determine the surgical stage and / or surgical event associated with the input information. The machine learning database also includes action information indicating the action that each of the autonomous arms 1100 and 1210 should take in response to each surgical stage and / or surgical event stored in the machine learning database (e.g., controlling the autonomous arm 1210 to make an incision at a relevant location during the surgical stage of "making an incision" and controlling the autonomous arm 1210 to perform an appropriate cauterization during the surgical event of "bleeding"). Thus, the machine learning-based surgical planning device 1402 can determine the associated action that the autonomous arms 1100 and / or 1210 should take in response to the surgical stage and / or surgical event classification output by the machine learning algorithm. The information indicating the associated action is provided to the robotic control system 1408, which then provides a signal to the autonomous arms 1100 and / or 1210 to perform the associated action.

[0122] The planning device 1402 may be included in a control unit 1401 that has a robotic control system 1408, thereby enabling direct electronic communication between the planning device 1402 and the robotic control system 1408. Alternatively or additionally, the robotic control system 1408 may receive signals from other devices 1407 via a communication network 1405 (e.g., the Internet). This allows the autonomous arms 1100 and 1210 to be remotely controlled based on processing performed by the other devices 1407. In one example, the device 1407 is a cloud server with sufficient processing power to quickly run complex machine learning algorithms, thereby arriving at more reliable surgical stage and / or surgical event classifications. Different machine learning algorithms may be run by each device 1407 using the same training data stored in an external (e.g., cloud-based) machine learning database 1406 accessible to each of the different devices 1407. Thus, each of the devices 1407 does not need its own machine learning database (such as the machine learning database 1404 of the planning device 1402) and the training data is centrally updated and available to all of the devices 1407. Each of the devices 1407 further includes a trainer (such as trainer 1405) and a machine learning processor (such as the machine learning processor 1403) to execute its respective machine learning algorithm.

[0123] 11 shows an example of the arm unit 1114. The arm unit 1204 is similarly configured. In this example, the arm unit 1114 supports an endoscope as the imaging device 1102. However, in other examples, another imaging device 1102 or a surgical instrument 1103 (in the case of the arm unit 1114) or 1208 (in the case of the arm unit 1204) is supported.

[0124] The arm unit 1114 includes a base 710 and an arm 720 extending from the base 720. The arm 720 includes a plurality of active joints 721a-721f and supports the endoscope 1102 at the distal end of the arm 720. The connecting portions 722a-722f are generally rod-shaped members. The ends of the plurality of connecting portions 722a-722f are connected to each other by the active joints 721a-721f, a passive slide mechanism 724, and a passive joint 726. The base unit 710 acts as a fulcrum so that the arm shape extends from the base unit 710.

[0125] The position and posture of the endoscope 1102 are controlled by driving and controlling the actuators provided on the active joints 721a to 721f of the arm 720. In this example, the distal end of the endoscope 1102 is inserted into a patient's body cavity, which is the treatment site, to capture an image of the treatment site. However, the endoscope 1102 may also be another device, such as another imaging device or a surgical instrument. More generally, the device held at the end of the arm 720 is called a distal unit or distal device.

[0126] Here, the arm unit 700 will be described by defining the coordinate axes shown in FIG. 11 as follows. Furthermore, the vertical direction, longitudinal direction, and horizontal direction are defined according to the coordinate axes. That is, the vertical direction relative to the base 710 placed on the floor is defined as the z-axis direction and vertical direction. Furthermore, the direction perpendicular to the z-axis and in which the arm 720 extends from the base 710 (in other words, the direction in which the endoscope 1102 is positioned relative to the base 710) is defined as the y-axis direction and longitudinal direction. Furthermore, the direction perpendicular to the y-axis and z-axis is defined as the x-axis direction and horizontal direction.

[0127] The active joints 721a to 721f rotatably connect the connecting parts to each other. The active joints 721a to 721f each have an actuator and a rotation mechanism that is driven to rotate around a predetermined rotation axis by driving the actuator. By controlling the rotational drive of each of the active joints 721a to 721f, it is possible to control the drive of the arm 720, for example, to extend or retract (fold) the arm unit 720.

[0128] The passive slide mechanism 724 is one aspect of a passive configuration change mechanism, and connects the connecting portion 722c and the connecting portion 722d so that they can move back and forth along a predetermined direction. The passive slide mechanism 724 is operated by, for example, a user to move forward and backward, and the distance between the active joint 721c on one end of the connecting portion 722c and the passive joint 726 is variable. This allows the overall configuration of the arm unit 720 to be changed.

[0129] Passive joint 736 is one aspect of a passive configuration change mechanism, and rotatably connects connecting portion 722d and connecting portion 722e to each other. Passive joint 726 is rotated by, for example, a user, and the angle formed between connecting portion 722d and connecting portion 722e is variable. This allows the overall configuration of arm unit 720 to be changed.

[0130] In one embodiment, the arm unit 1114 has six active joints 721a to 721f, and six degrees of freedom are realized for driving the arm 720. That is, while the drive control of the arm unit 1114 is realized by drive control of the six active joints 721a to 721f, the passive slide mechanism 726 and the passive joint 726 are not subject to drive control.

[0131] 11, the active joints 721a, 721d, and 721f are arranged so that the rotation axis direction is the longitudinal axis of the connected coupling units 722a and 722e and the imaging direction of the connected endoscope 1102. The active joints 721b, 721c, and 721e are arranged so that the rotation axis direction is the x-axis direction, that is, the direction in which the connection angle between the connected coupling units 722a to 722c, 722e, and 722f and the endoscope 1102 changes within the yz plane (the plane defined by the y-axis and z-axis). In this way, the active joints 721a, 721d, and 721f have a function of performing so-called yawing, and the active joints 421b, 421c, and 421e have a function of performing so-called pitching.

[0132] In the arm unit 1114, six degrees of freedom are realized for driving the arm 720, and therefore the endoscope 1102 can be moved freely within the movable range of the arm 720. Fig. 11 shows a hemisphere as an example of the movable range of the endoscope 723. If the center point RCM (remote motion center) of the hemisphere is set as the imaging center of the treatment site imaged by the endoscope 1102, then by moving the endoscope 1102 on the spherical surface of the hemisphere while the imaging center of the endoscope 1102 is fixed to the center point of the hemisphere, it is possible to image the treatment site from various angles.

[0133] FIG. 12 shows an example of a master console 1104. Two control units 900R and 900L are provided, one for the right hand and one for the left hand. The surgeon places both arms or elbows on the support table 50 and grasps the operation units 1000R and 1000L with his right and left hands, respectively. In this state, the surgeon operates the operation units 1000R and 1000L while looking at an electronic display 1110 showing the surgical site. The surgeon may remotely control the position or direction of each operation unit 1000R and 1000L attached to one or more slave devices, or perform a grasping action using each surgical instrument, by displacing the position or direction of each operation unit 1000R and 1000L.

[0134] Some embodiments of the present technology are defined by the following numbered clauses: (1) 1. A computer-assisted surgery system including an image capture device, a display, a user interface, and circuitry, the circuitry comprising: receiving information indicative of a surgical scenario and a surgical process associated with the surgical scenario; acquiring an artificial image of the surgical scenario; outputting the artificial image for display on the display; receiving, via the user interface, permission information indicating whether permission is granted to execute the surgical process if it is determined that the surgical scenario will occur; The computer-assisted surgery system is configured to: (2) The circuit receiving an actual image captured by the image capturing device; determining whether the actual image indicates the occurrence of the surgical scenario; If the actual image indicates the occurrence of the surgical scenario, determining whether there is permission to perform the surgical process; and If permission to execute the surgical process is given, control is performed to execute the predetermined process. 2. The computer-assisted surgery system according to claim 1, configured to: (3) the artificial image is obtained using a visualization of features of an artificial neural network configured to output information indicative of the surgical scenario when a real image of the surgical scenario captured by the image capture device is input to the artificial neural network; and the real image is determined to indicate the occurrence of the surgical scenario if, when the real image is input to the artificial neural network, the artificial neural network outputs information indicative of the surgical scenario; 3. The computer-assisted surgery system according to claim 2. (4) 10. The computer-assisted surgery system of claim 1, wherein the surgical process includes controlling a surgical device to perform a surgical operation. (5) 10. The computer-assisted surgery system according to any preceding claim, wherein the surgical process includes adjusting the field of view of an image capturing device. (6) the surgical scenario is one in which bodily fluids may impinge on the imaging device; and the surgical process includes adjusting the position of the image capture device to reduce the risk of collision. 6. The computer-assisted surgery system of claim 5. (7) the surgical scenario is one in which a different field of view of the image capture device would be beneficial; and the surgical process includes adjusting the field of view of the image capture device to the different field of view. 6. The computer-assisted surgery system of claim 5. (8) The surgical scenario is a surgical scenario in which an incision is made, and the different view provides an improved view of the performance of the incision; 8. The computer-assisted surgery system of claim 7. (9) the surgical scenario includes the incision deviating from the planned incision; and the different field of view providing an improved view of the deviation. 9. The computer-assisted surgery system of claim 8. (10) the surgical scenario is a surgical scenario in which an object is dropped, the surgical process includes adjusting the field of view of the image capture device to keep the dropped item within the field of view. 6. The computer-assisted surgery system of claim 5. (11) the surgical scenario is a surgical scenario in which evidence of an event not within the field of view of the image capture device is within the field of view; the surgical process includes adjusting the field of view of the image capture device so that the event is within the field of view. 6. The computer-assisted surgery system of claim 5. (12) 12. The computer-assisted surgery system according to claim 11, wherein the event is bleeding. (13) the surgical scenario is a surgical scenario in which an object obstructs the field of view of the image capture device; the surgical process includes adjusting the field of view of the image capture device to avoid the obstructing object. 6. The computer-assisted surgery system of claim 5. (14) the surgical scenario is a surgical scenario in which a working area approaches a boundary of the field of view of the image capture device; the surgical process includes adjusting the field of view of the image capture device so that the working area remains within the field of view. 6. The computer-assisted surgery system of claim 5. (15) the surgical scenario is a surgical scenario in which the image capturing device may collide with another object, the surgical process includes adjusting the position of the image capture device to reduce the risk of collision. 6. The computer-assisted surgery system of claim 5. (16) The circuit comparing the real image with the artificial image; and If the similarity between the real image and the artificial image exceeds a predetermined threshold, the surgical process is executed. 4. The computer-assisted surgery system according to claim 2 or 3, configured as follows: (17) The surgical process is one of a plurality of surgical processes that can be executed when it is determined that the surgical scenario will occur; each of the plurality of surgical processes is associated with a respective level of invasiveness; and Each surgical process other than the surgical process is permitted to be performed if the invasiveness level of the other surgical process is equal to or less than the invasiveness level of the previous surgical process, and if permission to perform the surgical process is permitted, 2. A computer-assisted surgery system according to any one of the preceding claims. (18) 2. A computer-assisted surgery system according to any preceding claim, wherein the image capturing device is a surgical camera or a medical visual scope. (19) The computer-assisted surgery system according to any one of the preceding claims, wherein the computer-assisted surgery system is a computer-assisted medical visual scope system, a master-slave system, or an open surgery system. (20) 1. A surgical control device, comprising: receiving information indicative of a surgical scenario and a surgical process associated with the surgical scenario; acquiring an artificial image of the surgical scenario; outputting the artificial image for display on a display; receiving, via a user interface, permission information indicating whether permission is granted to execute the surgical process if it is determined that the surgical scenario will occur; A surgical control device comprising a circuit configured to: (twenty one) 1. A surgical control method, comprising: receiving information indicative of a surgical scenario and a surgical process associated with the surgical scenario; acquiring an artificial image of the surgical scenario; outputting the artificial image for display on the display; receiving, via the user interface, permission information indicating whether permission to execute the surgical process is available if it is determined that the surgical scenario will occur; A surgical control method comprising: (twenty two) 22. A program for controlling a computer to execute the surgical control method according to claim 21. (twenty three) 23. A non-transitory storage medium storing the computer program according to claim 22.

[0135] Many modifications and variations of the present disclosure are possible in light of the above teachings. It is therefore to be understood that, within the scope of the appended claims, the present disclosure may be practiced other than as specifically described herein.

[0136] To the extent that embodiments of the present disclosure are described as being implemented at least in part by a software-controlled data processing apparatus, it will be understood that non-transitory machine-readable media containing software, such as optical disks, magnetic disks, semiconductor memories, and the like, are also considered to represent embodiments of the present disclosure.

[0137] It will be appreciated that, for clarity, the above description has described embodiments with reference to different functional units, circuits and / or processors. However, it will be apparent that any suitable distribution of functionality between different functional units, circuits and / or processors may be used without detracting from the embodiments.

[0138] The described embodiments may be implemented in any suitable form including hardware, software, firmware, or any combination of these. The described embodiments may optionally be implemented at least partly as computer software running on one or more data processors and / or digital signal processors. The elements and components of any embodiment may be physically, functionally, and logically implemented in any suitable way. Indeed, functionality may be implemented in a single unit, in multiple units, or as part of other functional units. Thus, the disclosed embodiments may be implemented in a single unit, or may be physically and functionally distributed between different units, circuits, and / or processors.

[0139] Although the present disclosure has been described in connection with some embodiments, it is not intended to be limited to the specific form set forth herein. Moreover, while features may appear to be described in connection with particular embodiments, those skilled in the art will recognize that the various features of the described embodiments may be combined in any manner suitable for practicing the present technology.

Claims

1. 1. A computer-assisted surgery system comprising an image capture device, a display, a user interface, a storage medium, and circuitry, the circuitry comprising: receiving information indicating a surgical scenario and a surgical process associated with the surgical scenario from the storage medium; obtaining an artificial image of the surgical scenario from an artificial neural network; outputting the artificial image for display on the display; receiving, via the user interface, permission information indicating whether permission is granted to execute the surgical process if it is determined that the surgical scenario will occur; receiving an actual image captured by the image capturing device; determining whether the actual image indicates the occurrence of the surgical scenario; If the actual image indicates the occurrence of the surgical scenario, determining whether there is permission to perform the surgical process; and Controlling a surgical robot to perform the surgical process if permission to perform the surgical process is granted It is configured as follows: Computer-assisted surgery system.

2. the artificial image is obtained using a visualization of features of the artificial neural network, the artificial neural network being configured to output information indicative of the surgical scenario when a real image of the surgical scenario captured by the image capture device is input to the artificial neural network; and the real image is determined to indicate the occurrence of the surgical scenario if, when the real image is input to the artificial neural network, the artificial neural network outputs information indicative of the surgical scenario; The computer-assisted surgery system of claim 1 .

3. The computer-assisted surgery system of claim 1 , wherein the surgical process includes controlling a surgical device to perform a surgical action.

4. The computer-assisted surgery system of claim 1 , wherein the surgical process includes adjusting a field of view of an image capture device.

5. the surgical scenario is one in which bodily fluids may impinge on the imaging device; and the surgical process includes adjusting the position of the image capture device to reduce the risk of collision. The computer-assisted surgery system of claim 4.

6. the surgical scenario is one in which a different field of view of an image capture device would be beneficial; the surgical process includes adjusting the field of view of the image capture device to the different field of view. The computer-assisted surgery system of claim 4.

7. The surgical scenario is a surgical scenario in which an incision is made, and the different view provides an improved view of the performance of the incision; The computer-assisted surgery system of claim 6.

8. the surgical scenario includes the incision deviating from a planned incision; and the different field of view providing an improved view of the deviation. The computer-assisted surgery system of claim 7.

9. the surgical scenario is a surgical scenario in which an object is dropped, the surgical process includes adjusting the field of view of the image capture device to keep the dropped item within the field of view. The computer-assisted surgery system of claim 4.

10. the surgical scenario is a surgical scenario in which evidence of an event not within the field of view of the image capture device is within the field of view; the surgical process includes adjusting the field of view of the image capture device so that the event is within the field of view. The computer-assisted surgery system of claim 4.

11. The computer-assisted surgery system of claim 10 , wherein the event is bleeding.

12. the surgical scenario is a surgical scenario in which an object obstructs the field of view of the image capture device; the surgical process includes adjusting the field of view of the image capture device to avoid the obstructing object. The computer-assisted surgery system of claim 4.

13. the surgical scenario is a surgical scenario in which a working area approaches a boundary of the field of view of the image capture device; the surgical process includes adjusting the field of view of the image capture device so that the working area remains within the field of view. The computer-assisted surgery system of claim 4.

14. the surgical scenario is a surgical scenario in which the image capturing device may collide with another object, the surgical process includes adjusting the position of the image capture device to reduce the risk of collision. The computer-assisted surgery system of claim 4.

15. The circuit comparing the real image with the artificial image; and If the similarity between the real image and the artificial image exceeds a predetermined threshold, the surgical process is executed.

2. The computer-assisted surgery system according to claim 1, configured to:

16. the surgical process is one of a plurality of surgical processes that can be executed when it is determined that the surgical scenario will occur; each of the plurality of surgical processes is associated with a respective level of invasiveness; and each surgical process in the plurality of surgical processes other than the surgical process is permitted to be performed by the circuit if the surgical process has permission to be performed when an invasiveness level of the surgical process is equal to or less than the invasiveness level of the surgical process; The computer-assisted surgery system of claim 1 .

17. The computer-assisted surgery system of claim 1 , wherein the image capture device is a surgical camera or a medical vision scope.

18. The computer-assisted surgery system according to claim 1 , wherein the computer-assisted surgery system is a computer-assisted medical visual scope system, a master-slave system, or an open surgery system.

19. A surgical control device including a display interface, a user interface, a storage medium, and circuitry, receiving information indicating a surgical scenario and a surgical process associated with the surgical scenario from the storage medium; obtaining an artificial image of the surgical scenario from an artificial neural network; outputting the artificial image for display to a display via the display interface; receiving, via the user interface, permission information indicating whether permission is granted to execute the surgical process if it is determined that the surgical scenario will occur; receiving an actual image captured by an image capturing device; determining whether the actual image indicates the occurrence of the surgical scenario; If the actual image indicates the occurrence of the surgical scenario, determining whether there is permission to perform the surgical process; and Controlling a surgical robot to perform the surgical process if permission to perform the surgical process is granted A surgical control device comprising a circuit configured to:

20. 1. A surgical control method, comprising: receiving information indicating a surgical scenario and a surgical process associated with the surgical scenario from a storage medium; obtaining an artificial image of the surgical scenario from an artificial neural network; outputting the artificial image for display on a display; receiving, via a user interface, permission information indicating whether permission to execute the surgical process is available if it is determined that the surgical scenario will occur; receiving an actual image captured by an image capture device; determining whether the actual image indicates an occurrence of the surgical scenario; If the actual image indicates the occurrence of the surgical scenario, determining whether there is permission to perform the surgical process; and If permission to perform the surgical process is granted, controlling the surgical robot to perform the surgical process; Including, Surgical control method.

21. 21. A program for controlling a computer to execute the surgical control method of claim 20.

22. A non-transitory storage medium that stores the program according to claim 21.

Images