Surgery support system, data processing device, and method

Abstract

A surgical support system includes: an eye behavior monitoring device, which is operable to monitor the eye behavior of a surgeon performing a surgical treatment to obtain the surgeon's eye behavior data; a surgical data generating device, which is operable to generate surgical data associated with the surgical treatment performed by the surgeon; a surgical intervention device, which is operable to perform intervention treatment for intervening in the surgeon's performance of the surgical treatment; and a data processing device, which is operable to: determine a value of an intervention parameter associated with the surgical treatment using the obtained eye behavior data; determine an acceptable range of the value of the intervention parameter using the generated surgical data; and if the value of the determined intervention parameter is outside the acceptable range of the value of the determined intervention parameter, control the surgical intervention device to perform intervention treatment for intervening in the surgeon's performance of the surgical treatment.

Description

Technical Field

[0001] The present disclosure relates to a surgery support system, a data processing device, and a method. Background Art

[0002] The "background" description provided herein is for the purpose of presenting the context of the present disclosure as a whole. To the extent described in the background section, the works of the currently named inventors and the aspects of the description that were not otherwise prior art at the time of filing are neither explicitly nor implicitly admitted as prior art to the present disclosure.

[0003] Serious surgical errors are referred to as "never events" because they are usually avoidable and therefore do not occur. Never events include: leaving surgical tools inside a patient and operating on the wrong patient. Less serious (but still dangerous and expensive) errors such as a serious severance of a blood vessel are referred to as technical errors. Despite having processes such as surgical timeouts and checklists in place to prevent this, surgical errors continue to occur in medical facilities around the world. For example, serious cases have been found to increase in the UK in recent years and studies have shown that medical errors are the third leading cause of death in the United States.

[0004] Surgical errors range from wrong-site surgery or incorrect treatment to a small slip of the hand in a dangerous spot, and each can impose serious risks on the patient. Common errors occur due to lack of information, which increases the level of uncertainty surgeons experience when making decisions. For example, this can result from poor surgical team or healthcare provider communications or missing data. Surgical specialists also experience their own unique problems, making error classification more complicated.

[0005] Human factors are also a major (and unpredictable) cause of surgical errors - making poor decisions, choosing the wrong technique for a specific action, and simply performing an action incorrectly can have serious consequences for the patient and the hospital. Further, surgeons may avoid confessing mistakes or not having sufficient knowledge during treatment in order to escape workplace or legal accountability, a problem that could be avoided if there were comprehensive checks to prevent these mistakes from happening in the first place.

[0006] The future may see increased collaboration between surgical robots and human surgeons in operating rooms, which may provide a way to reduce the number of avoidable surgical errors that occur. The system enables adaptable task responsibility relationships, whereby the roles of the robot and the human can change depending on the status of the surgery, and the robot can act as an additional safety net to prevent surgical errors from occurring.

[0007] However, there remain questions about how to successfully achieve collaboration between surgical robots and human surgeons, and more generally, how to further reduce the number of avoidable surgical errors that occur. Summary of the invention

[0008] The present disclosure is defined by the following claims.

[0009] The above paragraphs have been provided by way of general introduction and are not intended to limit the scope of the following claims.The described embodiments together with further advantages can be better understood by reference to the following detailed description taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] A more complete appreciation of the present disclosure and many of its attendant advantages will be readily obtained as it becomes more readily understood by reference to the following detailed description when considered in conjunction with the accompanying drawings, in which:

[0011] [ Figure 1 ] Figure 1 schematically illustrates a surgical support system according to an embodiment;

[0012] [ Figure 2 ] Figure 2 schematically illustrates exemplary information used by a surgical support system in determining an appropriate surgical intervention therapy according to an embodiment;

[0013] [ Figure 3 ] Figure 3 A first exemplary implementation of the present technology is schematically shown;

[0014] [ Figure 4A ] Figure 4A A second exemplary implementation of the present technology is schematically shown;

[0015] [ Figure 4B ] Figure 4B A second exemplary implementation of the present technology is schematically shown;

[0016] [ Figure 5A ] Figure 5A A third exemplary implementation of the present technology is schematically shown;

[0017] [ Figure 5B ] Figure 5B A third exemplary implementation of the present technology is schematically shown;

[0018] [ Figure 6 ] Figure 6 A flow chart showing a first method according to an embodiment is shown; and

[0019] [ Figure 7 ] Figure 7 A flow chart showing a second method according to an embodiment is shown. DETAILED DESCRIPTION

[0020] Reference is now made to the drawings, wherein like reference numerals designate identical or corresponding parts throughout the several views.

[0021] The present technology uses eye behavior monitoring technology to collect eye behavior data to help evaluate the performance of human surgeons and intervene during surgical treatment when the eye behavior data indicates possible surgical errors.

[0022] Eye behavior monitoring technology is relatively well established and implemented through various existing technologies for the purposes of augmented reality (ARA) and virtual reality (VR) platforms. An advantage achieved by developing eye behavior monitoring technology is the ability to identify content that an individual is visually interested in. Although this capability is currently most widely used in AR and VR headsets, it also has applications in advertising, market research, and medical accessibility.

[0023] At its most basic level, eye movements indicate what a person is looking at. However, eye movements can also be used to gain information about a person's ability to perform in a particular situation. Studies have shown that expert and junior doctors behave differently in their gaze patterns during surgery due to individual skill levels (i.e., the pattern of movement of locations in space that the doctor is looking at) (NPL 1). Similarly, the subtle eye movements made by elite horse riders when planning a jump can indicate their likely success (NPL 2). The visual strategy of an experienced horse rider differs from that of a less competitive horse rider in the amount of time the rider spends visually fixating on a specific phase of a jump and how early the fixation begins.

[0024] Research has also shown that novice performers are able to learn the eye movements of more skilled individuals. After observing the eye movements of expert pilots, novice military pilots were found to have improved visual responses to simulated emergency procedures (NPL 3). In this case, in more than 80% of cases, the difference between novice and skilled pilots was so significant that the algorithm was able to distinguish between the two.

[0025] Specifically, aspects of eye behavior that are expected to go awry include:

[0026] 1. Short dwell time (time spent viewing the area of ​​interest).

[0027] 2. Reduce fixation frequency (the number of times a fixation is focused on an area of ​​interest in a given period of time)

[0028] 3. Glance rate and distance. Glance refers to the rapid movement of the eyes that suddenly changes the fixation point. Glances can be used to distinguish expert from novice drivers. Specifically, experts have a smaller glance rate and distance than novices because they can use peripheral vision more effectively than novices. Glance rate refers to the speed of eye movement when changing the fixation point during a glance. Glance distance refers to the distance the eyes move when changing the fixation point during a glance.

[0029] 4. Overview of important visual features.

[0030] 5. Mismatch between pupil dilation (reflecting effort) and the difficulty of the task being assessed. Pupil dilation increases with increasing cognitive load and decreases with decreasing cognitive load. Lack of pupil dilation during difficult tasks indicates a lack of concentration or cognitive engagement.

[0031] Embodiments of the present technology use eye behavior monitoring technology to help reduce surgical errors. This is accomplished by addressing one or more of the following:

[0032] 1. How to preempt and prevent surgical errors without unduly distracting the surgeon during the operation.

[0033] 2. How to determine how the surgeon's uncertainty affects the patient and allow for some degree of uncertainty where necessary (i.e., to prevent harm to the patient through inaction).

[0034] 3. How to choose the appropriate response to a predicted surgical error based on the potential severity and risk to the patient.

[0035] 4. How to reduce the chance of errors during high-pressure, dangerous situations that require surgeons to work quickly.

[0036] 5. How to accurately identify the cause of the surgeon's uncertainty within his field of view and implement a customized uncertainty response.

[0037] Figure 1 A surgical support system 100 according to an embodiment is shown. The surgical support system 100 includes an eye behavior monitoring device (eye monitor) 101, one or more surgical data generating devices (e.g., one or more devices 109 to 115), one or more surgical intervention devices (e.g., one or more devices 109 to 115), and a data processing device 104.

[0038] The eye behavior monitoring device 101 is operable to perform eye behavior monitoring of a surgeon performing a surgical procedure to obtain eye behavior data of the surgeon. In an embodiment, the eye behavior monitoring device uses any suitable eye tracking technology known in the art to obtain eye tracking data. The eye tracking data includes data indicating the surgeon's gaze (i.e., the point in space that the surgeon is looking at). For example, the gaze is determined by the eye behavior monitoring device based on the position of the surgeon's pupil relative to the surgeon's head and / or the position of the surgeon's head. In an embodiment, the eye behavior monitoring device uses any suitable pupil monitoring technology to obtain pupil monitoring data indicating the behavior of the pupil of each of one or more of the surgeon's eyes (e.g., the surgeon's pupil dilation or scanning rate and / or distance).

[0039] Thus, the eye behavior monitoring device 101 performs eye tracking and pupil monitoring to monitor the behavior of the surgeon's eyes. Eye tracking includes: tracking the surgeon's gaze (e.g., based on live captured video images of the surgeon's eyes) to determine eye tracking data (e.g., dwell time on an area of ​​interest, fixation frequency on an area of ​​interest, and / or whether an area of ​​interest has been ignored). The generated eye tracking data can be associated with information about the area of ​​interest. Pupil monitoring includes: monitoring the surgeon's pupils (e.g., based on live captured video images of the surgeon's eyes) to determine pupil monitoring data (e.g., the surgeon's pupil dilation and / or scanning rate and / or distance). Eye tracking data and pupil monitoring data are embodiments of eye behavior data.

[0040] Information may be derived from the eye behavior data by the data processing device 104 to indicate an increased likelihood that the surgeon will make an error during the surgical procedure.

[0041] In one embodiment, if the surgeon has an acceptably short dwell time on the region of interest, the likelihood of an error being present is considered increased. For example, the dwell time is unacceptably short if it is less than a predetermined dwell time threshold (e.g., the predetermined dwell time threshold is determined based on the treatment being performed and / or the stage of treatment). In an embodiment, the dwell time refers to the average time a surgeon spends viewing the region of interest during a surgical procedure (or during a particular stage of a surgical procedure).

[0042] In another embodiment, if the surgeon has an unacceptably low fixation frequency for the region of interest, then the likelihood of an error is considered to be increased. For example, the fixation frequency is unacceptably low if it is less than a predetermined fixation frequency threshold (e.g., the predetermined fixation frequency threshold is determined based on the treatment being performed and / or the stage of treatment).

[0043] In another embodiment, owing to having higher glance rate and / or distance, the possibility of error considered as occurring increases.For example, if greater than predetermined glance rate and / or distance threshold, then be considered as glance rate and / or distance as unacceptably high (for example, according to the treatment carried out and / or the stage for treatment, determine predetermined glance rate and / or distance threshold).

[0044] In another embodiment, after the surgeon ignores an important visual feature, the probability of an error being considered increased. A visual feature is ignored when the surgeon's gaze does not fall on the visual feature (indicating that the surgeon is not paying attention).

[0045] In another embodiment, if there is a mismatch between pupil dilation (reflecting effort) and the assessed task difficulty, then the likelihood of an error is considered to have increased. For example, if it is greater than a predetermined pupil dilation threshold, then the pupil dilation is unacceptably large (e.g., the predetermined pupil dilation threshold is determined based on the treatment being performed and / or the stage of treatment). In order to take into account the fact that different surgeons may have different corresponding ranges of pupil sizes, the pupil dilation and the predetermined pupil dilation threshold can be determined as a percentage of the reference pupil dilation. For example, if the size of the pupil diameter is doubled compared to the size of the reference pupil dilation, the pupil dilation is determined to be +50% (positive 50%). If the size of the pupil diameter is halved compared to the size of the reference pupil dilation, the pupil dilation is determined to be -50% (negative 50%). The reference pupil dilation is determined during a calibration procedure performed prior to surgery (e.g., the reference pupil dilation is obtained by having the surgeon look at a camera (not shown) of the eye behavior monitoring device 101 under predetermined lighting conditions and measuring the pupil dilation).

[0046] The data processing device 104 includes a processor 105, a memory 106, a communication interface 107, and a storage medium 108. The communication interface 107 is used to input electronic information to the data processing device 104 and output electronic information from the data processing device 104. For example, the eye behavior data from the eye behavior monitoring device 101 and the surgical data from one or more surgical data generating devices are received as input information through the communication interface 107. In addition, the intervention signal used to control one or more surgical intervention devices to perform intervention treatment is sent to one or more surgical intervention devices as output information through the communication interface 107. The communication interface 107 also allows data to be exchanged between any two devices connected to the communication interface 107. The processor 105 enables the data processing device 104 to complete its operation by processing appropriate electronic instructions. The memory 106 is used to store the electronic instructions processed by the processor 105 and to store the input and output data associated with the electronic instructions. The storage medium 108 (for example, in the form of a hard disk drive, a solid state drive, a tape drive, etc.) is used for long-term storage of data. For example, each of the processor 105 , the memory 106 , the communication interface 107 , and the storage medium 108 is implemented using suitable circuits.

[0047] The communication interface 107 is operable to receive surgical data associated with a surgical procedure performed by a surgeon. During the surgical procedure, surgical data is received from one or more surgical data generating devices. Figure 1 In the embodiment of , there are multiple surgical data generating devices, each providing surgical data to the communication interface 107.

[0048] An exemplary surgical data generating device is a surgical robot 109. The surgical robot 109 is a robotic system that assists surgeons in performing surgery. Various surgical robots are known in the art and surgeries involving surgical robots are known as robot-assisted surgeries. For example, the surgical data provided to the communication interface 107 by the surgical robot 109 includes values ​​of one or more parameters associated with one or more functions of the robot performed by the robot during the surgical treatment (e.g., the speed of the cutting blade used by the robot or the depth of the robot's robotic arm within the patient's body). For example, the one or more parameters are monitored by appropriate circuitry of the robot.

[0049] Another exemplary surgical data generating device is an external camera system 110. This includes one or more cameras (not shown) that capture images of the operating room in which the surgery is performed. For example, the captured images include the surgeon, the patient, other members of the surgical team, and other surgical devices in the room. The surgical data provided to the communication interface 107 by the external camera system 110 includes the captured images.

[0050] Another embodiment of a surgical data generating device is an internal camera system 111. This includes one or more cameras (not shown, for example, one or more endoscopic cameras known in the art) that capture images within the patient's body. For example, the captured images may also include the surgeon's hands and any surgical devices (e.g., a cutting tool held by the surgeon or the end of an arm of a surgical robot 109) that enter the part of the patient's body captured by the internal camera system during the operation. The surgical data provided to the communication interface 107 by the internal camera system 111 includes the captured images.

[0051] Another embodiment of the surgical data generating device is a patient monitoring system 112. The patient monitoring system monitors one or more physiological parameters of the patient during the operation and generates data indicating the value of each of these physiological parameters. This allows the patient's state to be monitored during the operation. An embodiment of a patient monitoring system is an electrocardiogram (ECG) that generates ECG data. The surgical data provided to the communication interface 107 by the patient monitoring system 112 includes data indicating the value of each monitored physiological parameter of the patient.

[0052] Another embodiment of the surgical data generating device is an audio monitoring system 113. This includes one or more microphones (not shown) that capture sounds in the operating room in which the surgery is performed. For example, the captured sounds include language uttered by the surgeon or other members of the surgical team and auditory signals (e.g., warnings) emitted by other surgical devices. The surgical data provided to the communication interface 107 by the audio monitoring system 113 includes data indicating the captured sounds. The data processing device 104 is configured to analyze the captured sounds to determine what the captured sounds are. For example, this is performed by performing a frequency analysis of the captured sounds and comparing the generated spectrum with the spectrum of known sounds stored in the storage medium 108 (suitable techniques thereof are known in the art). For example, this allows the data processing device 104 to discern the surgeon's language (allowing the surgeon to provide surgical data to the system via sound) or to discern the meaning of specific auditory signals emitted by other surgical devices.

[0053] Another embodiment of a surgical data generating device is a surgeon monitoring system 112. The surgeon monitoring system monitors one or more physiological parameters of the surgeon during surgery and generates data indicating the value of each of these physiological parameters. For example, the surgeon monitoring system includes a heart rate monitor that monitors the surgeon's heart rate and / or a sweat monitor that monitors how much the surgeon is sweating (perspiring). This helps allow the surgeon's stress level to be monitored. For example, when the surgeon is under more stress, his heart rate tends to increase and he tends to sweat more, and when the surgeon is under less stress, his heart rate tends to decrease and he tends to sweat less. The surgical data provided to the communication interface 107 by the surgeon monitoring system 114 includes data indicating the values ​​of the surgeon's various monitored physiological parameters.

[0054] Another embodiment of the surgical data generating device is a user interface 115. The user interface allows a user (e.g., a surgeon or another member of the surgical team) to manually enter surgical data provided to the communication interface 107. It also allows data (e.g., an alarm signal) output by the communication interface 107 to be provided to the user in an understandable manner. In one embodiment, the user interface 115 includes a combination of a touch screen or a head mounted display (HMD) and an input device (e.g., a handheld controller or a voice recognition device). The user interface 115 may also include a tactile feed device (e.g., an electric vibrator) and / or an audio speaker. In one embodiment, the surgical data entered by the user is data indicating the type of surgical treatment to be performed. For example, before starting the surgical treatment, the type of surgical treatment is selected using an appropriate graphical user interface (GUI) menu system displayed on the touch screen or HMD. In another embodiment, the data output by the communication interface 107 includes an alarm signal. In response to receiving the alarm signal, the touch screen or HMD outputs a visual alarm, the speaker (if present) outputs an auditory alarm, and the tactile feed device (if present) outputs a tactile alarm. Thus, an alarm is issued to the user by visual, auditory, and / or tactile alarms.

[0055] In the event that analysis of the surgeon's ocular behavior data indicates that intervention is required to reduce the risk of surgical error, the communication interface 107 is further operable to output an intervention signal to one or more surgical intervention devices. The one or more surgical intervention devices perform the intervention treatment in response to receiving the intervention signal.

[0056] In one embodiment, the intervention signal refers to an alarm signal and the intervention treatment includes outputting an alarm to the user. For example, the alarm signal can cause the user interface 115 to output an auditory, visual, and / or tactile alarm (as described above). In the case of an alarm, no action is taken to interrupt the treatment (i.e., the surgeon is not prevented from continuing the surgical treatment, and therefore, it is the surgeon's determination as to whether the surgical treatment continues). In this case, the user interface 115 is a surgical intervention device. Further, the alarm signal can be changed according to the amount by which the error probability exceeds the acceptable error probability. For example, in the case where the error probability far exceeds the acceptable error probability (i.e., greater than a predetermined threshold), the output audio (e.g., a beep or a high-pitched alarm) of the alarm signal may be larger (i.e., above a predetermined volume threshold). In another embodiment, in the case where the alarm signal includes a visual alarm (e.g., a visible mark or notification indicating the error probability displayed on the user interface 115), the visual representation can be changed. For example, the visual representation can be displayed on multiple displays, displayed by a larger portion of the display, or displayed at a more central position of the display. Based on the comparison of the error probability to an acceptable error probability, the system may change the alarm signal from just one of an audible, visual, or tactile alarm to a combination of more than one thereof.

[0057] In another embodiment, the intervention signal is an adjustment signal, and the intervention treatment includes: adjusting the operation of a surgical device used by a surgeon to complete the surgical treatment to preempt an error in the surgical treatment. In the event of an adjustment, action is taken to interrupt the surgery to preempt an error (i.e., to prevent the surgeon from continuing the surgical treatment, and therefore, the surgeon does not make a determination as to whether the surgical treatment should continue). In this case, the surgical device whose operation is adjusted is a surgical intervention device.

[0058] exist Figure 1 In the embodiment of the present invention, the surgical robot 109 and the user interface 115 are embodiments of a surgical intervention device capable of adjusting its operation in response to receiving an adjustment signal from the communication interface 107.

[0059] An example of an adjustment made by the surgical robot 109 is when the surgeon is controlling the robot's arms to make a surgical incision on a patient. The robot receives the adjustment signal and in response temporarily suspends the function of the arms.

[0060] An example of an adjustment made via the user interface 115 is when the surgeon is observing his actions on the patient's body during the surgical procedure via a video feed (captured by an endoscopic camera of the internal camera system 111, etc.) displayed using the user interface 115. The user interface receives the adjustment signal and responsively temporarily pauses the display of the video feed.

[0061] In both embodiments, the surgeon is temporarily prevented (i.e., for a predetermined limited period of time, e.g., 30 seconds) from continuing with the surgical procedure to reduce the risk of errors occurring. For example, this allows the surgeon to mentally take a break and, if necessary, plan to make adjustments to his technique or consult with a fellow surgeon to reduce the likelihood of him making an error (thereby allowing the procedure to be performed safely).

[0062] The various surgical data generating devices that provide surgical data to the communication interface 107 do so via a suitable wired and / or wireless connection (as known in the art). The wired and / or wireless connection may be part of a computer network such as the Internet. The various surgical intervention devices that receive intervention signals from the communication interface 107 do so via a suitable wired or wireless connection (as known in the art). The wired and / or wireless connection may be part of a computer network such as the Internet.

[0063] In an embodiment, the data processing device 104 uses the eye behavior data to determine the likelihood of the surgeon making an error in the surgical treatment. The eye behavior data is obtained by the eye behavior monitoring device 101 and sent to the communication interface 107 via a suitable wired or wireless connection (as known in the art). The wired and / or wireless connection can be part of a computer network such as the Internet. The data processing device 104 uses the surgical data to determine the acceptable likelihood of the surgeon making an error in the surgical treatment. If the likelihood of the surgeon making an error in the surgical treatment (referred to as the determined error likelihood) exceeds the acceptable likelihood of the surgeon making an error in the surgical treatment (referred to as the acceptable error likelihood), the data processing device 104 controls one or more surgical intervention devices to perform appropriate intervention treatment by transmitting appropriate intervention signals (e.g., alarm signals and / or adjustment signals).

[0064] In an embodiment, by using the surgeon's eye behavior monitoring to determine the surgeon's likelihood of making an error and thereby determining appropriate intervention, the occurrence of errors becomes more predictable and reduced. At the same time, unnecessary interruptions to the surgeon's workflow are reduced because the surgeon has to perform any additional actions to perform eye behavior monitoring during the surgical procedure.

[0065] In an embodiment, the surgical data indicates at least one of a type of surgical procedure performed and a current stage of the surgical procedure. The data processing device 104 uses the surgical data to determine an acceptable probability of a surgeon making an error in the surgical procedure.

[0066] The type of surgical treatment identifies which surgical treatment the surgeon is performing. For example, the surgical treatment type may be a heart transplant, a coronary artery bypass graft, or a knee replacement. The current stage of the surgical treatment is one of multiple stages of the surgical treatment. Each stage is a sub-treatment of the surgical treatment. The various stages of the surgical treatment must be successfully completed in the proper order to allow the surgical treatment to be successfully completed. The necessary stages (and their order) are typically different for different surgical treatment types. Examples of stages include: making an incision on a particular organ of the patient; removing or removing a portion of an organ of the patient; and suturing the incision on the particular organ of the patient.

[0067] In an embodiment, each type of surgical treatment and / or one or more stages of a surgical treatment type is associated with one or more corresponding potential error events. If the surgeon makes an error, a potential error event refers to an event that potentially occurs during the current stage (e.g., accidental damage to the patient's healthy organs). Each potential error event is associated with a corresponding acceptable error probability of the potential error event occurring. During the surgical treatment, the surgeon's eye behavior data is used to determine the determined error probability of each related potential error event occurring at a given time. For the potential error event, if the determined error probability of the potential error event exceeds the acceptable error probability, appropriate intervention treatment is performed.

[0068] Typically, a potential error event may be associated with a surgical procedure. In this case, the determined error likelihood continues to be updated based on the surgeon's eye behavior data for the entire duration of the surgical procedure. Alternatively, a potential error event may be associated with a specific phase of the surgical procedure. In this case, the determined error likelihood continues to be updated based on the surgeon's eye behavior data only for the duration of that specific phase.

[0069] In an embodiment, information indicating the type of surgical procedure is entered by the user before the surgical procedure is started via the user interface 115. This information is provided as surgical data to the data processing device 104. The data processing device 104 looks up the indicated type of surgical procedure in a database stored in the storage medium 108. The database identifies each stage of the indicated type of surgical procedure, each potential error event associated with the indicated type of surgical procedure (and the stage of the type of surgical procedure associated with the potential error event, if applicable), and an acceptable error probability associated with each identified potential error event.

[0070] Figure 2 An example of information stored in a database is shown in FIG.

[0071] Figure 2The type of surgical treatment in the example is a heart transplant. The heart transplant is shown to have three stages ("Stage 1", "Stage 2", and "Stage 3"), each stage being associated with one or more potential error events. Each potential error event is in turn associated with an acceptable probability of error and one or more interventional treatments. It should be recognized that this is a simplified embodiment for ease of illustration. In practice, a surgical treatment such as a heart transplant may have many more stages, and each stage may have many more associated potential error events.

[0072] although Figure 2 , however, potential error events may be generally associated with cardiac transplantation as a surgical treatment (rather than just with a specific stage of cardiac transplantation). In this case, potential error events and their associated acceptable error probabilities and one or more interventional treatments are identified in the database but are not associated with a specific stage of cardiac transplantation. In this case, for example, the "Stage" column may read "All" (indicating that potential error events and their associated acceptable error probabilities and one or more interventional treatments are applicable to all stages of cardiac transplantation).

[0073] Potential error events may also be of a general type. A general potential error event refers to a potential error event that is not exclusive to a specific type of error occurring (e.g., at a specific part of the patient's body), but generally has a higher acceptable likelihood of occurring in situations where the surgeon is at increased risk of an error. For example, even if the precise nature of the potential error cannot be determined, if the surgeon suddenly has a very wide through-hole dilation (e.g., indicating a severe lack of concentration due to sudden illness or fatigue), then the general likelihood of an error occurring may be determined to be unacceptably high. For example, a monitored eye behavior parameter associated with a general potential error event is an eye behavior parameter that indicates the surgeon's overall level of concentration, fatigue. An intervention associated with a general potential error event indicates that the potential error event is general (e.g., by displaying an appropriate visual alert using the user interface 115 or sounding a specific audible warning sound), but does not indicate a specific part of the patient's body. In terms of storage and retrieval by the data processing device 108 (e.g., such as Figure 2 ), general potential error events are handled in the same manner as any other potential error events.

[0074] One or more interventional treatments associated with each potential error event are divided into two categories. The first category is a non-patient risk category. For potential error events involved when a patient is in a non-risk condition, this defines one or more interventional treatments applied when the determined error probability exceeds the acceptable error probability. When the value of each physiological parameter in one or more relevant physiological parameters of the patient (as indicated by the surgical data received from the patient monitoring system 112) is within the normal range, the patient is in a non-risk condition. For example, one or more relevant physiological parameters of a given surgical treatment type and their corresponding normal ranges are determined in advance (e.g., based on the knowledge of the accepted medical community) and stored in the storage medium 108. The second category is a patient risk category. For potential error events involved when a patient is in a risky condition, this defines one or more interventional treatments applied when the determined error probability exceeds the acceptable error probability. When the value of each physiological parameter in one or more relevant physiological parameters of the patient is outside the normal range, the patient is in a risky condition. In one embodiment, the relevant physiological parameters indicate the state of one or more patient vital signs.

[0075] Determining an intervention based on whether the patient is in a hazardous condition or a non-hazardous condition helps to safely apply the intervention. For example, when the patient is in a non-hazardous condition, the safest intervention may involve an adjustment in which the surgeon is temporarily prevented from continuing the surgical treatment. Because the patient is in a non-hazardous condition, it is determined to be safer to temporarily suspend treatment (thereby delaying the completion of treatment) to reduce the risk of surgeon error. On the other hand, when the patient is in a hazardous condition, the safest intervention may only involve an alarm (i.e., no adjustment). Thus, the surgeon is prevented from continuing the surgical treatment. Because the patient is in a hazardous condition, it is determined to be safer to allow the surgeon to continue treatment than to delay the completion of treatment (although there is a risk of surgeon error).

[0076] It should be appreciated that even though only two categories of interventions are shown, there may be any number of categories, each associated with a corresponding set of ranges of values ​​for one or more physiological parameters of the patient. Then, when the measured values ​​of one or more physiological parameters of the patient are within the set of ranges associated with that category, the intervention of the specific category is applied (when the determined probability of error exceeds the acceptable probability of error). This allows the most appropriate intervention to be applied based on the patient's physical state, thereby helping to improve patient safety.

[0077] exist Figure 2In the embodiment of FIG. 1 , stage 1 of a heart transplant involves a surgeon making an incision in the flesh between a patient's lung and an artery. Stage 1 is associated with two potential error events. The first event is damage to the lung (acceptable error probability of 0.25), and the second event is damage to the artery (acceptable error probability of 0.25).

[0078] For the first potential error event (lung injury), if the determined likelihood of the surgeon's error (based on the surgeon's eye behavior data) exceeds 0.25, intervention therapy is implemented. If the patient is in a non-critical condition (and therefore avoiding potential surgical errors is more important than avoiding delays in completing stage 1), the intervention therapy includes the communication interface 107 outputting an alarm signal to initiate an audible warning (to warn the surgeon of an unacceptably high likelihood of an error), and further, outputting an adjustment signal to initiate a temporary pause in displaying the video feed used by the surgeon to view the area in which the incision is to be made (thereby preventing the surgeon from continuing to make the incision). On the other hand, if the patient is in a critical condition (and therefore avoiding delays in completing stage 1 is more important than avoiding potential surgical errors), the intervention therapy includes the communication interface 107 outputting only an alarm signal (thereby initiating an audible warning to warn the surgeon of an unacceptably high likelihood of an error by pausing display of the video feed, but not preventing the surgeon from continuing to make the incision).

[0079] For the second potential error event (arterial injury), if the determined likelihood of the surgeon's error (based on the surgeon's eye behavior data) exceeds 0.25, then intervention therapy may also be implemented. If the patient is in a non-critical condition (and therefore avoiding potential surgical errors is more important than avoiding delays in completing stage 1), then the intervention therapy includes the communication interface 107 outputting an alarm signal to initiate an audible warning and a tactile warning (to alert the surgeon of an unacceptably high likelihood of an error), and further, outputting an adjustment signal to initiate a temporary pause in displaying the video feed used by the surgeon to view the area in which the incision is to be made (thereby preventing the surgeon from continuing to make the incision). On the other hand, if the patient is in a critical condition (and therefore avoiding delays in completing stage 1 is more important than avoiding potential surgical errors), then the intervention therapy includes the communication interface 107 outputting only an alarm signal (thereby initiating an audible warning and a tactile warning to alert the surgeon of an unacceptably high likelihood of an error by pausing display of the video feed, but not preventing the surgeon from continuing to make the incision).

[0080] In this case, the second potential error event (arterial injury) is associated with an auditory warning and a tactile warning, while the first potential error event (lung injury) is associated with only an auditory warning. This helps the surgeon identify potential error events more quickly and can also be used to reflect the relative severity of potential error events. For example, a more serious potential error event (e.g., in this case, arterial injury) can be associated with a greater number of warning types (in this case, auditory warnings and tactile warnings) initiated simultaneously, while a less serious potential error event (e.g., in this case, arterial injury) can be associated with a smaller number of warning types (in this case, only auditory warnings) initiated simultaneously.

[0081] exist Figure 2 In the embodiment of the present invention, stage 2 of the heart transplant involves the surgeon attaching blood vessels to the transplanted heart. Stage 2 is associated with two potential error events. The first event is damage to the surrounding organs or muscle groups (the acceptable error probability is 0.35), and the second event is damage to the transplanted heart itself (the acceptable error probability is 0.20).

[0082] For the first potential error event (injury to a peripheral organ or muscle group), if the determined surgeon's error probability (based on the surgeon's eye behavior data) exceeds 0.35, an intervention treatment is implemented. For this potential error event, even if it is not necessary, it is not a sufficiently serious potential error event for the surgeon to pause (even if the patient is in a non-dangerous condition). Thus, in this case, the intervention treatment is the same for patients in non-dangerous conditions and patients in dangerous conditions. That is, the intervention treatment includes the communication interface 107 outputting an alarm signal to initiate the augmented reality (AR) visualization superimposed on the surgical image of the portion of the peripheral organ or muscle group that is prone to injury (in this case, this is the area of ​​interest) (displayed using the user interface 115). However, an adjustment signal is not output through the communication interface 107, and thus, the surgeon is not prevented from continuing treatment (i.e., the surgeon is allowed to make a decision about whether to continue treatment or take a break). The portion of the peripheral organ or muscle group indicated by the superimposed AR visualization can be detected based on the surgeon's eye behavior data. In one embodiment, it is a portion where the surgeon has a residence time and / or fixed frequency below a predetermined residence time and / or fixed frequency value. The AR image visualization alerts the surgeon to an unacceptably high probability that he or she has made an error and informs him or her of the nature of the potential error (ie, damage to a portion of the identified surrounding organ or muscle group).

[0083] For the second potential error event (transplant heart damage), if the determined surgeon's error probability (based on the surgeon's eye behavior data) exceeds 0.20, an intervention treatment is implemented. For this potential error event, it is a sufficiently serious potential error event that the surgeon should suspend (even if the patient is in a dangerous condition). Thus, in this case, the intervention treatment is the same for patients in non-dangerous conditions and patients in dangerous conditions. That is, the intervention treatment includes the communication interface 107 outputting an alarm signal to initiate an augmented reality (AR) visualization superimposed on an image of a portion of the transplant heart that is prone to damage (in this case, this is the area of ​​interest) (displayed using the user interface 115) and initiating an auditory warning. Further, the intervention treatment includes the communication interface 107 outputting an adjustment signal to suspend the operation of the surgical robot (e.g., surgical robot 109) currently used by the surgeon to complete the treatment (thereby temporarily preventing the surgeon from performing the treatment). Again, the portion of the transplant heart indicated by the superimposed AR visualization can be detected based on the surgeon's eye behavior data. In one embodiment, it is a portion where the surgeon has a residence time and / or fixed frequency value below a predetermined residence time and / or fixed frequency value. The AR image visualization alerts the surgeon to an unacceptably high probability of error and informs him of the nature of the potential error (ie, damage to the identified portion of the transplanted heart).

[0084] exist Figure 2In an embodiment, stage 3 of a heart transplant involves the surgeon using a surgical robot 109 to make a further smaller and more precise incision. For example, the surgeon uses the user interface 115 to control the robot to make the incision (the control signal is sent from the user interface 115 to the surgical robot 109 via the data processing device 104). As the incision is made, a predicted path for the incision is determined based on the surgeon's eye behavior data. For example, it can be determined that the predicted path is a straight line defined between the current position of a cutting tool (e.g., a scalpel) used to make the incision and the surgeon's current gaze position. The predicted path of the incision is shown superimposed on a displayed image of a portion of the patient's body, and the incision is made along the predicted path. Stage 3 is associated with only one potential error event. This causes damage to a sub-surface blood vessel located on the path of the incision. This may occur if the surgeon (e.g., due to a loss of concentration) does not notice when the sub-surface blood vessel is positioned along the predicted path of the incision. The acceptable error probability of the potential error event is 0.25. Thus, for a potential error event at stage 3, if the determined surgeon's error probability (based on the surgeon's eye behavior data) exceeds 0.25, then intervention therapy is implemented. The determined error probability may be determined based on the distance between the cutting tool used to make the incision and any sub-surface vessel located along the predicted incision path (larger distances are associated with lower determined error probability and smaller distances are associated with higher determined error probability).

[0085] If the patient is in a non-dangerous condition, the interventional treatment includes that the communication interface 107 outputs an adjustment signal and controls the robot to override the control instructions from the surgeon and controls the cutting tool to deviate from the predicted incision path to avoid sub-surface blood vessels. The override is temporary so that after avoiding potential error events, control returns to the surgeon. Thus, the surgeon still controls the robot (keeping the surgeon in control of the treatment and thereby allowing the patient to benefit from the surgeon's personal experience and ability to quickly interpret a variety of types of information), except for situations where the surgeon's likelihood of causing a potential error event becomes unacceptably high (in this case, the robot temporarily overrides the surgeon's control to avoid errors). The interventional treatment may also include that the communication interface 107 outputs an alarm signal and initiates a tactile warning and initiates an augmented reality (AR) visualization superimposed on an image of a sub-surface blood vessel that may be injured (in this case, this is an area of ​​interest) (displayed using the user interface 115). In this case, the output of the alarm signal is optional. For example, whether to output an alarm signal may depend on the surgeon's preconfigured preferences or on the extent to which the robot must override the surgeon's control. For example, in the latter case, if the robot must override the surgeon's control in less than a predetermined threshold time (suggesting a relatively small potential deviation from the predicted incision path), no alarm signal is output (because the potential deviation is only small, it is more beneficial for the surgeon not to be distracted by the AR visualization and haptic warning than to be informed of the increased possibility of an error). On the other hand, if the robot must override the surgeon's control in more than a predetermined threshold time (suggesting a relatively large potential deviation from the predicted incision path), an alarm signal is output (because the potential deviation is large, it is more beneficial for the surgeon to be informed of the increased possibility of an error than to be distracted by the AR visualization and haptic warning).

[0086] If the patient is in a critical condition, the intervention therapy does not include the communication interface 107 outputting an adjustment signal to control the robot overriding the control instructions from the surgeon. This allows the surgeon to always maintain control of the robot when the patient is in a critical condition (when the surgeon's personal experience and ability to interpret a wide variety of types of information will be most beneficial). However, the intervention therapy does include the communication interface 107 outputting an alarm signal to initiate a tactile warning and to initiate the superposition of an augmented reality (AR) visualization on an image (displayed using the user interface 115) of a sub-surface vessel that may be injured (in this case, this is the area of ​​interest). In the event that the patient is in a critical condition, an alarm signal is always output through the communication interface 107 to allow the surgeon to be informed of the increased possibility that something is wrong (thereby providing the surgeon with the most possible information when making a decision about the treatment of a critical patient).

[0087] thus, Figure 2 The embodiments in the present invention illustrate how different acceptable error probabilities are defined and different types of interventions are implemented based on the type of surgery, the current stage of the surgery, potential error events associated with the type of surgery or the current stage of the surgery, and the patient's status (e.g., critical or non-critical). It should be appreciated that for the sake of clarity, Figure 2 This is a very simple embodiment, and in reality, the database stored in the storage medium 108 may relate to a larger number of surgical types (not just a single surgical type "heart transplant"), surgical stages, potential error events, acceptable error probability, and intervention types. However, the principle of operation remains as described (except that it occurs with a larger data set).

[0088] For various surgical types or surgical stages determined by the manufacturer, the manufacturer of the surgical support system 100 can preconfigure the information in the database with the most appropriate values ​​for each acceptable probability of error and the most appropriate intervention treatment. Specifically, for each potential error event, the manufacturer can determine in advance each acceptable probability of error value by evaluating the potential results of the error for the potential results of intervention in a less serious error. For example, this information can be determined by reviewing historical surgical data. Alternatively or in addition, for various surgical types or surgical stages, the surgeon can customize the acceptable probability of error values ​​and the most appropriate intervention treatment according to his personal preferences (e.g., using the user interface 115 to access and edit a database stored in the storage medium 108).

[0089] As mentioned previously, in an embodiment, the surgeon may select the type of surgical procedure (e.g., "heart transplant") to be performed prior to commencing the surgical procedure using a suitable GUI menu system, etc. During the surgical procedure, the data processing device 104 may be configured to detect the current stage of the surgical procedure (e.g., "stage 1," "stage 2," or "stage 3") in one or more different ways (thereby being able to look up appropriate potential error events for that stage and the associated acceptable error probability and type of intervention).

[0090] In one embodiment, each time treatment moves to a new phase, a user (e.g., a surgeon or another member of the surgical team) can manually update the system. This can be done, for example, via user interface 115 (e.g., by selecting the current phase from a list of phases of the current treatment displayed on a GUI menu system, etc.) or via audio monitoring system 114 (e.g., by speaking a predetermined command, such as one or more words that uniquely identify the desired phase of treatment, picked up by audio monitoring system 113 and discernible by data processing device 104, etc.).

[0091] In another embodiment, the data processing device 104 is configured to automatically detect the current stage of treatment. In one embodiment, this operation can be accomplished using suitable machine vision image comparison techniques (such as methods known in the art and therefore not discussed in detail here). At specified intervals during the treatment, images are captured by the external and / or internal camera systems 110, 111 and one or more features of the captured images are compared with one or more corresponding features of each image in a previous instance of surgical treatment. Each image in the previous instance of surgical treatment is an image known to be captured at a specific stage of the previous instance of surgical treatment. The specific stage of the image of the previous instance of surgical treatment that has the closest match to the captured image is determined to be the current stage of treatment. In an embodiment, a machine learning classification technique is used to determine one or more features of the captured image and each image in the previous instance of treatment compared thereto. That is, each image in the previous instance of treatment is manually classified as belonging to a specific stage of treatment and a machine learning algorithm automatically determines one or more features that distinguish the image from images belonging to other stages of treatment. Then, one or more such features are analyzed in a newly captured image (whose classification is unknown) to determine the possible category of the newly captured image.

[0092] Thus, for example, in one or more previous instances of cardiac transplantation therapy, Figure 2 In a cardiac transplant surgery treatment in which at least one image of each of Stage 1, Stage 2, and Stage 3 has been captured and manually classified by a human user as "Stage 1," "Stage 2," or "Stage 3." A machine learning algorithm automatically determines one or more features of each of these images that are indicative of the specific stage associated with the image. That is, given an image with a manually applied classification, the machine learning algorithm will automatically determine one or more features that distinguish a "Stage 1" image from a "Stage 2" or "Stage 3" image, one or more features that distinguish a "Stage 2" image from a "Stage 1" or "Stage 3" image, and one or more features that distinguish a "Stage 3" image from a "Stage 1" or "Stage 2" image. Figure 2 During the course of treatment, each image captured by the external or internal camera system 110, 111 is analyzed to determine whether the image is most likely "stage 1", "stage 2", or "stage 3" based on one or more features of previously captured "stage 1", "stage 2", and "stage 3" images. Then, it is determined that the stage most likely associated with the image is the current stage of treatment.

[0093] Various machine learning algorithms are known in the art and therefore will not be discussed in detail here. It will be appreciated that the greater the number of images of various stages of previous instances of treatment and the greater the number of previous instances of treatment from which these images were generated, the more reliable the determination of the current stage of the current treatment. For example, previous images may also be captured from different angles and / or under different lighting conditions to help provide a more reliable determination of the current stage of treatment (because the viewing angle and / or lighting conditions of the captured images of the current treatment may change for different instances of treatment). For example, data indicating one or more features of images of previous instances of treatment for analysis of newly captured images of the current treatment is stored in the storage medium 108.

[0094] In an embodiment, the surgeon (or another member of the surgical team) can manually confirm the automatic determination of the current stage of surgical treatment during the treatment process. This helps reduce the risk of surgical errors due to incorrect determination of the current stage of treatment (e.g., an incorrect determination of "stage 3" when in fact "stage 2" was completed means that incorrect potential error events, acceptable error probability, and / or intervention treatment may be applied). In one embodiment, the manual confirmation includes: entering a confirmation command via the user interface 115 and / or the audio monitoring system 112. For example, the user interface 115 displays the message "Automatic Treatment Stage Detection = X - Press OK or say CONFIRM to confirm" (wherein, for Figure 2 , X is the name of the stage, for example, "Stage 1", "Stage 2", or "Stage 3"). If the user presses "OK" (for example, "OK" is a virtual button displayed on the touch screen of the user interface 115) or says "Confirm" by voice (the sound is picked up by the audio monitoring system 113), the data processing device 104 knows that it has correctly determined the current surgical stage. On the other hand, if the user does not complete the confirmation action within a predetermined time limit (for example, 5 seconds or 10 seconds), the data processing device 104 cannot rely on automatically determining the current stage of the treatment, but waits for further input from the user to manually select the current stage of the treatment.

[0095] The concept of automatically determining the current stage of a treatment based on a comparison between one or more features of a newly captured image of the current treatment and one or more corresponding features of one or more images of a previous instance of the treatment can be applied to other types of surgical data collected during the current and previous instances of the surgical treatment. For example, this can be applied to: (i) surgical data generated by a surgical robot (e.g., movement of an articulated arm from an actuator); (ii) surgical data generated by a patient monitoring system; (iii) audio data collected by an audio monitoring system 113 during the current and previous instances of the surgical treatment (including verbal communications between members of the surgical team, hearing robotic arms, etc.); and / or (iv) surgical data generated by a surgeon monitoring system 114. In addition, for example, control data for an inflator (not shown) or an energy device can also be used to identify the current stage of the surgical treatment.

[0096] Different types of surgical data may be used to independently identify specific events occurring during a surgical procedure that are indicative of a specific stage of the surgical procedure being performed. In one embodiment, the tools used by the surgeon at any given time during the procedure may be detected using image recognition of images captured by external and / or internal camera systems 110, 110 on the tools and a monitoring device (e.g., an accelerometer forming part of the surgeon monitoring system 114), as opposed to being fixed to a surface, the monitoring device outputting a signal as surgical data to the communication interface 107 when the tool in question is picked up. The specific combination of tools used (e.g., a database stored in the storage medium 108) may be mapped to specific stages of the procedure, thereby allowing monitoring of the stages occurring.

[0097] In embodiments, in addition to the eye behavior data, the surgeon's movements and / or body language (e.g., detected using image recognition of images captured by the external and / or internal camera systems 110, 111 and / or by a surgeon monitoring system 114 that includes a monitoring device on the surgeon (e.g., an accelerometer) that outputs signals as surgical data indicative of the surgeon's movements) may also be used to help determine the surgeon's likelihood of error. For example, if surgical signals output by the monitoring device indicate that the surgeon has deviated from expected movements (e.g., as indicated by the tools used by the surgeon) or from particular body language associated with safe surgical practice, an appropriate scaling factor (e.g., larger for larger deviations from safe surgical practice and smaller for smaller deviations from safe surgical practice) is applied to the determined likelihood of error to account for this.

[0098] In an embodiment, the data processing device 104 is operable to compare the obtained eye behavior data of the surgeon with predetermined eye behavior data associated with a predetermined likelihood that the surgeon made an error in the surgical treatment. The data processing device 104 is operable to determine the likelihood that the surgeon made an error in the surgical treatment using the predetermined likelihood associated with the predetermined eye behavior data and the comparison result between the obtained eye behavior data of the surgeon and the predetermined eye behavior data.

[0099] In an embodiment, the value of each of a plurality of eye behavior parameters is monitored. For example, one or more of the eye behavior parameters of dwell time on an area of ​​interest, fixation frequency on an area of ​​interest, rate and / or distance of glances, whether an area of ​​interest is ignored, and pupil dilation of the surgeon are monitored. In an embodiment, the area of ​​interest is a portion of a field of view of a predetermined size and shape of the surgeon and is located at a predetermined position in the surgeon's field of view. In another embodiment, the area of ​​interest is detected based on a suitable image recognition technique.

[0100] The value of each eye behavior parameter (included in the eye behavior data collected by the eye behavior monitoring device 101) is compared with one or more predetermined values ​​of the eye behavior parameter to determine the error probability associated with the parameter (parameter error probability). For example, the predetermined eye behavior parameter value is a threshold value, and the parameter error probability associated with a specific eye behavior parameter depends on the relationship between the monitored value of the eye behavior parameter and one or more threshold values ​​of the eye behavior parameter.

[0101] In one embodiment, the eye behavior parameter has a single threshold. For example, there may be a single threshold for the dwell time on a specific region of interest (e.g., 0.5 seconds). If the dwell time on the monitored region of interest is less than the threshold (e.g., less than 0.5 seconds), the parameter error probability is set to 1. On the other hand, if the dwell time on the monitored region of interest is greater than the threshold (e.g., greater than 0.5 seconds), the parameter error probability is set to 0.

[0102] In another embodiment, the ocular behavior parameter has multiple thresholds. This allows to limit the possibility of more number of parameter errors and the possibility of parameter errors depends on the position of the monitoring parameter value relative to multiple thresholds. For example, the dwell time on a specific area of ​​interest can have three thresholds of (e.g., 0.25 seconds, 0.5 seconds, and 0.75 seconds) monitoring parameter values ​​with continuously increasing sizes. If the monitored dwell time is less than the first (lowest) threshold (e.g., less than 0.25 seconds), it is determined that the parameter error possibility is the highest possible value (e.g., 1). If the monitored dwell time is between the first threshold and the second (next lowest) threshold (e.g., between 0.25 seconds and 0.5 seconds), it is determined that the parameter error possibility is a lower value (e.g., 0.75). If the monitored parameter value is between the second threshold and the third (highest) threshold (e.g., between 0.5 seconds and 0.75 seconds), it is determined that the parameter error possibility is still a lower value (e.g., 0.25). If the monitored parameter value is greater than the third threshold (e.g., greater than 0.75 seconds), it is determined that the parameter error possibility is the lowest possible value (e.g., 0).

[0103] It should be appreciated that while the use of thresholds is exemplified for the "dwell time on a region of interest" eye behavior parameter, the use of thresholds may also be applied to any other eye behavior parameter in the form of a numerical value related to the surgeon's likelihood of error (e.g., fixation frequency, saccade position and / or rate, and pupil dilation on a region of interest). For example, based on the surgeon's historical eye behavior data recorded during previous surgical treatments and data indicating errors that occurred during these previous surgical treatments (including the type of surgery and / or stage during which each error occurred), for each potential error event associated with each type of surgery and / or stage, specific thresholds for each monitored eye behavior parameter and the likelihood of parameter error defined relative to the threshold are selected in advance. For example, a suitable machine learning algorithm may be applied to the data set to determine the threshold and the associated likelihood of parameter error. In an embodiment, the likelihood of parameter error for a parameter of whether to ignore a region of interest is 0 (when the region of interest is not ignored) or 1 (when the region of interest is ignored).

[0104] For a given potential error event (defined for a specific operation type and / or stage), the parameter error probability of each monitored eye behavior parameter is combined in an appropriate manner to obtain an overall determined error probability (compared with the acceptable error probability of the potential error event). In one embodiment, the determined error probability is the weighted sum of the parameter error probability determined for each monitored eye behavior parameter. For example, if five eye behavior parameters are monitored (e.g., the dwell time on the region of interest, the fixed frequency on the region of interest, the rate and / or distance of the scan, whether to ignore the region of interest, and the pupil dilation of the surgeon), the parameter error probability determined for each of these parameters is added together by appropriate weights to obtain an overall determined error probability property. In an embodiment, it is determined that each parameter error probability is between 0 and 1 (0 indicates the lowest possible error probability and 1 indicates the highest possible error probability). The weight is selected so that the overall determined error probability is also between 0 and 1. For example, if the five monitoring parameters are weighted equally, the weight of each parameter will be 0.2 to ensure that the overall determined error probability is between 0 and 1. In another embodiment, if two specific monitoring parameters are more indicative of an error than three other monitoring parameters (e.g., based on historical eye behavior data), the specific two monitoring parameters may be weighted higher than the other three monitoring parameters (e.g., 0.35 for each of the two or more indicative parameters and 0.1 for each of the three other parameters). The weights of the individual eye behavior parameters may be adjusted based on the relevance of that eye behavior parameter to the potential error event involved (e.g., again based on historical eye behavior data).

[0105] Thus, in an embodiment, for each ocular behavior parameter monitored for a specific potential error event, a parameter error probability is determined based on the monitored value of the parameter and one or more thresholds (if applicable) compared to the monitored parameter value. An overall determined error probability is then determined by combining the parameter error probabilities of all monitored parameters (e.g., as a weighted sum). The overall determined error probability is then compared to an acceptable error probability defined for the potential error event to determine whether an intervention treatment should be performed.

[0106] In one embodiment, eye behavior parameters related to pupil dilation (e.g., parameters of pupil dilation and glance rate and / or distance) may be monitored to determine the surgeon's overall level of concentration, fatigue, etc. This eye behavior parameter may be referred to as a "concentration" or "fatigue" parameter. In another embodiment, eye behavior parameters related to eye tracking (e.g., parameters of dwell time on an area of ​​interest, fixation frequency on an area of ​​interest, and whether an area of ​​interest is ignored) may be monitored to determine the surgeon's experience or skill level. This eye behavior parameter may be referred to as an "experience" or "skill level" parameter.

[0107] For each potential error event (such as Figure 2 , for each surgical type and / or stage defined), when calculating the overall determined error probability, the specific eye behavior parameters monitored, any parameter thresholds, the parameter error probability associated with the parameter thresholds, and the weights applied to the parameter error probability are determined in advance. This allows the determined error probability of each potential error event to be calculated in a more appropriate manner. The surgeon determines this information about each potential error event in advance and stores it in the storage medium 108 as part of the database (together with the acceptable error probability and intervention treatment for the potential error event).

[0108] As mentioned previously, for example, by collecting this information from previous instances of surgical treatment using known results, the specific ocular behavior parameters being monitored, any parameter thresholds, the likelihood of parameter errors associated with the parameter thresholds, and weights can be determined. For example, a suitable machine learning algorithm can be used on data of ocular behavior parameter values ​​measured during previous instances of surgical treatment and data indicating measures of success for these previous instances of surgical treatment (e.g., based on patient medical records). Typically, this data can be referred to as historical data.

[0109] This allows the machine learning algorithm to learn the most relevant ocular behavior parameters for each potential error event for each surgical treatment type and / or stage, thereby allowing the determination of the ocular behavior parameters monitored for that potential error event and the determination of the relative weights of the various ocular behavior parameters used to calculate the overall determined error probability. It also allows (if relevant) the determination of the values ​​of these ocular behavior parameters for which the probability of a potential error event occurring is increased (or decreased) by a specific amount, thereby allowing the definition of thresholds and parameter error probabilities associated with these thresholds.

[0110] For example, based on the previous example of heart transplant treatment, it was determined that the occurrence of the potential error event "lung injury" was strongly correlated with a dwell time of less than 0.5 seconds on the patient's lungs and a pupil dilation of more than 50%. At this time, it was found that other eye behavior parameters (such as the fixation frequency on the lungs, etc.) were less strongly correlated with the occurrence of the potential error event "lung injury". In this case, when determining the overall error probability or potential error event, the eye behavior parameters of the dwell time and pupil dilation on the lungs have a higher weight than other eye behavior parameters (such as the fixation frequency on the lungs, etc.). Further, it was determined that a dwell time threshold of 0.5 seconds was appropriate and a pupil dilation threshold of 50% was most appropriate. Therefore, by comparing the dwell time and pupil dilation values ​​monitored during the current heart transplant treatment with the thresholds of 0.5 seconds and 50%, respectively, the parameter error probability of each of these parameters during the current heart transplant treatment is determined.

[0111] Historical data can also be used to determine the parameter error probability for a given parameter when the monitored value of that parameter is compared to an associated threshold value. For example, historical data may indicate that for a dwell time on the lungs of less than 0.5 seconds, the probability of an error is 0.5, while for a dwell time greater than 0.5 seconds, the probability of an error is 0.15. Therefore, these values ​​are associated with the determined threshold value of 0.5 seconds. Thus, in current heart transplant treatments, if the monitored dwell time on the lungs is less than 0.5 seconds, the parameter error probability for the potential error event "lung injury" is determined to be 0.5. On the other hand, if the monitored dwell time on the lungs is greater than 0.5 seconds, the parameter error probability for the potential error event "lung injury" is determined to be 0.15. The determined parameter error probability is then combined with the determined parameter error probability of each other monitored ocular behavior parameter (e.g., via a weighted sum) and compared with the acceptable error probability for the potential error event "lung injury." Figure 2 In an embodiment, if the determined error probability exceeds an acceptable error probability of 0.25, intervention treatment is performed.

[0112] For each potential error event, a region of interest is determined in advance (the values ​​of its eye behavior parameters such as dwell time and fixation frequency are calculated) so that if the surgeon's attention (i.e., the position in space that the surgeon is looking at) is on the region of interest, the event can be detected. For different potential error events, the region of interest may be different. For example, for the potential error event "lung injury", the region of interest is the patient's lungs. On the other hand, for the potential error event "arterial injury", the region of interest is the patient's artery.

[0113] Objects within the surgeon's field of view are mapped in advance to corresponding positions within the surgeon's field of view to be able to determine what object the surgeon is viewing at a given time. For example, if the surgeon is viewing a live video image of the patient (e.g., an image captured by an external or internal camera system 110, 111 is sent to the communication interface 107, and then, sent from the communication interface 107 to be displayed on the screen of the user interface 115), the field of view of the live video image is determined to be the surgeon's field of view and the area of ​​the screen on which the video being viewed by the surgeon is displayed is determined to be the surgeon's gaze position. The data processing device 104 uses appropriate image recognition techniques (as known in the art) to detect relevant objects in the image (including any areas of interest) and the position of each image on the screen. When the surgeon's gaze position on the screen (as determined by the eye behavior monitoring device 101) overlaps with the position of the detected object on the screen, it is determined that the surgeon is viewing the detected object.

[0114] For example, based on a previously completed machine learning process in which an image (e.g., lungs, arteries, etc.) is classified using a suitable machine learning algorithm, an object within the surgeon's field of view (specifically, an area of ​​interest, such as the patient's lungs or arteries) can be identified by the data processing device 104. Thus, objects in newly captured images of the patient can be detected based on the previous machine learning classification. Once the object is detected, it is determined (using the eye behavior data) when the surgeon's gaze position overlaps with the position of the detected object. When there is an overlap, it is determined that the surgeon is viewing the detected object. When there is no overlap, it is determined that the surgeon is not viewing the detected object. In this way, eye behavior parameters such as dwell time and fixation frequency on the detected object (e.g., area of ​​interest) can be determined.

[0115] In embodiments, the eye behavior monitoring device 101 can be combined with a user interface 115. For example, the user interface 115 can include a display (e.g., in the form of a touch screen or HMD) on which images of the surgical procedure (captured by the external or internal camera system 110, 111) are displayed. The eye behavior monitoring device 101 (including another camera for capturing video images of the surgeon's eyes to obtain eye tracking and pupil monitoring data) is mounted in an appropriate position relative to the display to monitor the surgeon's eyes as he views the displayed images.

[0116] Figure 3An exemplary embodiment is shown in which a surgeon 300 is wearing an HMD device 301. The surgeon wears the HMD device 301 while performing surgery to view images inside the patient's body (as captured by the internal camera system 111) while the surgeon controls the arms 109A and 109B of the surgical robot 109 to perform surgical treatment on the patient. An image 305 displayed to the surgeon by the HMD device 301 is shown. The HMD device 301 includes, in addition to a display that displays the image 305. The HMD device 301 also includes an eye behavior monitoring device 101. This allows the position of the displayed image that the surgeon is viewing at any given time to be determined.

[0117] The image 305 is divided into a plurality of portions 303. Each portion 303 represents a portion of the image 305 that the surgeon is viewing. In the present embodiment, a detection has been performed on a region 304. Region 304 is composed of a plurality of image portions 303, and it has been detected that the surgeon's eye movements relative to these portions indicate that the probability of error in a potential error event determined has exceeded an acceptable probability of error. Therefore, an interventional treatment (e.g., such as an AR overlay) is initiated in which region 304 is highlighted in the image 305. This alerts the surgeon that the risk of error in which his eye movements indicate that a region of the patient's body appears within region 304 is unacceptably high. The interventional treatment may be accompanied by one or more other interventional treatments (such as, for example, an auditory warning of a tactile feed, etc.).

[0118] For example, if the surgeon's dwell time on the area is unacceptably low or if the frequency of fixation on the area is unacceptably low, then area 304 is highlighted. In an embodiment, area 304 is associated with a potential error event, and therefore, the acceptable error probability to which the determined error probability is compared is the acceptable error probability for the potential error event. Area 304 may be associated with a potential error event even if it contains a region of interest associated with the potential error event. For example, if the treatment is stage 1 of a heart transplant and the patient's lungs are detected within area 304 (as a region of interest), then the potential error event associated with area 304 is determined to be "lung injury." Based on the surgeon's eye behavior data, if the determined error probability exceeds the acceptable error probability of "lung injury" (e.g., based on Figure 2 , is 0.25), then an intervention associated with “lung injury” is initiated. On the other hand, if the patient’s artery is contained in region 304, then the potential error event associated with region 304 is determined to be “arterial injury”. Based on the surgeon’s eye behavior data, if the determined error probability exceeds the acceptable error probability for “lung injury” (e.g., based on Figure 2 , is 0.25), then an intervention treatment associated with “lung injury” is initiated.

[0119] Figure 4A and Figure 4B A more detailed demonstration of an exemplary intervention during Stage 1 of a heart transplant is shown. For the sake of illustration, only a single ocular behavior parameter of the surgeon's dwell time is considered and used to determine the error probability of each potential error event (in this case, "lung injury" and "arterial injury"). However, it should be recognized that in reality, multiple different ocular behavior parameters are monitored simultaneously and used to determine the error probability of each potential error event.

[0120] In this case, the surgeon plans to make an incision at a location 402 between the patient's lung 400 and an artery 404. The surgeon views the treatment via a display of the user interface 115 showing an image 401 of the lung 400, the artery 404, and the incision location 402. The surgeon's eyes are tracked to determine the surgeon's position on different areas of the image 401 (e.g., similar to Figure 3 4. The portion 303 in FIG. 4 shows the dwell time of the image 401 (on each region into which the image 401 is divided). Region 403 indicates a region with an acceptable dwell time (e.g., a dwell time above a predetermined dwell time threshold). Region 403 includes the lungs 400, but not the arteries 404. Therefore, the determined error probability is below the acceptable error probability of the potential error event "lung injury". However, the determined error probability is above the acceptable error probability of the potential error event "arterial injury". In this case, for ease of explanation, it is assumed that the predetermined dwell time thresholds of the respective potential error events "lung injury" and "arterial injury" are the same.

[0121] In response thereto, an intervention is completed. In this case, the intervention includes relaying an augmented reality projection 405 of the artery 404 to alert the surgeon that the likelihood of damaging the artery 404 is unacceptably high. Figure 4B This is shown in . Thus, the surgeon is made aware of potential errors before they occur and can adjust his practice to ensure that he provides adequate dwell time in the artery as well as the lung, thus addressing the potential errors and reducing their likelihood of occurrence.

[0122] Figure 5A and Figure 5BAn embodiment of stage 3 of a "heart transplant" treatment is shown in which sub-surface vessel damage is avoided by an interventional treatment involving automatic adjustment of the incision path. In this case, the surgeon makes an incision 501 along a pre-planned predicted incision path 502A. The surgeon views the surgical site via the displayed image 500. The surgeon performs this operation using a scalpel 503 held by a surgical robot 109. The surgeon controls the robot to move the scalpel 503 to make the incision. It has been detected that the predicted path 502A includes a sub-surface vessel 504. The scalpel is sufficiently close to the sub-surface vessel (i.e., less than a predetermined distance) so that the data processing device 104 determines that the surgeon did not notice the sub-surface vessel and therefore the sub-surface vessel may have suddenly become serious. In response to this, as Figure 5B , the predicted incision path 502A is automatically adjusted to provide a new incision path 502B that avoids the sub-surface blood vessels 504. For example, depending on the degree to which the predicted incision path 502A is adjusted to achieve the surgeon's new incision path 502B and / or preconfigured preferences, the intervention of adjusting the predicted path may be accompanied by other interventions (e.g., Figure 2 AR error visualization and tactile warnings).

[0123] It should be recognized that the interventions discussed are only embodiments and other interventions are contemplated. For example, in addition to identifying areas with unacceptably high probability of determining errors, the AR projection can also provide additional information to the surgeon to help suggest why the intervention was initiated and what can be done to return the determined probability of error to an acceptable level. For example, if the intervention includes: displaying an AR projection, the AR projection can also indicate to the surgeon (e.g., using text or a predetermined color coding scheme) why the AR projection has been shown in addition to indicating a portion of the image associated with the potential error event. For example, if the AR projection is displayed due to insufficient dwell time, a message "Insufficient dwell time" is provided as part of the AR projection. Alternatively or in addition, if the AR projection is displayed due to insufficient fixed frequency, a message "Insufficient fixed frequency" is provided as part of the AR projection. For example, other interventions include: issuing an alarm to another person (e.g., another surgeon) or an AI (artificial intelligence) assistant to whom the surgeon seeks help or even passing control to another person (e.g., a more experienced surgeon) who manipulates the surgical treatment until the stage of finding that the determined probability of error is unacceptable is completed. For example, in surgical training, the system may be useful.

[0124] In embodiments, in addition to providing interventional treatment based on the surgeon's ocular behavior data, the surgeon's ocular behavior data can be used to determine the surgeon's current competence and / or confidence prior to initiating the surgical treatment (e.g., using a known relationship between the surgeon's ocular behavior and competence and / or confidence.) The order in which the stages of the surgical treatment are to be completed is then determined based on the surgeon's determined competence and / or confidence (for treatments where this may occur).

[0125] For example, before starting treatment, images of various stages in previous instances of surgical treatment are displayed to the surgeon from various angles and the surgeon's eye behavior data is collected while the surgeon is viewing these images. In response to the analysis of the eye behavior data, the data processor 104 determines the relevant surgical visual field area that may be associated with a higher probability of error. If possible (depending on the type of treatment), the surgeon is instructed to complete the treatment stage involving the area associated with the lower probability of error that is completed first. This helps to provide a successful start for the treatment. This helps to establish the surgeon's workflow and improve its confidence in the later and more difficult stages of the treatment (associated with a higher probability of error). For example, the data processing device 104 determines the order of the stages of the treatment (for treatments where this may occur) by determining the probability of error of each stage based on, for example, the surgeon's eye behavior data and sorting the stages in order from the lowest probability of error to the highest probability of error. Then, the stages are completed in the order determined by the sorting (so that the stage with a higher probability of error that is successful is after the stage with the lowest probability of error that occurs first).

[0126] In an embodiment, when the surgeon completes treatment by viewing an image displayed using the display of the user interface 115 and the determined error probability exceeds the acceptable error probability for a given potential error event, the data processing device may control the user interface 115 to display a close-up (i.e., zoomed-in) version of the area of ​​the displayed image associated with the unacceptably high error probability (e.g., the portion of image 401 including the Figure 4A and Figure 4B The artery 404 in the image 500 or the portion thereof includes Figure 5A and Figure 5B When the magnified image is displayed, the eye behavior data of the surgeon viewing the magnified image is collected. This enables the determination of sub-regions within the magnified image that are associated with an unacceptably high probability of error.

[0127] When the displayed visual field is wide enough, this allows determining the specific cause of an unacceptably high probability of error, so that the unacceptably high probability of error has multiple potential causes. For example, when displaying an image 401 including lungs 400 and arteries 401, it is difficult to determine whether the determined probability of error is associated with the lungs or arteries from specific eye behavior parameters (e.g., high glance distance and / or rate). By displaying (a) a region of the image 401 containing the lungs 400 and (b) a region of the image 401 containing the arteries 404, and performing further eye behavior analysis of the surgeon on each of the continuously displayed enlarged images when viewing the enlarged images, the source of uncertainty can be narrowed down to the lungs 400 or arteries 404 (e.g., if a high glance distance and / or rate is maintained when displaying an enlarged image of the lungs 400, rather than when displaying an enlarged image of the arteries 404, it is determined that the unacceptably high probability of error associated with the high glance distance and / or rate is related to the lungs 400, rather than the arteries 404). This can provide the surgeon with a wider surgical field of view (enabling the surgeon to monitor a greater number of events simultaneously) while allowing more accurate determination of the source of any unacceptably high probability of error.

[0128] In an embodiment, when an area of ​​a displayed image is associated with an unacceptably high probability of error based on the surgeon's eye behavior data, an image (or at least an area of ​​the image associated with a high probability of error) can be displayed with different visual features (e.g., different applications of error colors, lines, shading, etc.) in an attempt to provide the surgeon with a clearer image (i.e., an image that is easier for the surgeon to understand what is happening). A plurality of different predetermined visual features (image effects) can be applied continuously to a displayed image (or an associated image area) of a user interface 115 (controlled by the data processing device 104). After applying various image effects, the surgeon's eye behavior data is collected to determine whether the probability of error is reduced due to the applied effect. The continuous display of different effects and collections of associated eye behavior data is another embodiment of an interventional treatment (specifically, it represents an adjustment of the image displayed on the screen of the user interface 115 that is dependent on the surgeon completing the surgical treatment). This allows various predetermined effects to be applied continuously to the image (or associated image area) until the probability of error is reduced to an acceptable level. If no effect results in a sufficient reduction in the probability of error, another interventional treatment can be initiated. Applying a variety of different image effects to the images makes the images easier for the surgeon to understand and thus reduces the error potential associated with the images, allowing unacceptably high error potential to be corrected (e.g., pausing display of the image or pausing operation of the surgical robot 109) potentially before applying a more invasive intervention. This helps reduce interruptions to the surgeon's workflow.

[0129] Figure 6 A method according to an embodiment of the present technology is shown. The method starts at step 600. In step 601, the type of surgical treatment (e.g., heart transplant) is determined. For example (as previously described), this type is manually selected by the surgeon using the user interface 115 before starting the treatment. In step 602, during the treatment, the current stage of the treatment is determined (e.g., stage 1, stage 2, or stage 3 of heart transplant). For example (as previously described), this operation is automatically performed by the data processing device 104 based on image recognition. In step 603, the error probability of the first potential error event among the potential error events associated with the determined stage of the surgical treatment (e.g., for stage 1 of heart transplant, it is "lung injury") is determined. In step 604, it is determined whether the error probability of the determined potential error event exceeds the acceptable error probability of the potential error event.

[0130] At step 604, if the determined error probability does not exceed the acceptable error probability, the method proceeds to step 609, where it is determined whether the current surgical stage is over. If the current surgical stage is not over, the method proceeds to step 605, where the next potential error event associated with the current stage of the surgical treatment is selected (e.g., for stage 1 of a heart transplant, "arterial injury"). The method then returns to step 603, where the error probability of the selected next potential error event is determined. If the current surgical stage is over, the method proceeds to step 610, where it is determined whether the treatment is over. If the treatment is not over, the method returns to step 602 so that the next surgical stage is determined (e.g., if stage 1 is over, it is stage 2 of a heart transplant, or if stage 2 is over, it is stage 3 of a heart transplant). Then, at step 603, the error probability of the first potential error event of the next surgical stage is determined. At step 611, if the treatment is over, the method ends.

[0131] At step 604, if the determined error probability does not exceed the acceptable error probability, the method proceeds to step 606, where it is determined whether the patient is in a dangerous condition. If the patient is in a dangerous condition, the method proceeds to step 607, where the intervention treatment preconfigured to be performed when the patient is in a dangerous condition is performed. If it is determined that the patient is not in a dangerous condition, the method proceeds to step 608, where the intervention treatment preconfigured to be performed when the patient is not in a dangerous condition is performed. The treatment then proceeds to step 609.

[0132] Although the above-described embodiments relate to determining the likelihood of a surgeon making an error based on the acquired eye behavior data, it should be recognized that the present principles can be extended to any parameter (intervention parameter) where the value of the parameter can be determined based on the surgeon's eye behavior and intervention therapy can be determined based thereon. For example, an intervention parameter can indicate the surgeon's experience level or fatigue level. In the case where the eye behavior data indicates the surgeon's experience level, intervention therapy can be implemented for less experienced surgeons (e.g., surgeons whose determined intervention parameter indicating the surgeon's experience is less than a predetermined threshold) rather than for experienced surgeons (e.g., surgeons whose determined intervention parameter indicating the surgeon's experience is greater than a predetermined threshold). Similarly, in the case where the eye behavior data indicates the surgeon's fatigue level, intervention therapy can be implemented for more fatigued surgeons (e.g., surgeons whose determined intervention parameter indicating the surgeon's fatigue is greater than a predetermined threshold) rather than for less fatigued surgeons (e.g., surgeons whose determined intervention parameter indicating the surgeon's fatigue is less than a predetermined threshold). Parameters indicating the surgeon's likelihood of error, the surgeon's experience level, and / or the surgeon's fatigue level are all embodiments of intervention parameters.

[0133] Figure 7 A flow chart of a method implemented by a data processing device 104 according to an embodiment is shown. The method starts at step 700. In step 701, the communication interface 107 receives eye behavior data from the eye behavior monitoring device 101. In step 702, the communication interface 107 receives surgical data generated by one or more surgical data generating devices 109 to 115. In step 703, the processor 105 uses the obtained eye behavior data to determine the value of the intervention parameter associated with the surgical treatment. In step 704, the processor 105 uses the generated surgical data to determine the acceptable range of the value of the intervention parameter. In step 705, it is determined whether the value of the determined intervention parameter is outside the acceptable range of the value of the determined intervention parameter. If not, the method ends at step 707. If yes, the method proceeds to step 706, wherein the communication interface 107 outputs a signal to control the surgical intervention device (e.g., the user interface 115 or the surgical robot 109) to perform an intervention treatment for intervening the surgeon to perform the surgical treatment. Then, the method ends at step 707.

[0134] In embodiments, surgical data (e.g., data from the patient monitoring system 112) indicative of the patient's condition (e.g., critical or non-critical) may be used to determine an acceptable probability of error (or more generally, an acceptable range of intervention parameters) in lieu of or in addition to the type of intervention for a given treatment type and phase. For example, the acceptable probability of error may be higher for a critical patient than for a non-critical patient (thereby allowing the surgeon more autonomy). It should be appreciated that surgical data indicative of the patient's condition may be used in any suitable manner to determine the type of intervention treatment and when to implement the intervention treatment.

[0135] Some embodiments of the present technology are defined by the following numbered clauses:

[0136] (1) A surgical support system comprising:

[0137] an ocular behavior monitoring device operable to monitor the ocular behavior of a surgeon performing a surgical treatment to obtain ocular behavior data of the surgeon;

[0138] a surgical data generating device operable to generate surgical data associated with a surgical procedure performed by a surgeon;

[0139] a surgical intervention device operable to perform an interventional treatment for an interventional surgeon to perform a surgical treatment; and

[0140] Data processing apparatus operable to:

[0141] using the acquired ocular behavior data to determine values ​​of intervention parameters associated with surgical treatment;

[0142] determining an acceptable range of values ​​for an intervention parameter using the generated surgical data; and

[0143] If the determined value of the intervention parameter is outside an acceptable range of determined values ​​of the intervention parameter, the surgical intervention device is controlled to perform an intervention therapy for the intervention surgeon to perform the surgical therapy.

[0144] (2) A surgical support system according to paragraph (1), wherein the eye behavior data is one of pupil monitoring data and eye tracking data.

[0145] (3) A surgical support system according to paragraph (1) or (2), wherein the intervention parameter indicates a possibility of a surgeon making an error in the surgical treatment and an acceptable range of values ​​of the intervention parameter indicates an acceptable possibility of a surgeon making an error in the surgical treatment.

[0146] (4) The surgical support system according to paragraph (3), wherein the data processing device is operable to:

[0147] comparing the obtained surgeon's ocular behavior data with predetermined ocular behavior data associated with a predetermined likelihood that the surgeon will make an error in the surgical procedure;

[0148] The likelihood that the surgeon made an error in the surgical treatment is determined using the predetermined likelihood associated with the predetermined eye behavior data and the result of the comparison between the obtained surgeon's eye behavior data and the predetermined eye behavior data.

[0149] (5) The surgical support system according to paragraph (4), wherein the generated surgical data indicates at least one of a type of surgical treatment, a current stage of the surgical treatment, and a current condition of a patient undergoing the surgical treatment.

[0150] (6) The surgical support system according to clause (5), wherein the data processing device is operable to select predetermined eye behavior data using the generated surgical data.

[0151] (7) The surgical support system according to clause (5), wherein an intervention treatment is selected using the generated surgical data.

[0152] (8) A surgical support system according to any one of paragraphs (3) to (7), wherein the intervention treatment includes: outputting an alarm signal indicating that the determined probability of the surgeon making an error in the surgical treatment exceeds the surgeon's acceptable probability of the surgeon making an error in the surgical treatment.

[0153] (9) The surgical support system according to paragraph (8), wherein the alarm signal is a visual indication displayed on a display on which the surgical image is displayed.

[0154] (10) A surgical support system according to clause (9), wherein the visual indication is superimposed near the area of ​​interest in the surgical image.

[0155] (11) A surgical support system according to any one of paragraphs (3) to (10), wherein the intervention treatment includes: providing an adjustment signal to a surgical device used by a surgeon to perform a surgical function in the surgical treatment, so as to control the surgical device to adjust the surgical function performed to prevent errors in the surgical treatment in advance.

[0156] (12) The surgical support system according to any one of paragraphs (3) to (11), wherein the data processing device is operable to:

[0157] using the acquired ocular behavior data to determine an area of ​​the surgeon's field of view, the area of ​​the surgeon's field of view being associated with an error for which a likelihood has been determined; and

[0158] The intervention treatment is determined based on the determined characteristics of the surgeon's field of view.

[0159] An intervention treatment is determined based on the determined characteristics of the surgeon's field of view.

[0160] (13) The surgical support system according to clause (12), wherein the data processing device is operable to:

[0161] outputting an image representing the surgeon's field of view for display so that the surgeon can view it;

[0162] When the surgeon views the displayed image, controlling the eye behavior monitoring device to obtain further eye behavior data of the surgeon; and

[0163] Further ocular behavior data from the surgeon was used to determine therapeutic intervention.

[0164] (14) The surgical support system according to clause (13), wherein the data processing device is operable to:

[0165] using the obtained further eye behavior data to determine a sub-region of the surgeon's field of view, the sub-region of the surgeon's field of view being associated with an error for which a likelihood has been determined; and

[0166] An intervention therapy is determined based on the determined characteristics of the sub-region of the surgeon's field of view.

[0167] (15) The surgical support system according to any one of paragraphs (3) to (14), wherein the data processing device is operable to:

[0168] determining a region of the surgeon's field of view using the acquired ocular behavior data, the region of the surgeon's field of view being associated with an error for which a likelihood has been determined;

[0169] Outputting an image representing the surgeon's field of view for display to be viewed by the surgeon; and

[0170] controlling the eye behavior device to obtain further eye behavior data of the surgeon while the surgeon views the displayed image;

[0171] Therein, the intervention treatment includes: adjusting display characteristics of the displayed image and using further eye behavior data of the surgeon to determine the adjusted display characteristics to reduce the possibility of errors associated with the surgeon's field of view.

[0172] (16) A data processing device for a surgical support system, the surgical support system comprising: an eye behavior monitoring device operable to monitor the eye behavior of a surgeon performing a surgical treatment to obtain eye behavior data of the surgeon; a surgical data generating device operable to generate surgical data associated with the surgical treatment performed by the surgeon; and a surgical intervention device operable to perform intervention treatment for intervening in the surgeon performing the surgical treatment, wherein the data processing device comprises a circuit configured to:

[0173] receiving the acquired eye behavior data;

[0174] receiving the generated surgical data;

[0175] using the acquired ocular behavior data to determine values ​​of intervention parameters associated with surgical treatment;

[0176] determining an acceptable range of values ​​for an intervention parameter using the generated surgical data; and

[0177] If the determined value of the intervention parameter is outside the acceptable range of values ​​of the determined intervention parameter, a signal is output to control the surgical intervention device to perform an intervention treatment for the intervention surgeon to perform the surgical treatment.

[0178] (17) A method of operating a surgical support system, the surgical support system comprising: an eye behavior monitoring device operable to monitor the eye behavior of a surgeon performing a surgical treatment to obtain eye behavior data of the surgeon; a surgical data generating device operable to generate surgical data associated with the surgical treatment performed by the surgeon; and a surgical intervention device operable to perform intervention treatment for intervening in the surgeon's performance of the surgical treatment, wherein the method comprises:

[0179] receiving the acquired eye behavior data;

[0180] receiving the generated surgical data;

[0181] using the acquired ocular behavior data to determine values ​​of intervention parameters associated with surgical treatment;

[0182] determining an acceptable range of values ​​for an intervention parameter using the generated surgical data; and

[0183] If the determined value of the intervention parameter is outside the acceptable range of values ​​of the determined intervention parameter, a signal is output to control the surgical intervention device to perform an intervention treatment for the intervention surgeon to perform the surgical treatment.

[0184] (18) A program for controlling a computer to execute the method according to paragraph (17).

[0185] (19) A storage medium storing the program according to paragraph (18).

[0186] In light of the above teachings, various modifications and variations of the present disclosure are possible. Therefore, it should be understood that within the scope of the appended claims, the present disclosure can be implemented in other ways than those specifically described herein.

[0187] So far, the embodiments of the present disclosure have been described as being implemented at least in part by a data processing device controlled by software. It should be recognized that non-volatile computer-readable media carrying the software, such as optical disks, magnetic disks, semiconductor memories, etc., are also considered to represent embodiments of the present disclosure.

[0188] It will be appreciated that for clarity, the embodiments have been described above with reference to different functional units, circuits, and / or processors. However, it will be apparent that any suitable distribution of functionality between different functional units, circuits, and / or processors may be used without departing from the embodiments.

[0189] The embodiments described in real time may be implemented in any suitable form including hardware, software, estimation, or any combination thereof. Optionally, the described embodiments may be implemented at least in part as computer software running on one or more data processors and / or digital signal processors. The elements and components of any embodiment may be physically, functionally, and logically implemented in any suitable manner. In some embodiments, the functions may be implemented in a single unit, in multiple units, or as part of other functional units. Therefore, the disclosed embodiments may be implemented in a single unit or may be physically and functionally distributed between different units, circuits, and / or processors.

[0190] Although the present disclosure has been described in conjunction with the following embodiments, it is not intended to be limited to the specific forms set forth herein. In addition, although it may appear that features may be described in conjunction with specific embodiments, it will be appreciated by those skilled in the art that the various features of the described embodiments may be combined in any manner suitable for real-time technology.

[0191] References

[0192] NPL 1: Tien, T., Pucher, PH, Sodergren, MH et al. Surg Endosc (2015) 29:405. https: / / doi.org / 10.1007 / s00464-014-3683-7

[0193] NPL 2:Tim Stockdale“World’s first study of rider’s eye movementscould reveal the key to show jumping success”http: / / www.timstockdale.com / worlds-first-study-of-riders-eye-movements-could-reveal-the-key-to-show-jumping-success /

[0194] NPL 3:SUNY Downstate Medical Center."Novice pilots improve visualresponses to simulation by watching experts'eye movements:Eye movementsreliably distinguish between novice and expert military pilots."ScienceDaily.ScienceDaily,27November 2017.<www.sciencedaily.com / releases / 2017 / 11 / 171127152038.htm>

Claims

1. A surgical support system, comprising: an ocular behavior monitoring device operable to monitor the ocular behavior of a surgeon performing a surgical treatment to obtain ocular behavior data of said surgeon; surgical data generating means operable to generate surgical data associated with said surgical treatment performed by said surgeon; a surgical intervention device operable to perform an interventional treatment for intervening said surgeon in performing said surgical treatment; as well as Data processing apparatus operable to: determining a value of an intervention parameter associated with the surgical treatment using the acquired ocular behavior data, the value of the intervention parameter being used to determine the intervention treatment, the intervention treatment comprising outputting an alarm signal to initiate superimposing an augmented reality visualization on a surgical image of a portion of a peripheral organ or muscle group susceptible to injury; determining an acceptable range of said values ​​of said intervention parameters using the generated surgical data; and controlling the surgical intervention device to perform the intervention treatment for intervening the surgeon in performing the surgical treatment if the determined value of the intervention parameter is outside the acceptable range of the determined values ​​of the intervention parameter, wherein the intervention parameter indicates a probability that the surgeon will make an error in the surgical treatment, and the acceptable range of values ​​of the intervention parameter indicates an acceptable probability that the surgeon will make an error in the surgical treatment, and The generated surgical data indicates at least one of the type of surgical treatment, the current stage of the surgical treatment, and the current condition of the patient undergoing the surgical treatment.

2. The surgical support system according to claim 1, wherein: The eye behavior data is one of pupil monitoring data and eye tracking data.

3. The surgical support system according to claim 1, wherein: The data processing apparatus is operable to: comparing the obtained eye behavior data of the surgeon with predetermined eye behavior data associated with a predetermined likelihood that the surgeon will make an error in the surgical procedure; The likelihood of the surgeon making an error in the surgical treatment is determined using the predetermined likelihood associated with the predetermined eye behavior data and a result of a comparison between the obtained eye behavior data of the surgeon and the predetermined eye behavior data.

4. The surgical support system according to claim 3, wherein: The data processing means is operable to select the predetermined eye behaviour data using the generated surgical data.

5. The surgical support system according to claim 1, wherein: The intervention treatment is selected using the generated surgical data.

6. The surgical support system according to claim 1, wherein: The intervention includes outputting an alarm signal indicating that the determined likelihood of the surgeon making an error in the surgical procedure exceeds the acceptable likelihood of the surgeon making an error in the surgical procedure.

7. The surgical support system according to claim 6, wherein: The alarm signal is a visual indication displayed on a monitor showing the surgical image.

8. The surgical support system according to claim 7, wherein: The visual indication is superimposed proximate a region of interest in the surgical image.

9. The surgical support system according to claim 1, wherein: The interventional treatment includes providing an adjustment signal to a surgical device used by the surgeon to perform a surgical function in the surgical treatment, so as to control the surgical device to adjust the performed surgical function to prevent errors in the surgical treatment in advance.

10. The surgical support system according to claim 1, wherein: The data processing apparatus is operable to: using the acquired ocular behavior data to determine a region of the surgeon's field of view, the region of the field of view being associated with an error for which a likelihood has been determined; and The intervention treatment is determined based on the determined characteristics of the surgeon's field of view.

11. The surgical support system according to claim 10, wherein: The data processing apparatus is operable to: Outputting an image representing the field of view of the surgeon for display so as to be viewed by the surgeon; while the surgeon views the displayed image, controlling the eye behavior monitoring device to obtain further eye behavior data of the surgeon; and The intervention treatment is determined using the further ocular performance data of the surgeon.

12. The surgical support system according to claim 11, wherein: The data processing apparatus is operable to: using the obtained further eye behavior data to determine a sub-region of the surgeon's field of view, the sub-region being associated with an error for which a likelihood has been determined; and The intervention therapy is determined based on the determined characteristics of the sub-region of the surgeon's field of view.

13. The surgical support system according to claim 1, wherein: The data processing apparatus is operable to: using the acquired ocular behavior data to determine a region of the surgeon's field of view, the region of the field of view being associated with an error for which a likelihood has been determined; Outputting an image representing the field of view of the surgeon for display so as to be viewed by the surgeon; and controlling the eye behavior monitoring device to obtain further eye behavior data of the surgeon while the surgeon views the displayed image; Wherein the intervention treatment comprises: adjusting display characteristics of the displayed image and determining the adjusted display characteristics using the further eye behavior data of the surgeon to reduce the possibility of errors associated with the field of view of the surgeon.

14. A data processing device for a surgical support system, the surgical support system comprising: an ocular behavior monitoring device operable to monitor the ocular behavior of a surgeon performing a surgical treatment to obtain ocular behavior data of said surgeon; surgical data generating means operable to generate surgical data associated with said surgical treatment performed by said surgeon; and a surgical intervention device operable to perform an interventional treatment for intervening the surgeon in performing the surgical treatment, wherein the data processing device comprises circuitry configured to: receiving the acquired eye behavior data; receiving the generated surgical data; determining a value of an intervention parameter associated with the surgical treatment using the acquired ocular behavior data, the value of the intervention parameter being used to determine the intervention treatment, the intervention treatment comprising outputting an alarm signal to initiate superimposing an augmented reality visualization on a surgical image of a portion of a peripheral organ or muscle group susceptible to injury; determining an acceptable range of values ​​for said intervention parameters using the generated surgical data; and outputting a signal to control the surgical intervention device to perform the intervention treatment for intervening the surgeon in performing the surgical treatment, if the determined value of the intervention parameter is outside the acceptable range of the determined value of the intervention parameter, wherein the intervention parameter indicates a probability that the surgeon will make an error in the surgical treatment, and the acceptable range of values ​​of the intervention parameter indicates an acceptable probability that the surgeon will make an error in the surgical treatment, and The generated surgical data indicates at least one of the type of surgical treatment, the current stage of the surgical treatment, and the current condition of the patient undergoing the surgical treatment.

15. A storage medium storing a program which, when executed by a computer including the storage medium, causes the computer to function as a data processing device for a surgery support system, the surgery support system comprising: an ocular behavior monitoring device operable to monitor the ocular behavior of a surgeon performing a surgical treatment to obtain ocular behavior data of said surgeon; surgical data generating means operable to generate surgical data associated with the surgical treatment performed by the surgeon; and surgical intervention means operable to perform an intervention treatment for intervening in the surgeon's performance of the surgical treatment, wherein the data processing means is configured to: receiving the acquired eye behavior data; receiving the generated surgical data; determining a value of an intervention parameter associated with the surgical treatment using the acquired ocular behavior data, the value of the intervention parameter being used to determine the intervention treatment, the intervention treatment comprising outputting an alarm signal to initiate superimposing an augmented reality visualization on a surgical image of a portion of a peripheral organ or muscle group susceptible to injury; determining an acceptable range of values ​​for said intervention parameters using the generated surgical data; and outputting a signal to control the surgical intervention device to perform the intervention treatment for intervening the surgeon in performing the surgical treatment, if the determined value of the intervention parameter is outside the acceptable range of the determined value of the intervention parameter, wherein the intervention parameter indicates a probability that the surgeon will make an error in the surgical treatment, and the acceptable range of values ​​of the intervention parameter indicates an acceptable probability that the surgeon will make an error in the surgical treatment, and The generated surgical data indicates at least one of the type of surgical treatment, the current stage of the surgical treatment, and the current condition of the patient undergoing the surgical treatment.

Images