DEVICE AND METHOD FOR DETERMINING A STATE OF A WORK PROCESS
Patent Information
- Application Number
- DE102017205048
- Authority / Receiving Office
- DE · DE
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2017-03-24
- Publication Date
- 2025-03-06
- Estimated Expiration
- 2037-03-24
Smart Images
Figure 00000000_0000_ABST
Abstract
Description
Technical background
[0001] Embodiments of the present invention relate to methods and apparatus for determining a state of a workflow. Background of the invention
[0002] Workflow modeling can be used to specifically support people in their work—this offers the opportunity to increase efficiency, prevent errors, and enhance satisfaction. For this purpose, a workflow, such as a medical intervention, is recorded using multisensory methods.
[0003] By analyzing the data and comparing it with a workflow model, for example, the actual progress and anomalies can be identified and implemented into interactive assistance for the person in charge. This is necessary to implement needs-based assistance, i.e., assistance that can offer appropriate assistance functions depending on the current context (actual progress).
[0004] The prerequisite for this is that a workflow model is available in a suitable (machine-readable) format. The development of such models is often a lengthy process. This is illustrated here using an example from medicine. If, for example, a model is developed solely by technical experts, a lack of medical domain knowledge can lead to errors in the model and low acceptance by the medical domain experts. A paternalistic approach based on fixed workflow models that are not sufficiently comprehensible to the physician can be a further hurdle. These hurdles can lead to the entire assistance system being rejected by those being assisted. This problem can also be expected in other application areas.
[0005] In the field of assistance, a distinction can be made between analog and electronic assistance systems. The former take the form of tools or structures, for example. Electronic assistance systems include robotic systems and information assistants. These can directly intervene in the course of action or influence it through information and decision support. A current patent relating to computer-assisted intervention can be found under publication number US 9,532,848 B2.
[0006] Approaches to generate a non-parameterized model of a workflow based on a description of the workflow and to parameterize the model after creation during a subsequent learning phase in order to obtain a parameterized model for the state determination of the workflow are described, for example, in the following publications: (1) PHILIPP, P. [et al.]: Situation detection for an interactive assistance in surgical interventions based on dynamic bayesian networks. In: GRAS 2016 / 6th Joint Workshop on New Technologies for Computer / Robot Assisted Surgery, September 12-14, 2016, Pisa, Italien, 2 S. - ISBN 9791220014779. URL: https: / / www.researchgate.neUprofile / Patrick Philipp2 / publication / 308074430 Situati on Detection for an Interactive Assistance in Surgical Interventions Based on Dynamic Bayesian Networks / links / 584fd0a308ae4bc8993b354f.pdf, (2) PHILIPP, P. [et al.]: Framework for an interactive assistance in diagnostic processes based on probabilistic modeling of clinical practice guidelines. In: Emerging trends in applications and infrastructures for computational biology, bioinformatics, and systems biology. Amsterdam : Morgan Kaufmann, an imprint of Elsevier, 2016. S. 371-390. - ISBN 978-0-12-804203-8, (3) KATIC, Darke (et al.]: A system for context-aware intraoperative augmented reality in dental implant surgery. In: International Journal of Computer Assisted Radiology and Surgery, Vol. 10, 2015, No. 1, S. 101-108. - ISSN 1861-6410, (4) TRAN, Dinh Tuan; SAKURAI, Ryuhei; LEE, Joo-Ho. An improvement of surgical phase detection using latent dirichlet allocation and hidden markov model. In: Innovation in Medicine and Healthcare 2015. Springer, Cham, 2016. S. 249-261, (5) DERGACHYOVA, Olga, et al. Automatic data-driven real-time segmentation and recognition of surgical workflow. International journal of computer assisted radiology and surgery, 2016, 11. Jg., Nr. 6, S. 1081-1089, (6) LOUKAS, Constantinos; GEORGIOU, Evangelos. Surgical workflow analysis with Gaussian mixture multivariate autoregressive (GMMAR) models: a simulation study. Computer Aided Surgery, 2013, 18. Jg., Nr. 3-4, S. 47-62, (7) PADOY, Nicolas, et al. Statistical modeling and recognition of surgical workflow. Medical image analysis, 2012, 16. Jg., Nr. 3, S. 632-641, (8) WEEDE, Oliver, et al. Workflow analysis and surgical phase recognition in minimally invasive surgery. In: Robotics and Biomimetics (ROBIO), 2012 IEEE International Conference on. IEEE, 2012. S. 1080-1074, (9) STAUDER, Ralf, et al. Random forests for phase detection in surgical workflow analysis. In: International Conference on Information Processing in Computer- Assisted Interventions. Springer, Cham, 2014. S. 148-157. (10) US 2014 / 0 067 413 A1. (11) ANOUAR TADLAOUI, Mouenis ; AAMMOU, Souhaib ; KHALDI, Mohamed: Developement of Bayesian networks from unified modeling language for learner modelling. In: International journal of advanced computer science and applications, Vol. 6, 2015, No. 2, S. 139-144. - ISSN 2156-5570. DOI: 10.14569 / IJACSA.2015.060220.
[0007] There is therefore a need for improved condition determination during a work process. Summary of the invention
[0008] The present invention is based on the object of providing a device and a method which enables the state determination of a workflow using a model, wherein a learning phase for the model, during which the workflow must be repeatedly carried out and recorded, is not required.
[0009] This object is achieved by a device according to claim 1 and by a method according to claim 23.
[0010] The present invention provides a device with a sensor device configured to detect a workspace in which a workflow can be executed and to provide a sensor signal. Furthermore, the device has a processing device configured to obtain a graphical knowledge model of the workflow, for example, a deterministic model of the workflow, and to create a dynamic sequence model of the workflow, for example, a probabilistic model of the workflow, based on the graphical knowledge model. Furthermore, the processing device is configured to determine a state of a dynamic sequence model of the workflow based on the sensor signal.Furthermore, the device has an interface for providing a state signal, wherein the device is designed to provide the state signal based on the determined state of the dynamic sequence model.
[0011] The present invention is based on the idea that a graphical knowledge model is more accessible or understandable for a domain expert, for example, a medical professional, than a dynamic sequence model. At the same time, dynamic sequence models offer a reliable representation and tracking of a workflow. Since dynamic sequence models are usually based on complex mathematical structures, they are difficult to access for non-specialist personnel. Graphical knowledge models, on the other hand, can be represented similarly to a conventional flowchart. The flowchart can, for example, contain textual information and is therefore very easily accessible even to non-specialist personnel, i.e., persons without knowledge of probabilistic workflow modeling.Furthermore, with regard to determining the state of a workflow, the dynamic sequence model offers the advantage of enabling the interpretation of sensor information in terms of probability, even in the case of unexpected events, thus enabling a precise determination of the state. For example, the observation of an unlikely object can be ignored using the dynamic sequence model, as the detection may have been based on a faulty or noisy sensor signal. Thus, the dynamic sequence model allows for a more reliable mapping of states from a planned workflow to an actually executing workflow.
[0012] The processing device is designed to map output pins and objects from activity nodes of the graphical knowledge model to nodes of the dynamic sequence model. The described mapping rule for mapping the output points of the graphical knowledge model to nodes of the dynamic sequence model can be advantageously used to efficiently and easily obtain a dynamic sequence model based on parts, for example, the output pins, of the graphical knowledge model.
[0013] According to embodiments, the device is configured to adapt the dynamic sequence model based on the sensor signal. The described device can flexibly adapt to a field of application by adapting the dynamic sequence model to the application conditions. In other words, the device provides a self-learning mechanism for adapting to changing environmental conditions.
[0014] According to embodiments, the device is designed to provide an instruction for an assistance device designed to influence the workflow, in the current or in a subsequent state of the dynamic sequence model, based on the state signal. The described device can advantageously provide information that the assistance device can use to influence the workflow in a desired direction. For example, the assistance device can have a gripper arm that can assist a worker in lifting heavy objects or assembling workpieces. The display of general information or possible courses of action, for example on a screen or using head-mounted glasses in the sense of virtual or augmented reality, represents a further possibility for favorably influencing the workflow.
[0015] According to exemplary embodiments, the dynamic sequence model is based on a dynamic Bayesian network. Bayesian networks are particularly advantageous for sequence modeling because they are suitable for incorporating prior knowledge about action sequences or workflows into state decisions and for modeling the dynamic or temporal behavior of sensor signals and the sequence of features obtained from them. Furthermore, events detected in the sensor signal can be classified and evaluated using Bayesian networks.
[0016] According to embodiments, the graphical knowledge model is based on an activity diagram and is designed to depict the workflow. An activity diagram can be advantageous for enabling simple modeling of a workflow. In particular, activity diagrams are easy to understand, especially for people who cannot readily understand modeling in the form of a dynamic sequence model.
[0017] According to exemplary embodiments, the activity diagram is based on the Unified Modeling Language (UML). The use of UML enables the well-structured and standardized creation of activity diagrams. This can achieve a high degree of order, which in turn can lead to easier understanding of the graphical knowledge model.
[0018] According to embodiments, the processing device is designed to assign activity nodes of the graphical knowledge model to states of the workflow. Furthermore, the processing device is designed to assign nodes of the dynamic sequence model to states of the dynamic sequence model. The described embodiment can advantageously map elements of the graphical knowledge model to the dynamic sequence model using a simple mapping rule. Thus, portions of the graphical knowledge model can be mapped to the dynamic sequence model with little effort, for example, in an automated manner.
[0019] According to embodiments, the processing device is designed to assign annotations to the activity nodes or annotations to branching nodes of the graphical knowledge model and edges of the graphical knowledge model to nodes (Xt0,X00) of the dynamic sequence model. The described device can easily translate or transfer a graphical knowledge model into a dynamic sequence model using the described mapping rule.
[0020] According to embodiments, the processing device is designed to create the dynamic sequence model by linking an activity during the workflow to an object of the workflow in the graphical knowledge model. Furthermore, the processing device is designed to obtain an object flow of the graphical knowledge model based on the link and to map the object flow to a directed path of the dynamic sequence model, wherein an object flow can also be synonymously referred to as a data flow. Here, the directed path is arranged between a root node and an associated child node. The described mapping can be advantageously used to assign observations of the sensor signal of objects to corresponding states of the dynamic sequence model.
[0021] According to embodiments, the state is a first state, and the device is configured to detect an event in the workspace based on the sensor signal. Furthermore, the device is configured to determine a second state of the dynamic sequence model based on the first state and on a probability of occurrence of the event in the first or second state. The described embodiment can advantageously utilize probabilities of occurrence of events, for example, observation of tools, to evaluate the probability of the observation in connection with a new state, for example, the second state. In other words, if the device considers an observation to be probable for a state, the device can enable a decision toward a more probable state based on the dynamic sequence model.For example, a state transition from the first state to the second state can be triggered based on the observation of a tool that is more likely to occur in the second state than in the first state.
[0022] According to embodiments, the device is configured to obtain or derive the occurrence probability from the graphical knowledge model. The described embodiment can advantageously accept occurrence probabilities from operating personnel who are not experienced in handling dynamic sequence models by enabling input via the graphical knowledge model. Thus, even a person who is not experienced in handling dynamic sequence models can still very easily parameterize the device. In particular, expert knowledge from people who are very familiar with the workflow but not with dynamic sequence models can be transferred into the dynamic sequence model.
[0023] According to embodiments, the processing device is designed to determine the state of the dynamic sequence model using transition probabilities based on nodes of the dynamic sequence model. For example, if a node of the dynamic sequence model corresponds to a state of the workflow, the device can determine the probability of a state change using the dynamic sequence model based on the knowledge that the dynamic sequence model is in that state and, based on this, switch to a new state, for example, a state associated with a chronologically subsequent work step. Using transition probabilities can enable a precise determination of a transition from one state to another state.
[0024] According to embodiments, the processing device is designed to adapt the occurrence probabilities and / or the transition probabilities using recorded sensor data. The described embodiment can advantageously use the recorded sensor data describing the working area to adapt the probability values that parameterize the dynamic sequence model. This can compensate for any missing or incorrect initial parameterization.
[0025] According to embodiments, a probability of occurrence of an event is correlated with a remaining or changing state of the dynamic sequence model. Furthermore, a transition probability describes a probability of changing from a first state to a second state, and the processing device is designed to estimate the current state during the workflow based on evaluations of the sensor signal. The described embodiment can advantageously be adapted using recorded sensor data, so that a person skilled in the art only needs to perform an initial initialization. The initial initialization can, for example, include providing probabilities of occurrence of events or transition probabilities of states, as is known to a physician, for example, during an operation.
[0026] According to embodiments, the processing device is configured to determine the state based on a plurality of temporally spaced detections of the working area based on a temporal averaging of the sensor signals. Using temporal averaging, noise or interference components of the sensor signal, i.e., the detection, can be suppressed. Thus, improved sensor signals can be provided to the processing device to more reliably determine the state of the dynamic sequence model.
[0027] According to embodiments, the sensor device is designed to provide the sensor signal based on a visual detection of the work area. Visual detection can be easily performed, for example, using a camera, which allows a wide range of information about the work area to be easily captured.
[0028] According to embodiments, the sensor device is designed to provide the sensor signal based on acoustic detection of the work area. Acoustic detection can be advantageous, for example, to identify the tools used in a work step based on noise or to enable classification of the current work step based on dialogues between personnel.
[0029] According to embodiments, the sensor device is designed to provide the sensor signal based on radio frequency detection of the work area. Detection of radio frequency identification numbers (RFIDs) can, for example, enable the detection of tools used.
[0030] According to embodiments of the invention, the device comprises a user interface configured to receive input from a user that at least partially describes the workflow. Furthermore, the device is designed to provide the graphical knowledge model based on the user input. The described embodiment can enable simple parameterization of the graphical knowledge model for a user, for example, via a computer using a keyboard and mouse.
[0031] The present invention provides a system for workflow support, comprising a device as described herein, a work area, and an assistance device. The described system can advantageously assist a worker in a difficult or complicated task.
[0032] The present invention provides a method. The method comprises detecting a workspace in which a workflow can be executed. The method further comprises providing a sensor signal based on the detected workspace. The method further comprises obtaining a graphical knowledge model of the workflow and creating a dynamic sequence model of the workflow based on the graphical knowledge model of the workflow. The method further comprises determining a state of the dynamic sequence model based on the sensor signal. Furthermore, the method comprises providing a state signal based on the determined state of the dynamic sequence model. The method can be expanded to include all features and functionalities mentioned in connection with the device described herein. Short character description
[0033] Embodiments according to the present invention are explained in more detail below with reference to the accompanying figures. They show: Fig. 1 is a schematic block diagram of a device according to embodiments of the invention; Fig. 2 shows a workflow diagram of a graphical knowledge model according to embodiments of the invention; Fig. 3a-c show elements of workflow diagrams according to embodiments of the invention; Fig. 4 shows a creation of a workflow using a graphical knowledge model and a dynamic sequence model according to embodiments of the invention; Fig. 5 shows a dynamic portion of a dynamic Bayesian network for two time periods according to embodiments of the invention; Fig. 6 a dynamic Bayesian network according to embodiments of the invention, rolled out over three time steps; and Fig. 7 a method according to embodiments of the invention. Detailed description of the implementation examples
[0034] Workflows, according to the invention, can describe any type of work. Examples of workflows include operations for medical personnel, manufacturing and / or assembly of products. In general, a workflow can be a regularly repeated activity in various areas. A workflow can be modeled in a workflow plan, which generally represents different work steps.
[0035] Fig. Figure 1 shows a schematic block diagram of a device 100 according to embodiments of the invention. The device 100 comprises a sensor device 110, a processing device 120, and an interface 130. Furthermore, the device 100 can comprise an optional user interface 140 and / or an optional assistance device 150.
[0036] The sensor device 110 is configured to detect a workspace 160 in which a workflow can be executed and to provide a sensor signal 112 based on the detection. The processing device 120 is configured to obtain a graphical knowledge model 122 of the workflow and to create a dynamic sequence model 124 based on the graphical knowledge model 122. Furthermore, the processing device 120 is configured to determine a state of the dynamic sequence model 124 of the workflow based on the sensor signal 112. The processing device 120 can access both the graphical knowledge model 122 and the dynamic sequence model 124. The interface 130 is configured to provide a state signal, wherein the device 100 is configured to obtain the state signal based on the determined state of the dynamic sequence model 124.
[0037] The sensor device 110 can, for example, be configured to detect the work area 160 such that tools occurring or appearing therein can be detected. Based on the detection, the sensor signal 112 can provide the processing device 120 with information about occurring objects. The processing device 120 can, based on the sensor signal and a current state of the dynamic sequence model 124 associated with a state of the workflow, provide an estimate or a statement about the current state or a subsequent state of the workflow. For example, if the sensor signal detects a specific object, the device 100 can assess the probability of the object occurring in the current state. The assessment can, for example, include the processing device evaluating how likely the object occurring is in the current state.Based on the evaluation, the processing device 120 can switch to a state in which the specified object is more likely to occur, for example, if the occurrence in the current state is assessed as unlikely. Based on the determined state of the dynamic sequence model 124, the processing device 120 can estimate a state of the workflow and make this estimate or this state available via the interface 130 in the form of the state signal. Based on the state signal, for example, interventions for the workflow can be planned. For example, a worker performing the workflow can be warned if, for example, a step has been skipped or an incorrect tool is used.Likewise, for example based on the estimated state, an assistance device, such as assistance device 150, can interact with the workflow, in other words, influence, intervene, or participate in the sequence of actions. Assistance device 150 could, for example, be a gripper arm that supports a worker. For example, processing device 120 could recognize that an object is too heavy to be lifted by a human and, as a result, instruct a gripper arm to lift the object. In general, device 100 offers the advantage that the derivation of dynamic sequence model 124 is based on a graphical knowledge model 122. This allows for easy parameterization of the workflow, since a graphical knowledge model is generally more accessible or understandable for people who are not familiar with probability theory.In particular, graphical knowledge modeling allows for simple graphical modeling, for example, using a workflow diagram. Graphical modeling can be easier to understand for personnel who are not otherwise involved in probabilistic workflow planning.
[0038] Fig. 2 shows a flowchart 200 of a graphical knowledge model according to embodiments of the invention. The flowchart 200 can, for example, describe a cholecystectomy, i.e., a medical workflow plan for removing the gallbladder. States 210, 215, 220, 225, 230, 235, 240, 245, and 250 represent states of the workflow, wherein the states in connection with a workflow plan can also be referred to as work steps. In the first state 210, or the work step, for example, a pneumoperitoneum can be created, i.e., gas can be pumped into the patient's abdomen to inflate the patient's abdomen. This process can, for example, last 186 seconds, which is represented by the annotation 210a.
[0039] In a second step 215, for example, trocars can be placed, where trocars are tubes with sharpened ends. Step or state 215 is assigned a probability of 0.95 that trocars 200a will be used or detected in the work area, i.e., with 95 percent probability, the sensor device, e.g., sensor device 110, will detect trocars in the detection of the work area. State 215 lasts, for example, 195 seconds, which is indicated by annotation 215a.
[0040] In a third state 220, for example, the bile duct can be dissected. This state can last, for example, 558 seconds, which is indicated by annotation 220a. During the third state 220, the probability of occurrence of trocars 200a remains 95%. However, additional tools can now also be detected, such as a grasper 200b with a probability of 90%, a dissecting device (severing device) 200c with a probability of 92%, and an HF coagulation device or cutter 200d with a probability of 59%.
[0041] In a fourth state 225, for example, the bile duct can be clamped and cut (clamping and cutting). The described fourth state 225 can last, for example, 114 seconds, indicated by annotation 225a. During the fourth state, the known devices or tools 200a and 200b can occur with the known occurrence probabilities. In addition, tools 200e, e.g., laparoscopic scissors (lap. scissors) with a probability of 90%, and 200f, e.g., a stapler (clamping device, tacker), with a probability of 91%, can occur.
[0042] In a fifth step 230, for example, the gallbladder artery can be exposed, whereby this state can last, for example, 108 seconds, indicated by annotation 230a. During the preparation of the gallbladder artery, the known devices 200a, 200b, and 200c can appear with the known probabilities.
[0043] In a sixth state 235, for example, the gallbladder artery can be clamped and cut. This sixth state 235 can last, for example, 115 seconds, indicated by annotation 235a. For this purpose, the already known tools 200a, 200b, 200e, and 200f can be observed with the known occurrence probabilities.
[0044] Subsequently, an optional state 240 can be reached in the workflow; this is indicated by branch 236. In branch 236, a case distinction is made, which is conditioned, for example, by the worker's knowledge. For example, a physician may recognize before or during the operation that the optional step 240 might be necessary. The physician can indicate this to the device 100 via the user interface 140, for example. The system may remain in state 240 for 600 seconds, for example, according to annotation 240. The optional state 240 may, for example, be assigned to a work step that requires imaging of the bile duct. During state 240, the sensor device should only detect the tools 200a, for example, the aforementioned trocars.The dynamic system can recognize that step 240 is present even without input, since it is highly likely that only trocars are used here, whereas more tools are used in step 245. Furthermore, it is known from annotation 234 at junction 236 (expert knowledge) that intraoperative imaging (240) is necessary in 45% of surgeries, thus a transition from 235 to 240 occurs.
[0045] After performing the optional work step 240, the standard workflow can be returned via node 241. In a further state 245, which, according to annotation 245a, can last 534 seconds, the gallbladder, for example, can be removed (removed). During this state, the known tools 200a, 200b, 200c, 200d, 200e, and 200f can be observed with the known occurrence probabilities.
[0046] In a final step 250, which, according to annotation 250a, can last 192 seconds, coagulation of the liver bed can be performed. During state 250, the known tools 200a and 200d can occur with the known probabilities.
[0047] In the workflow plan 200, the tools or objects 200a-f are connected to the states 210, 215, 220, 225, 230, 235, 240, 245, and 250 via object flows using output pins 200a'-f'. Even though flowchart 200 was primarily described in connection with a medical workflow, a workflow plan according to workflow plan 200 can also describe, for example, a workflow for assembling motor vehicles or other complex activities. The workflow plan can have any number of states to which objects 200a-f can be assigned in the form of output pins 200a'-f', or their probabilities of occurrence for a specific state, for example, via user input. Here, the probability of occurrence of an object or event can vary in different workflow states, ie object 200a could be expected to be 95% in a first step and to be expected to be 60% in a second step.These probabilities can indicate how frequently or likely an event occurs during the state, such as a specific object being used or present in the workspace. Alternatively, the associated probability can also refer to another event, such as a change in an object's location or state. This could also include partial or complete consumption of the object.
[0048] Paths between the states, such as paths or edges 211, 216, 221, 226, 231, or 246, can describe a transition from one state to another, for example, when the activities in a state are executed, an associated time period 210a, 215a, 220a, 225a, 230a, 235a, 240a, 245a, or 250a has elapsed, and / or another criterion is met. States 210, 215, 220, 225, 230, 235, 240, 245, and 250 can be understood as activity nodes, upon reaching which an associated activity is assumed in the workflow. Furthermore, an object flow connects an output pin 200a'-f' to an object 200a-f.
[0049] The graphical knowledge model 200 is not very suitable for direct sensor signal processing, but it does enable a convenient and simple assignment of activities to be performed in the workflow to the states / activity nodes 210, 215, 220, 225, 230, 235, 240, 245, and 250, as well as an assignment of probabilities of occurrence of certain events in the states, which is represented via the output pins 200a'-f'. Relationships and transitions between the states can be represented by the paths 211, 216, 221, 226, 231, or 246. In summary, the graphical knowledge model can represent an expected sequence of the work diagram. Deviations from the depicted workflow are not, or only inadequately, taken into account. This is, however, made possible by dynamic sequence models, which also include dynamic Bayesian networks, which, in connection with Fig. 5 and Fig. 6 are explained.
[0050] Devices according to the invention are configured to obtain a graphical knowledge model, such as the graphical knowledge model 200, and to convert it into a dynamic sequence model, and to sensor-detect the execution of the workflow and to determine states in the dynamic sequence model using or evaluating the sensor data.
[0051] The model 200 can be fed to the device 100 via the user interface 140 and used there to coordinate or support the described workflow.
[0052] Furthermore, Fig. Figure 2 shows the UML activity of a cholecystectomy. It begins with a sequential sequence of actions. The action "Intraoperative Cholangiography" is performed optionally. This activity is derived from a description in
[14] ,
[15] , and
[16] .
[0053] Fig. 3a-c show elements of workflow diagrams and / or complete workflow diagrams according to embodiments of the invention. In particular, Fig. 3a and Fig. 3b workflow diagrams with a start node 310 and an end point 390 and therefore represent complete workflow diagrams. On the other hand, the work steps arranged between the start node 310 and end node 390 can also serve as elements of other workflow diagrams.
[0054] Fig. Figure 3a shows an activity 1 that has a start node 310 and transitions from this start node to state A. When work actions of the workflow that are assigned to state A are completed or executed, a transition from state A to state B is indicated. This sequential order can be mapped by the processing device 120 to a dynamic sequence model and then used to determine the state. After completion of activities that are assigned to state B, the workflow flows into Fig. 3a to an endpoint 390.
[0055] Fig. 3b shows an activity 2 that describes a decision option or branching option within a workflow. Starting from a starting point 310, the workflow runs to a decision node 320. Depending on a decision criterion, here, for example, a threshold decision, a decision can be made here as to whether the workflow moves to state A, i.e., the activities assigned to state A are carried out, or whether the workflow switches to state B, i.e., the work steps assigned to state B are carried out. The threshold decision for selecting the branching path can be carried out, for example, by the processing device 120, and the resulting state of the workflow can be determined therefrom by the processing device 120.Furthermore, decision node 320 is attached to an annotation that describes information about the probability of switching to the respective branches. In other words, how often a particular branch is taken. The workflow plan then merges into node 330 and ends at the end node 390.
[0056] Fig. 3c shows excerpts of activities, Activity 3 and Activity 4, which can be part of workflows. Activity 3 shows the workflow state A, which is coupled to an object y, via an object flow, where the object y can occur in state A with a probability of 95%. Furthermore, an annotation is attached to state A, which can provide information about the state A. The link between the workflow state A and the object y can be used by the processing device 120 to infer the state A via the dynamic sequence model if the sensor signal detects the object y. Activity 4 can be interpreted analogously to Activity 3, where in a different representation, an object y is assigned to the state A with a probability of occurrence of 95% via an output pin of the activity node A and an object flow.Likewise, state A in activity 4 has an annotation that can provide information about state A. The connection between state A and object y in activity 3 can also be characterized as a flow or stream.
[0057] Fig. Figures 3a and b show two typical workflow models in the context of surgical procedures. In a), actions A and B are executed sequentially. In b), a decision must be made to perform either A or B. The decision is represented by a guard (expression). If this evaluates to "true," the corresponding action is executed. Fig. Figure 3c) shows two equivalent representations of a flowing object. This means that an object flow can occur while A is still being executed (not only after execution). Furthermore, the activities contain textual annotations that can be used to store additional information.
[0058] Fig. Figure 4 shows a representation of a workflow in a graphical knowledge model, here a UML activity diagram, and its transformation into a dynamic sequence model, here a dynamic Bayesian network. Fig. The example shown in Figure 4 is based on a medical example. Knowledge from medical textbooks and guidelines is used, either in combination with a medical expert or in a dialogue between domain experts, to define a workflow. Domain experts can be, for example, engineers or physicians. Based on the workflow plan, a UML activity diagram can be generated, which is then automatically translated into a dynamic Bayesian network. The device 100 can receive the defined workflow, here in the form of the UML activity diagram, via the user interface 140 in order to define the graphical knowledge model 122. Based on the graphical knowledge model 122, the device can now determine the dynamic sequence model 124, here the dynamic Bayesian network, and use it to determine the state.
[0059] Furthermore, Fig. 4 that medical knowledge can be transformed into a UML activity of the surgical workflow. The formalization process can be carried out through an expert dialogue or by the medical expert themselves.
[0060] The resulting activity serves as an interface to more complex models used for the actual capture of a surgical step. Embodiments according to the invention describe translation rules for transferring a given UML activity into a dynamic Bayesian network (DBN).
[0061] In the previous section, Bayesian networks were described as an example of a dynamic sequence model. Other examples of the foundations of dynamic sequence models include neural networks, hidden Markov models, Markov networks, or partially observable Markov decision processes.
[0062] In Hidden Markov Models, states, such as individual operational phases, are hidden from the observer (=technical system). However, the technical system attempts to infer possible states (e.g., the current state) using observations (=recorded sensor values). This can be considered a special case of DBN.
[0063] Neural networks or recurrent neural networks, such as long-short-term memory networks, can account for temporal or sequential dependencies in data. This not only allows for the recognition of whether something is currently being "cut" or "stitched" (action recognition), but also allows for the possibility that the first action was "cut" and therefore the next one is highly likely to be "stitched."
[0064] Markov networks (Markov Random Fields) are similar to Bayesian networks (BN), but the dependencies are undirected, i.e., in contrast to BNs, cyclic dependencies can be modeled directly.
[0065] Partially observable Markov decision processes (POMDP) are similar to hidden Markov models, but rather than being purely observational, they model actions. However, as with hidden Markov models, the course of partially observable Markov decision processes is hidden from the actual observer (=technical system).
[0066] Other alternatives to the graphical knowledge model, in addition to the already mentioned UML, are Business Process Modeling Notation (BPMN), Event Driven Process Chain (EPC) and colored Petri nets.
[0067] Fig. 5 shows a dynamic part of a dynamic Bayesian network 500 B →In particular, Fig. 5 shows a simplified graph of a two-disk temporal Bayesian network (2TBN). In the context of a DBN, a 2TBN (or: B → ) is used as a template for consecutive time steps t. A DBN consists of two networks. The tuple (B0, B → ). B0 represents the starting distribution and B → describes how a network develops over time. A Bayesian network (B0) serves as the initial starting point for consecutive time steps. For simplification, Xt−11:4 omitted, since in this example there is no direct dependency on Xt−10:4 Starting from a state Xt−10 In a first time period t-1, a transition to a second state Xt0, take place in a second time period t. Starting from the state Xt0 observations can be related to the state Xt0 Observations can be associated, for example, Xt1,Xt2,Xt3,Xt4 which, for example, are assigned occurrence probabilities of 4 tools. The occurrence probabilities can be calculated from the Fig. 2 are obtained for the observation of characteristics, such as the use of instruments. Furthermore, Fig. 2 the a-priori probability of individual phases as well as the probability of transitioning into individual phases are transferred to the root nodes of the dynamic Bayesian network. In other words, the Bayesian network can be mapped by mapping phases of a workflow, for example phases or states 210, 215, 220, 225, 230, 235, 240, 245 or 250 of the graphical knowledge model 200, to the nodes Xt−10,Xt0, .... a Bayesian network in the processing device 120. Furthermore, the Bayesian network can be created by mapping objects of a workflow and their probabilities, for example object probabilities of the objects 200a-f, to nodes Xt1,Xt2,…,Xt6 generated in the processing device 120.
[0068] The device 100 is designed to create the dynamic sequence model using the graphical knowledge model. For this purpose, the processing device 120 maps at least one, several, or all of the activity nodes 210, 215, 220, 225, 230, 235, 240, 245, 250 into a state or node of the dynamic sequence model. For this purpose, the processing device is designed to implement a mapping rule according to VB0={ve|e=(ei,ej)∈DF,ei∈A,ej∈O}∪{vr} to be applied, where V B0 is a set of nodes of the dynamic sequence model, v eis a node of the dynamic sequence model, (e i , e j ) an edge (edge) e or path between an object e j and an action e j of the graphical knowledge model, DF is a set of object flows of the graphical knowledge model, A is the set of actions or states of the graphical knowledge model, 0 is the set of object nodes of the graphical knowledge model, v r is a root node of the dynamic sequence model and where an object flow connects an action of set A with an object of set 0 in the graphical knowledge model. Furthermore, V B0 For example, describe an initial network structure of the sequence model, which is generated in a first step by the processing device 120.
[0069] This means that the processing device 120 is designed to process a node v e of the dynamic sequence model for an object e jto create a state or an action e i and whose connection is contained in the graphical knowledge model. The set V B0 is obtained by combining the nodes v e with the root node v r where the root node describes the state of the workflow.
[0070] The device 100 is designed to create the dynamic sequence model using the graphical knowledge model. For this purpose, the processing device creates nodes of the dynamic sequence model based on activity nodes of the graphical knowledge model according to VB→={ve**|e*=(ei*,ej*)∈DF,ei*∈A,ej*∈O}∪{vr*}∪VB0, where V B0 is a previously determined set of nodes of the dynamic sequence model, where vr* characterizes the root node of the dynamic sequence model, for example at an advanced time t that differs from a previous time t - 1. ve** describes child nodes that V B→ are added that characterize the objects or their probability of occurrence.
[0071] This means that the processing device 120 is designed to process nodes vr*,ve** an original or initial node set V B0 of the dynamic sequence model, which corresponds to a state (root node vr*) or an object (child node ve∗∗ whose linkage is contained in the graphical knowledge model to form the set V B→ to obtain.
[0072] The device 100 is designed, for example, to create the dynamic sequence model using the graphical knowledge model. For this purpose, the processing device 120 can generate directed paths E B0 between nodes of the dynamic sequence model based on object flows of the graphical knowledge model according to EB0={(vr,ej)|(ei,ej)∈DF,ei∈A,ej∈0} generate, where E B0 is the set of directed paths of the dynamic sequence model, v r is a root node of the dynamic sequence model that characterizes a state, DF is the set of object flows of the graphical knowledge model, A is the set of actions or states of the graphical knowledge model, and 0 is the set of object nodes of the graphical knowledge model.
[0073] This means that the processing device 120 is designed to generate a path (v r ,e j ) of the dynamic sequence model to create an object e jthat corresponds to a state or an action i assigned to a particular object, and whose link is contained in the graphical knowledge model, using the path. The set E B0 is created by connecting the root node v r with the objects e j where the root node describes the state of the workflow.
[0074] The device 100 is designed, for example, to create the dynamic sequence model using the graphical knowledge model. For this purpose, the processing device 120 can create directed paths of the dynamic sequence model based on structures of the graphical knowledge model according to EB→={(vr∗,ej∗)|(ei∗,ej∗)∈DF,ei∗∈A,ej∗∈0}∪{(vr,vr∗)} generate, whereby (vr∗,ej∗) a path between a current root node vr∗ and an object ej∗ describes, (ei∗,ej∗) a path between a current action ei∗, the root node vr∗ and an object ej∗ is and (vr,vr∗) a path between a current root node vr∗, for example, at time t, and a previously determined root node v r , for example at time t - 1. The set of paths E B0 is not added because the child nodes of v r in the 2TBN at time t-1 have no direct influence on the nodes at time t.
[0075] In other words, the processing device 120 is designed to connect temporally preceding root nodes with immediately temporally following root nodes, wherein the root nodes describe states of the workflow and additionally to generate paths between root nodes and child nodes, corresponding to links of actions or states with objects in the graphical knowledge model.
[0076] According to one embodiment, the processing device 120 is designed to create the 2TBN for the creation of the directed paths such that in time step t - 1 only those random variables are included that actually have an influence on the random variables in time step t (so-called interface variables). According to embodiments, this is only the variable Xt−10, and thus node v rThis can also be understood as the 2TBN being generated such that in time step t - 1 all random variables of the network are included. However, since not all variables necessarily have an influence on the next time step t, variables without influence do not receive an edge in the graph. For example, these are all children of the root from time step t - 1. This definition is consistent with equation (1). Note, however, that according to the described embodiments, for example in Fig. 5 and Fig. 6, P(Xt0:N|Xt−10:N)=P(Xt0:N|Xt−10) applies.
[0077] Furthermore, this means that a root node, for example Xt0, with a temporally following root node Xt+10 is linked by the processing device 120. Here, the nodes Xt0 and Xt+10 describe the same state of the workflow, or describe two consecutive states of the workflow based on the child nodes or the observations and time information, for example, how long a state lasts. Furthermore, Xt0 and Xt+10 be considered as a node or its temporal evolution or dynamics that characterizes the workflow. Alternatively, Xt0 be considered as a plurality of nodes, corresponding to a considered time t.
[0078] According to embodiments, the processing device is designed to assign annotations to the activity nodes or annotations to branching nodes of the graphical knowledge model and edges of the graphical knowledge model to nodes (Xt0,X00) of the dynamic sequence model. The root Xt0 by B→ can also contain parameters, namely the transition probabilities (matrix A), as described in equations (1) and (3). Here, the matrix A can be calculated according to equations (13) and Xt0 π can be calculated or generated according to equation (12) and X00 of B0 via equation (2).
[0079] Fig. 6 shows a dynamic Bayesian network 600 consisting of the pair (B0,B → ), which is rolled out over three time steps. For simplicity, some nodes have been omitted, which is graphically represented by "...".
[0080] Fig. 7 shows a method 700 according to embodiments of the invention, comprising the following steps: detecting 710 a workspace in which a workflow can be executed. Providing 720 a sensor signal based on the detected workspace. Obtaining 730 a graphical knowledge model of the workflow. Creating 740 a dynamic sequence model of the workflow based on the graphical knowledge model of the workflow. Determining 750 a state of the dynamic sequence model based on the sensor signal. Providing 760 a state signal based on the determined state of the dynamic sequence model.
[0081] In general, graphical knowledge models can be created offline, i.e., before execution of a workflow, and serve to parameterize the dynamic sequence model. Furthermore, dynamic sequence models can be characterized as real-time sequence models or online-capable classifiers, since they are used to recognize successive steps or states of a currently executing workflow based on the evaluation of the sensor signal. In other words, the subsequent target model (can serve to evaluate the actual sensor values ("online," i.e., while the sensor values are coming into the system, a successive classification is performed, meaning an attempt is made to recognize the current phase). In general, graphical knowledge models can also be referred to as parameterization models, since they are designed to enable the parameterization of a workflow.Furthermore, dynamic sequence models can also be called processing models because they are designed to directly process data based on sensor signals to enable detection of the state of the workflow.
[0082] An example of a dynamic sequence model is a DBN. While dynamic sequence models such as DBNs can theoretically be generated offline through user input, these models are too complex to be edited / generated in an expert dialog.
[0083] A represented UML activity can be converted into a dynamic sequence model by implementing or transferring the parameters. This mapping can be ambiguous, because a DBN can be implemented in different ways (network structures, etc.). Through the implementation, a concrete DBN is created from the knowledge represented in the graphical knowledge model, which can actually be used for classification. The conversion can therefore have smoothing properties, for example. The output of the model is smoothed (i.e. the classification result). It is smoothed by the model saving other classification results during the current execution and thus relating the current classification to previous classifications. A sequence of sensor data is therefore considered, not just individual feature values at a time. This means that the model can detect “false” orpay less attention to “noisy” data and thus smooth out the output.
[0084] Example: The model, or the processing device using the dynamic sequence model, has previously assumed, based on the incoming sensor data, that Phase 1 of a workflow predominates. Now, feature values that are typical of Phase 10 (e.g., due to sensor noise) enter the system for a short time. A system that does not consider the sequence would immediately classify it as Phase 10. A system that incorporates the sequence can smooth this out, for example, by initially retaining the previous hypothesis (Phase 1).
[0085] In other words, compared to the UML model, the DBN contains additional knowledge that is incorporated during the conversion to the DBN, for example, in the form of a specific target network structure being selected. This knowledge is thus added through the described conversion, for example, according to equations (6) to (11). Likewise, with the other possible dynamic sequence models, not all degrees of freedom of the target model are covered by the initial model. By specifying a specific conversion, these degrees of freedom are then initially set in the target model. For example, a degree of freedom can be the number of layers when using a neural network; this can be determined depending on conversion rules.
[0086] In other words, if a DBN is represented as a graphical knowledge model, this model cannot be used to classify the state of the workflow, meaning it cannot be used to evaluate sensor data online and classify features. This is because, in addition to the graphical knowledge model, the DBN makes further assumptions that are necessary for a specific classification (e.g., the specific network structure of the DBN, or, for example, the structure of the layers in a neural network). Furthermore, the parameters in the DBN can be configured in such a way that they can be advantageously used for classification (e.g., through smoothing, noise suppression).
[0087] The following describes the background and examples of graphical knowledge modeling. These models generally focus on capturing expert knowledge (not classifying observation sequences). Such a model can be presented to the operator in embodiments according to the invention, and the individual elements of the graphical knowledge model can be determined using machine-assisted dialogs. The knowledge can be represented in the form of a graph for ease of understanding. In UML activities, for example, this concerns the definition of possible sequences of actions. This can be expressed by directed edges (arrows) (A->B). A transition in such a model can be deterministic.Alternatively, using a branching method, it can be determined that in 50% of cases a first path is followed and in 50% of cases a second path is followed, whereby both the number of paths and the probabilities can be varied as desired.
[0088] The following describes the background and examples of sequence modeling. In a classification task, the prediction of the current class may be dependent on other classification results. This means that feature vectors flowing into the system must be considered over time. Example: speech recognition: If at time 1 the word "Guten" was predicted, at time 2 the prediction "Gulasch" (although possibly highly likely when viewed in isolation) is very unlikely in the context of the sequence, and the word "Tag", for example, is more likely. Classic probabilistic methods here include HMMs and DBNs. Neural networks are becoming increasingly important.Even if neural networks are not generally referred to as probabilistic models, they are referred to and understood as such in the context of the probability analysis presented here. Properties of these models can be represented as graphs. In DBNs, for example, the independence of random variables; in neural networks, for example, the interconnection of individual layers. These graphs are not (or only with difficulty) suitable for expert dialogue in the sense of formalizing expert knowledge. Therefore, graphical knowledge models are understood to be models designed for the graphical representation of a workflow, such as UML activities, and thus go beyond the mere possibility of graphical representation.
[0089] Devices and methods according to the invention provide for converting the graphical knowledge model into the dynamic sequence model and for monitoring the work area through sensor detection. The sensor data is evaluated to determine the state of the dynamic sequence model of the workflow. This means that the dependence of the graphical knowledge model on the workflow described therein is overcome by the devices, methods, and computer programs described herein.
[0090] Embodiments according to the invention describe a framework for expert-based workflow modeling (work plan modeling) for interactive work process support. The framework can involve all participating experts in a dialogue for knowledge formalization.
[0091] According to the examples, an activity from the Unified Modeling Language can be selected as the basis for formalization in a first step. This activity is so easy to understand that a formalization can be developed jointly in expert dialogue. This means (using the example of medicine) that physicians, knowledge engineers, and technicians can engage in the formalization process "on equal terms." This increases the efficiency and effectiveness of the formalization and, subsequently, the acceptance of the formalization by the decision-makers involved.
[0092] Because UML activities are easy to understand, it is easy for physicians to modify a workflow themselves. This may be necessary to incorporate short-term knowledge (e.g., as a result of medical conferences), personal preferences, or patient-specific conditions into the models. This can lower barriers, i.e., increase physicians' acceptance of the modeling, prevent the feeling of being "patronized," and close the gap between theoretical knowledge and practical action. A further advantage is the transfer / processing of workflow interrelationships into process steps for controlling the assistance, which are adapted based on observations of the sensor signals. This is achieved by using the dynamic sequence model obtained by translating the graphical knowledge model.
[0093] According to the implementation examples, the second step involves automatically translating the UML activity into a more complex model, a dynamic Bayesian network (DBN). This model is used to actually identify, for example, the current progress of the intervention. This model itself does not provide suitable interfaces for directly incorporating expert knowledge into the dialogue (see above). It is not understandable for all of the experts involved. For this reason, a transfer of the models is necessary.
[0094] The framework can be used to design assistance systems that are accepted by the assistants, for example due to their simplicity, and can actually be integrated into everyday work as a relief tool.
[0095] Differences to conventional concepts are explained below.
[0096] With existing systems, physicians cannot adapt the underlying workflow independently enough. The development of the workflow model by the various domain experts may be less efficient and effective due to a lack of common formalization. This can result in, for example, a lack of acceptance of the assistance system in part or in whole. This means that existing systems do not sufficiently reduce potential barriers and do not close the gap between theoretical knowledge and practical action.
[0097] Embodiments according to the invention can be used in all industries where work processes are supported interactively. These include assembly processes in industry, surgical interventions in medicine, emergency processes in disaster management, etc.
[0098] Aspects of the invention relate to surgical workflows, dynamic Bayesian networks, unified modeling language, support / assistance and planning.
[0099] Embodiments of the invention relate to expert-based sequence modeling of workflows in the context of surgical interventions
[0100] In today's medical field, the use of advanced computer-based assistance (both software and hardware) has become an evolving field [1]. For example, the global market for medical robotics and computer-assisted surgical equipment is projected to reach $4.6 billion by 2019 (using a five-year annual growth rate of 7%) [2].
[0101] Regarding the example case of surgical interventions, computer-based support can be implemented in numerous ways - therefore, the next subsections outline three typical types of support and their relationship to situational awareness.
[0102] In robot-assisted surgery [3], in which medical robots perform specific sub-actions of a surgery, the support should ideally integrate seamlessly into the workflow of the human surgical team. The basis is the assessment of the progress of the surgery, i.e., the current phase of an underlying medical workflow. Another type of computer-based support in a medical context, for example performed by the assistance device 150, is the initiation of administrative processes (e.g., an automatic notification of the medical ward that a surgery is about to end) or assistance in preparing a surgical report (e.g., providing a start and end time for a surgery) [4]. Both types depend heavily on both the recording of the progress of an ongoing surgery and the knowledge of a predefined surgical workflow.
[0103] To meet the challenge of the huge amount of available data (e.g., patient records or surgical guidelines including specific actions in emergency procedures), support that provides tailored (e.g., context-sensitive) information during an ongoing operation [5], for example, via the user interface 140, is of interest. This assumes that the current progress of an operation and possible derivations of a predefined surgical workflow support the surgical team (e.g.,) with appropriate examination results or options for action. The management of a surgical team opens up room for optimization. This includes, for example, decision support for the surgeon. In general, this can help to avoid treatment errors, improve patient outcomes, and maintain a high level of satisfaction for both patients and staff.
[0104] In the following, a modeling approach and corresponding prerequisites for tailored support are discussed, taking into account the specific conditions of an ongoing operation: first, a suitable representation of the underlying workflow; second, a reliable capture of the current progress by synchronizing this workflow model with sensor data. To address the first challenge of a proper workflow representation, a framework for an expert-based formalization of medical knowledge is introduced. The Unified Modeling Language (UML) can be used to provide a graphical representation of the workflow through UML activities, facilitating the dialogue between technical and medical experts.Regarding the second challenge of reliable progress tracking, the use of a dynamic sequence model based on dynamic Bayesian networks (DBN) is developed. To combine both aspects, a framework is proposed that enables the automatic translation of UML activities into DBN. To model workflows using a UML activity, medical knowledge must be concentrated and formalized. This formalization process can be realized through a dialogue between experts from the medical and technical fields, see . Fig. 4.
[0105] Due to their ease of understanding, UML activities also offer medical experts the opportunity to modify an already formalized workflow themselves. Because the activity representation serves as a workaround, barriers can be reduced: The fact that physicians understand the formalized representation of the workflow and can modify it themselves reduces their fear of being dictated and allows them, for example, to easily adapt a workflow to specific constraints. For the actual implementation of a support function that assists the physician during a surgery, UML activities are translated into more complex models. The structure and parameters of these models are generated automatically.
[0106] The concept of combining UML activities with other, more complex models is the subject of numerous research papers [6], [7], [8]. To the best of our knowledge, none of the current research papers teaches UML activities in the context of a dialogue between experts from different domains (e.g., experts from technical and medical fields). This also applies to the translation of UML activities into DBNs. In [1], the translation of UML activities of diagnostic processes into Bayesian networks (BNs) is presented. Within the scope of the present invention, it is shown how this approach can be adapted to surgical workflows. This can be done by extending the translation of UML activities to dynamic Bayesian networks (DBNs). The resulting DBNs are evaluated using a common use case for surgical processor modeling
[10] : cholecystectomy (the surgical removal of the gallbladder).This approach can easily be adapted to other applications, such as hip replacement
[11] .
[0107] UML activities are examined in more detail below.
[0108] UML is used and accepted in various disciplines worldwide
[12] . UML activities were chosen as an interface due to their ease of understanding for experts from the medical and technical fields
[13] . Understanding the workflow representation is a necessary prerequisite for dialogue between experts. A. Basics
[0109] UML activities can be used to graphically represent how a particular process or algorithm is executed
[17] . Activity diagrams are constructed from different notational elements connected by directed edges (arrows) and support the modeling of sequences of actions, as well as selection and iteration
[12] .
[0110] Fig. Figure 3 shows typical workflows that can occur in workflows such as cholecystectomy. The beginning of an activity is represented by a black dot (start node), and the end of an activity is symbolized by a double circle (activity end). Individual actions within the workflow are symbolized by rounded rectangles. Arrows represent the flow and thus the order in which the actions can be performed.
[0111] While sub-figures a) and b) in Fig. 3 show two different types of flow charts, Fig. c) represent two equivalent parts of an activity. The fact that an object flow can occur while action A is being executed (and not only after A is completed) is specified by the keyword {stream}. An abbreviated form is a black pin, shown in Activity 4. The activities in subfigures b) and c) include textual annotations linked to action A and a decision node, respectively, to store additional information. B. Modeling a surgical workflow
[0112] Fig. Figure 2 can be interpreted as a formalized model of a cholecystectomy (surgical removal of the gallbladder). In the upper part of this UML activity, a sequence of actions is shown. The first action 210 is the injection of carbon dioxide to inflate the abdomen (stomach). The average time of this action is represented by the corresponding annotation. Then, the trocars (sharpened tubes) 215 are used to break through the abdominal wall. Trocars are used, for example, to enable the placement of additional medical instruments during the operation. Black pins (exit pins) represent the observation probability (values in parentheses next to the tools 200a-f) of trocars in this action ( Fig. 2).
[0113] In subsequent procedures, the cystic duct and artery are exposed (220 and 2230), and clamped and severed (225 and 235). Intraoperative cholangiography (240) (radiographic imaging of the bile ducts with contrast medium) is optional, so a decision must be made. After merging the two possible ducts, the gallbladder is severed from the liver bed (245), and the liver bed is coagulated to stop the bleeding, i.e., tissue is destroyed or burned using electricity.
[0114] The following section discusses the background of Dynamic Bayesian Networks.
[0115] There are many reasons to consider dynamic Bayesian networks (DBNs)
[18] as a modeling tool for dynamic systems
[19] . With regard to modeling medical workflows, DBNs were chosen because they offer an appropriate trade-off between expressiveness and complexity and include sequence models that have proven successful in practice (e.g., hidden Markov models)
[10] . In addition, DBN models allow for improved modularity and interpretability due to their factored state space. Unlike a hidden Markov model, their state space can be expressed in a factored manner and not just as a single discrete random variable. Furthermore, with respect to Kalman filter models, DBNs allow for arbitrary probability distributions (not only unimodal linear Gaussian distributions)
[16] .
[0116] Although the DBN offers a knowledge engineer a good compromise between expressiveness and complexity, the formal representation of a DBN is less suitable for a collaborative dialogue between experts from the medical and technical fields. Furthermore, modification of the model by the medical expert themselves seems impractical, for example, due to its complexity. Consequently, translation rules are introduced to transfer a comprehensible UML activity into a DBN. A. Basics
[0117] A Bayesian network (BN) is a probabilistic graphical model (PGM) that combines graph-theoretic approaches with approaches from probability theory
[20] . A dynamic Bayesian network (DBN) is an extension of a BN that also considers the temporal dependencies of variables
[19] . A DBN is given by a pair: DBN=(B0,B→), where the Bayesian network B0 P(X00:N) used to calculate the a priori probability distribution over random variables X 0:N in a time step with index 0. Furthermore, B specifies → the conditional probability distribution over discrete time steps t by using: P(Xt0:N|Xt−10:N)=∏n=0NP(Xtn|Pa(Xtn)).
[0118] In (1) Pa(Xtn) the number of parents of Xtn in the corresponding graph. The parents can be in the same time slice (e.g., representing immediate causation) or in the previous one (i.e., assuming the model is a first-order Markov model). In the latter case, arcs point to time slices with increasing index, thus representing the causal flow of time
[19] . In Fig. 5 is an example graph of B →, which is also known as a two-disk temporal Bayesian network (2TBN). Since the child nodes in the 2TBN at time t-1 have no direct influence on the nodes at time t, they are not connected by edges to the root node at time t-1. In this case, with respect to equation (1), P(Xt0:N|Xt−10:N)=P(Xt0:N|Xt−10).
[0119] The nodes that have a direct influence on the nodes at time t are also referred to as interfaces. For simplicity, the time t-1 of a 2TBN can be represented or defined via the corresponding interface.
[0120] For T time slices, the composite distribution can be represented graphically by “rolling” the DBN, where B0 is the initial distribution and B → be used as a template for each subsequent time slice. Regarding a DBN that is rolled out for 3 time slices, Fig. 6. Similar to HMM, parameters of such a DBN with N children can be grouped as follows (see Fig. 6): P(X00=i)=π(i) P(Xt0=j|Xt−10=i)=A(i,j) P(Xt1=j|Xt0=i)=B(1)(i,j) P(XtN=j|Xt0=i)=B(N)(i,j)
[0121] Thus, (2) shows the initial probability distribution, which at time step 0 (X00) with root node X 0 It should be noted that the statement P(X = x) is a shorthand for an event ω ∈ Ω : f x (ω) = x, where the set of possible outcomes is given by Ω and f x maps an event ω to a possible value of a random variable X. Possible values x of X are denoted by Val(X)
[20] .
[0122] In (3), the probability distributions of state transitions are given. This expresses the dependencies over time (and between states). Consequently, A(i, j) is an adjacency matrix augmented by transition probabilities for nonzero entries. In (4) and (5), the probability distribution for observations concerning a child is given. Note that an HMM can be specified by a single matrix B(i, j), since the corresponding probability distribution cannot be factored as in the case of a DBN—i.e., graphically, the root node would have only one child containing the entire probability distribution.
[0123] The following discusses the translation of a UML activity into a DBN.
[0124] A UML activity is given, represented as a graph U = (N, F). The set N can be further divided into different sets of nodes: • A: Set of actions, for example 210, 215, 220, 225, 230, 235, 240, 245 and 250, • I,E: set of initial nodes, for example 205, set of final nodes, for example 255, • B: Set of decision and merge nodes (branching nodes), for example 236 and 241, • O: Set of object nodes, for example 200a-f.
[0125] The set of object nodes is given by the set of data pins. A node that is part of one of the node sets I, E, B is called a control node. Furthermore, the set F is given by the following: • KF: Control flow, i.e. activity edges connecting actions and control nodes as well as edges between themselves, for example connections 211, 216, 221, 226, 231 and 246. • DF: Object flow, ie activity edges connecting actions and object nodes or between control nodes and object nodes, for example 200a'-f'. A. Building the DBN structure
[0126] Formally, the translation [[U]] of a UML activity U into the network structure of B0 and B → of the DBN by the following: [[(N,F)]]=(VB0,EB0,VB→,EB→), using: VB0={ve|e=(ei,ej)∈DF,ei∈A,ej∈O}∪{vr}, EB0={(vr,ej)|(ei,ej)∈DF,ei∈A,ej∈O} VB→={ve∗∗|e∗=(ei∗,ej∗)∈DFei∗∈A,ej∗∈O} ∪{vr∗}∪VB0, EB→={(vr∗,ej∗)|(ei∗,ej∗)∈DF,ei∗∈A,ej∗∈O} ∪{(vr,vr∗)}.
[0127] According to (6), output pins of UML actions are converted into vertices (nodes) of the graph of B0. In addition, an auxiliary node v radded to the network to represent the root of the resulting Bayesian structure. Note that UML nodes with the same names (e.g., the same output pins of different actions) are indeed translated multiple times. However, since the target node has the same name in each case, it is added only once to the vertex set (node set) V B0 of the graph of B0. This happens because adding an element to a set of which the element is already a part does not change the set.
[0128] Furthermore, in (7) the object flow connecting an action to an object is transformed into a directed edge from the root node v r of the graph from B0 to the corresponding child.
[0129] The vertex set V B→ of the graph of B → is given by (8) and (9). First, output pins of UML actions are placed in vertices of the graph V B→Furthermore, the amount which vr∗ and the vertex set V B0 with the vertex set V B→ An asterisk (star) is used to avoid ambiguities and to mark vertices that belong to time slice t of the graph of B → heard. See Fig. 5 for an example of a simplified 2TBN. The simplification in Fig. 5 is that the child nodes of v r are not shown, since they do not have any edges in the 2TBN and therefore cannot serve as an interface to the next time slice. The set of edges E B→ (10) contains edges of the root node vr∗ to the corresponding children. The edges are derived from the object flow that connects an action to an object.
[0130] Finally, the root node v r of disk t-1 and the root node vr∗ of disk t by the directed edge (vr,vr∗) connected (11) and the set E B→ added. B. Parameterization of the DBN structure
[0131] In addition to the structure, the parameters of the DBN must be specified. First, the values of the (discrete) random variables, denoted by v r and vr* are linked. Xt0 are the random variables that are vr* for time steps t, and Xt−10 are the random variables associated with v rare linked. The values of these random variables correspond to a natural number, where each of the different possible phases of the operation is assigned a unique natural number by a bijective function f : A → {1, ... , N} using N = |A|. Without loss of generality, the numbers assigned by f correspond to a lexicographical sequence. One obtains: Val(Xt0)={f(v)|v∈A} Val(Xt−10)={f(v)|v∈A} where Val(·) is used to denote the set of values that a linked random variable can take. For all other random variables, a Bernoulli distribution is used. This means that random variables X 1:N , which are child nodes of v r and vr* are linked, are binary-valued, e.g., Val(Xti)={0, 1}, where i = 1, ...,N with N=|VB→\({vr*}∪VB0)| is used. Furthermore, the initial state distribution and the CPD (conditional probability distribution) of observations and state transitions are specified. Thus, the initial state distribution π is given by: π(f(v))={1,if (ie if) (u,v)∈KF∧u∈I0,otherwise (ie otherwise), where f : A → {1, ..., |A|} is the bijective function that assigns a unique natural number to each action v ∈ A, as defined above. Since there is only one dedicated initial state, the probability of its presence in the initial distribution is 1; all other states are assigned 0 as the initial probability (12).
[0132] The probability distribution of state transitions is given by (13). Its representation resembles an adjacency matrix with transition probabilities as edge weights (similar to HMM). The matrix A is given by: A(i, j)={u.an.du−1u.an.du,if u=v, u⊆A1−u.an.du−1u.an.du,if (u,v)∈KF∧{u,v}⊆A1,if (u,v)∈KFb.an.prob(u,v)∧u∈A∧v∈E⋅(1−(u.an.du−1u.an.du))if ∃b∈B:{(u,b), (b,v)}⊆KF∧∃w∈A:(w,b)∈KF0,otherwise, where i = f(u),j = f(v), with f : A → {1, ..., |A|} is again the bijective function that assigns a unique natural number to each action, as defined above. The number u.an.du gives the average duration of the phase associated with action u ∈ A, and b.an.prob(u, v) gives the probability of transitioning to a phase v, given that phase u is exited. Both u.an.du and b.an.prob(u, v) are specified in the annotation for the corresponding action u ∈ A and branch node b ∈ B, respectively, as explained in the section on Dynamic Bayesian Networks. Note that a directed edge from a source node u to a destination node v is given by the edge (u, v), whereas {u, v} denotes a set of a source and a destination node.
[0133] In UML activity, pins denote the features observed during an operation. Some of the pins o ∈ O are attached to different actions but represent the same feature, i.e., they have the same name. O Name is the set of unique pin names present in the activity. A surjective function g : 0 → O Name is defined. This function returns the name of a given object node o. A second, bijective function h : O Name → {1, ... , N} is defined, where N = |O Name | holds. This function h assigns to each element of O Name lexicographically a unique natural number.
[0134] To specify the underlying CPD for each feature, a matrix B k for each element o Name ∈ O Name defined: ∃! Bk ∀oname∈Oname:k=h(oname).
[0135] The elements bi,j+1k each matrix B k are given by the following: bi,j+1k={P(Xtk=j|Xt0=i),if j=11−P(Xtk=j|Xt0=i),otherwise where j∈Val(Xtk)={0, 1} and i∈Val(X0t)={f(v)|v∈A} applies as defined above.
[0136] The conditional probabilities P(Xtk=j|Xt0=i) are derived from the state of the target nodes. Each target node o ∈ O is associated with a probability value o.prob, A=(0.99120000.0088000000.99130.00480000.003900000.998100000.001900.008000.99820000000.0 093000.99070000000000.9946000.0054000.00170000.99830000000001.000000000.005100000.9949) B2=(0.100,100,100,100,100,900,900,900,900,900,900,900,900,900,100,100,100.10)T and P(Xtk=1|Xt0=i)={0.probif ∃(a,o)∈DFwitha∈A,o∈O,(h∘g)(o)=k,f(a)=iθ,otherwise.
[0137] Where θ = 0.1 represents the default value of a false-positive prediction of a feature. This allows a UML activity to be translated into a DBN, where vector π represents the initial state distribution, matrix A represents state transitions, and matrices B represent the k for the probability distributions of observations.
[0138] In the following, results of the described approach are discussed.
[0139] Regarding the UML activity of cholecystectomy, the translation into a DBN structure results in graphs that Fig. 5 and Fig. 6. In the present case, the structures have 6 children, as 6 different features are used. These features are (in lexicographical order) the use of: 1) clipping device 2) dissection device 3) grasper (gripper) 4) HF coag. / cutting 5) lap. scissors 6) trocars
[0140] The values of the (discrete) random variables associated with v r and vr* are as follows: Val(Xt0)={1,2,3,4,5,6,7,8,9}, Val(Xt−10)={1,2,3,4,5,6,7,8,9}.
[0141] This uses a lexicographical order of the names of surgical phases to assign the corresponding number. The order is as follows: 1) Clipping and cutting bile duct (clamping and cutting the bile duct) 2) Clipping and cutting cystic artery (clamping and cutting the gallbladder artery) 3) Detaching of the gallbladder 4) Dissection bile duct (dissection of the bile duct) 5) Dissection cystic artery (freeing the gallbladder artery) 6) Generation of pneumoperitoneum 7) Intraoperative Cholangiogram (Intraoperative Cholangiography) 8) Liver bed coagulation 9) Placement of trocars
[0142] Consequently, the initial state distribution is: π=(0,0,0,0,0,1,0,0,0,)T given because the surgical procedure (the described cholecystectomy) (and the corresponding UML activity) always begins with the creation of a pneumoperitoneum.
[0143] With regard to the state transition probabilities, A is given by (14). With regard to the equation in (13), the row entries of matrix (14) add up to 1. It should be noted that the values have been rounded to allow a compact representation of A. Matrix A contains entries that characterize state transitions. In particular, it can be seen that a state transition is less likely than a change to another state because the main diagonal elements of matrix A are larger than the elements on the side diagonals. Furthermore, most rows, with the exception of the 2nd row, contain only 2 entries; this describes the sequential processing of the workflow. In other words, a state can in most cases only remain in itself or can only change to a predefined other state. Row 2 describes the branching possibility, i.e., that it can either go to the 3rd state or to the 7th.State can be changed. This corresponds to the branching option to state 235 in . Fig. 2. For example, the values of the first row of matrix A using equation (13) are: A(1,1)=114−1114=113114≈0.9912. Since the phase with index 1 (clamping and cutting the bile duct) has only one possible subsequent phase with index 5 (freeing the gallbladder artery), according to equation (13) the matrix entry A(1,5) results in 1−113114≈0.0088. If a phase has no subsequent phase, only the transition into itself is possible. Consequently, the last phase with index 8 (liver bed coagulation) has the matrix entry A(8,8) = 1. If a phase has several possible subsequent phases, the probability of leaving the phase is divided. For example, A(2,2) = 0.9913. The probability of leaving the phase is then 0.0087. This is then divided according to the ratio of the transition probabilities of the possible subsequent phases. Specifically, this results in the matrix entries A(2,3) = 0.0048 and A(2,7) = 0.0039. The addition of both entries results in 0.0087 and thus describes the probability of leaving the current phase. Because a phase transition must occur after each time step (this includes the transition of a phase into itself), the sum of each row of the matrix is always 1.Please note that the decimal values are rounded to allow a compact representation of A.
[0144] For simplicity, not all observation probabilities are shown that are given by B k In (15) B 2 , which encodes the probabilities of the recorded use of the gripper depending on the surgical phase. The transposition (T) serves to facilitate the representation of the matrix B 2 in landscape format.
[0145] The nine columns of the matrix shown in (15) represent the different surgical phases. The first row corresponds to the probability that the gripper is not used in a phase. The second row is the probability that the gripper is used depending on the phase. This means that the columns sum to 1.
[0146] It should be noted that the probabilities contain two sources of expert knowledge: First, the probability that a specific instrument will be used depending on the specific phase (medical knowledge). Second, the probability that a specific instrument used depending on the specific phase will be detected (technical knowledge). Even if, for example, an expert from the medical field estimates the probability of using an instrument in a specific phase as 1, the technical expert must take into account the accuracy of the algorithms used, e.g., an accuracy of detecting a use of 90%. Furthermore, the processing device 120 can also use, for example, prior knowledge of detection algorithms to adjust the occurrence probabilities, i.e., to determine uncertainty regarding detection based on an uncertainty of the algorithm.
[0147] On the one hand, estimation errors introduced by a limited number of experts providing the corresponding probabilities and the influence of unknown variables not expressed in the UML model represent limitations of the proposed process. On the other hand, the resulting (expert-based) models can be further refined using knowledge sources, e.g., a large amount of observational and / or expert data, once they become available. To deal with unknown variables, it is further possible to introduce leaky variables in the context of constructs such as "noisy or"
[21] to represent unmodeled causes. Performance measures for the proposed DBN model can be found in
[22] . Thus, a DBN was used to estimate the progress of a synthetic operation.
[0148] Some conclusions of the described concept are discussed below.
[0149] Embodiments according to the invention introduce a framework for modeling surgical procedures. The framework uses UML activities as an interface for dynamic Bayesian networks (DBN). These models are used for the actual estimation of the progress of a surgical procedure. As an example application, a cholecystectomy (removal of the gallbladder) was presented, which is a common use case.
[0150] Activities are used as an interface because of their comprehensibility for experts from both the medical and technical fields. This is a necessary prerequisite for dialogue between experts and also enables workflow modifications by the attending physician themselves. This can help reduce barriers to workflow modeling—for example, reducing the fear of regimentation by the medical expert. At the same time, self-directed formalization can help close the gap between theoretical knowledge and practical solutions in the context of a surgical procedure.
[0151] Using translation rules, multiple extended models can be obtained from a single, understandable activity. In embodiments according to the invention, the focus was on translating a UML activity into dynamic Bayesian networks. This model can then be used to provide a basis for sophisticated support functions, taking the progress of an operation into account.
[0152] In summary, the use of medical workflows plays an important role in providing support functions in the context of surgical procedures. Workflow models can be used to assess the progress of an ongoing operation, enabling the attending physician to provide tailored (i.e., context-sensitive) support. This subsequently offers opportunities to prevent treatment errors, improve patient outcomes, and maintain a high level of satisfaction. According to exemplary embodiments, a framework is proposed that enables the formalization of medical workflows. It is promoted through a dialogue between medical and technical experts and is based on the Unified Modeling Language (UML).A simple, understandable UML activity serves as a starting point for the automatic generation of complex models that can be used to actually estimate the progress of a surgical procedure. Translation rules, according to exemplary embodiments, enable the transfer of a given UML activity into a dynamic Bayesian network (DBN). The methods are presented for the application example of a cholecystectomy (surgical removal of the gallbladder). BIBLIOGRAPHY [1] P. Philipp, Y. Fischer, D. Hempel, and J. Beyerer, “Framework for an interactive assistance in diagnostic processes based on probabilistic modeling of clinical practice guidelines,” in Emerging Trends in Applications and Infrastructures for Computational Biology, Bioinformatics, and Systems Biology. Elsevier, 2016, pp. 371-390. [2] A. McWilliams, Medical Robotics and Computer-Assisted Surgery: The Global Market. Baishideng Publishing Group Inc, 2014, vol. HLC036F. [3] R. A. Faust, Robotics in surgery: history, current and future applications. Nova Publishers, 2007. [4] N. Padoy, T. Blum, H. Feussner, M.-O. Berger, and N. Navab, „On-line recognition of surgical activity for monitoring in the operating room.“ in AAAI, 2008, pp. 1718-1724. [5] S. Glöser, „Frage der Woche an Prof. Dr. med. Markus Schwarz,“ Dtsch Arztebl International, vol. 112, no. 23, pp. [4]-, 2015, Accessed: 2016 / 11 / 01. [Online]. Available: http: / / www.aerzteblatt.de / int / article.asp?id=170832 [6] Z. Han, L. Zhang, and J. Ling, „Transformation of uml activity diagram to yawl,“ in Enterprise Interoperability IV. Berlin: Springer, 2010, pp. 289-299. [7] W. van der Aalst and K. M. Van Hee, Workflow management: models, methods, and systems. MIT press, 2000. [8] A. Hofstede, N. Russell, W. Aalst, and M. Adams, Modern Business Process Automation: YAWL and its Support Environment. Berlin Heidelberg: Springer, 2010. [9] H. Störrle, „Semantics of control-flow in UML 2.0 activities,“ in Visual Languages and Human Centric Computing, 2004 IEEE Symposium on, 2004, pp. 235-242.
[10] F. Lalys and P. Jannin, „Surgical process modelling: a review,“ International journal of computer assisted radiology and surgery, vol. 9, no. 3, pp. 495-511, 2014.
[11] I. D. Learmonth, C. Young, and C. Rorabeck, „The operation of the century: total hip replacement,“ The Lancet, vol. 370, no. 9597, pp. 1508 - 1519, 2007.
[12] OMG, „Unified Modeling Language Superstructure Version 2.4.1,“ 2011, Accessed: 2014 / 08 / 28. [Online]. Available: http: / / www.omg.org / spec / UML / 2.4.1 / Superstructure / PDF /
[13] P. Philipp, Y. Fischer, D. Hempel, and J. Beyerer, „Modeling of clinical practice guidelines for interactive assistance in diagnostic processes,“ in HIMS 2015, International Conference on Health Informatics and Medical Systems, july 27-30, Las Vegas, Nevada, USA, 2015, pp. 3-9.
[14] N. J. Soper, L. L. Swanström, and S. Eubanks, Mastery of endoscopic and laparoscopic surgery. Lippincott Williams & Wilkins, 2008.
[15] G.-Q. Ding, W. Cai, and M.-F. Qin, „Is intraoperative cholangiography necessary during laparoscopic cholecystectomy for cholelithiasis?“ World Journal of Gastroenterology: WJG, vol. 21, no. 7, p. 2147, 2015.
[16] N. Padoy, „Workflow and activity modeling for monitoring surgical procedures,“ Ph.D. dissertation, Université Henri Poincaré-Nancy I, 2010.
[17] C. Rupp, S. Queins, et al., UML 2 glasklar. München: Carl Hanser Verlag GmbH Co KG, 2012.
[18] T. Dean and K. Kanazawa, „A model for reasoning about persistence and causation,“ Computational intelligence, vol. 5, no. 2, pp. 142-150, 1989.
[19] K. P. Murphy, „Dynamic bayesian networks: representation, inference and learning,“ Ph.D. dissertation, University of California, Berkeley, 2002.
[20] D. Koller and N. Friedman, Probabilistic graphical models: principles and techniques. MIT press, 2009.
[21] F. J. Diez, „Parameter adjustment in Bayes networks. The generalized noisy ORgate,“ in Proceedings of the Ninth international conference on Uncertainty in artificial intelligence, 1993, pp. 99-105.
[22] P. Philipp, L. Schreiter, J. Giehl, Y. Fischer, J. Raczkowsky, M. Schwarz, H. Woern, and J. Beyerer, „Situation detection for an interactive assistance in surgical interventions based on dynamic bayesian networks,“ CRAS 2016, 6th Joint Workshop on New Technologies for Computer / Robot Assisted Surgery, 2016
Claims
[1] Device (100) comprising: a sensor device (110) designed to detect a work area (160) in which a work sequence can be carried out and to provide a sensor signal (112); a processing device (120) which is designed to obtain a description of a graphical knowledge model (122) of the workflow and to create and parameterize a dynamic sequence model (124) of the workflow based on the graphical knowledge model; and to determine a state of the dynamic sequence model of the workflow based on the sensor signal; an interface (130) for providing a status signal; wherein the device is designed to provide the state signal based on the determined state of the dynamic sequence model, wherein the graphical knowledge model has a plurality of activity nodes, wherein the processing device is designed to map output pins (200a'-f') and objects (200a-f) from activity nodes (210, 215, 220, 225, 230, 235, 240, 245, 250) of the graphical knowledge model to nodes (Xt1,Xt2,…,Xt6) of the dynamic sequence model, and wherein the processing device is designed to generate nodes of the dynamic sequence model based on activity nodes based on V B0 = {v e |e = (e i ,e j ) ∈ DF,e i ∈ A,e j ∈ O} ∪ {v r}, where v e a node of the set V B0 that represents an object (e i , e j ) an edge e or path between an object e j and an action e i of the graphical knowledge model, DF is a set of object flows of the graphical knowledge model, A is the set of actions or states of the graphical knowledge model and ei is an action or state, 0 is the set of object nodes of the graphical knowledge model and e j is an object node, v r is a root node, where an object flow connects an action of set A with an object of set 0 in the graphical knowledge model, or based on VB→={ve**|e*=(ei*,ej*)∈DF,ei*∈A,ej*∈O}∪{vr*}∪VB0, where V B0 is an original set of nodes of the dynamic sequence model, vr* a root node from a current step, (ei*,ej*) an edge e* or path between an object ej* and an action ei∗ of the graphical knowledge model, DF is a set of object flows of the graphical knowledge model, A is the set of actions or states of the graphical knowledge model and ei∗ is an action or state, 0 is the set of object nodes of the graphical knowledge model and ej∗ is an object node, a star indicates a link to a current state of the graphical knowledge model, where an object flow connects an action of set A with an object of set 0 in the graphical knowledge model. [2] The apparatus of claim 1, wherein the apparatus is configured to adapt the dynamic sequence model based on the sensor signal. [3] Apparatus according to claim 1 or claim 2, wherein the apparatus is adapted to provide an instruction to an assistant device (150) adapted to intervene in the workflow in the current or a following state of the dynamic sequence model based on the state signal. [4] Apparatus according to any one of claims 1 to 3, wherein the dynamic sequence model is based on a dynamic Bayesian network (500, 600). [5] Device according to one of claims 1 to 4, wherein the graphical knowledge model is based on an activity diagram (200) and is designed to depict the workflow. [6] Apparatus according to claim 5, wherein the activity diagram is based on the Unified Modeling Language (UML). [7] Device according to one of claims 1 to 6, wherein the processing device is designed to associate an activity node (210, 215, 220, 225, 230, 235, 240, 245, 250) of the graphical knowledge model with a state of the workflow, and to associate a node (Xt−10,Xt0) of the dynamic sequence model to the activity node of the graphical knowledge model. [8] Device according to claim 7, wherein the processing device is designed to assign annotations to the activity nodes (210, 215, 220, 225, 230, 235, 240, 245, 250) or annotations assigned to branching nodes (236) of the graphical knowledge model and edges (211, 216, 221, 226, 231, 246) of the graphical knowledge model to nodes (Xt0,X00) of the dynamic sequence model. [9] Device according to one of claims 1 to 8, wherein the processing device is designed to link an activity during the workflow to an object of the workflow in the graphical knowledge model for creating the dynamic sequence model in order to obtain an object flow of the graphical knowledge model, and to map the object flow to a directed path of the dynamic sequence model, wherein the directed path is arranged between a root node and an associated child node. [10] Device according to one of claims 1 to 9, wherein the processing means is designed to generate paths E B0 of the dynamic sequence model based on object flows of the graphical knowledge model according to EB0={(vr,ej)|(ei,ej)∈DF,ei∈A,ej∈0} to generate, where E B0 is the set of directed paths of the dynamic sequence model, v r is a root node of the dynamic sequence model, DF is the set of object flows of the graphical knowledge model, A is the set of actions or states of the graphical knowledge model, 0 is the set of object nodes of the graphical knowledge model, e i is an action of the set A, e j is an object of the set 0 and (e i , e j ) denotes an edge between an action and an object of the graphical knowledge model. [11] Device according to one of claims 1 to 10, wherein the processing device is designed to generate, in a further step, directed paths of the dynamic sequence model based on object flows of the graphical knowledge model according to EB→={(vr∗,ej∗)|(ei∗,ej∗)∈DF,ei∗∈A,ej∗∈0}∪{(vr,vr∗)} to generate, where E B→ is the set of directed paths of the dynamic sequence model in a current step, v r is a root node of the dynamic sequence model from a previous step, vr∗ is a root node from the current step, (vr,vr∗) is a path between the root node from the previous step and the current step, (vr∗,ej∗) a path between the root node of the current step and a child node ej∗ of the current step, where the path (vr∗,ej∗) of the dynamic sequence model with a path (ei∗,ej∗) of the graphical knowledge model, where ei∗ is an action of the graphical knowledge model in the current state and ej∗ an object of the graphical knowledge model in the current state. [12] The device according to any one of claims 1 to 11, wherein the state is a first state, and wherein the device is configured to detect an event in the workspace based on the sensor signal, and to determine a second state of the dynamic sequence model based on the first state and based on a probability of occurrence of the event in the first or second state. [13] Apparatus according to claim 12, wherein the apparatus is configured to obtain or derive the probability of occurrence from the graphical knowledge model. [14] Apparatus according to any one of claims 1 to 13, wherein the processing means is designed to determine the state of the dynamic sequence model using transition probabilities starting from nodes of the dynamic sequence model. [15] Device according to one of claims 13 to 14, wherein the processing device is designed to adapt the occurrence probabilities and / or the transition probabilities with the aid of recorded sensor data [16] Device according to one of claims 1 to 15, wherein a probability of occurrence of an event (200a-f) is correlated with a remaining or changing state of the dynamic sequence model, and wherein a transition probability describes a probability of a change from a first state to a second state, wherein the processing device is designed to estimate the current state during the workflow based on evaluations of the sensor signal. [17] Device according to one of claims 1 to 16, wherein the processing device is designed to determine the state of the dynamic sequence model based on a temporal averaging of the sensor signals based on a plurality of time-spaced detections of the working area. [18] Device according to one of claims 1 to 17, wherein the sensor device is designed to provide the sensor signal based on a visual detection of the work area. [19] Device according to one of claims 1 to 18, wherein the sensor device is designed to provide the sensor signal based on an acoustic detection of the working area. [20] Device according to one of claims 1 to 19, wherein the sensor device is designed to provide the sensor signal based on a detection of radio frequency identity numbers (RFIDs) of the work area. [21] The device according to any one of claims 1 to 20, wherein the device comprises a user interface configured to receive input from a user that at least partially describes the workflow, and wherein the device is configured to provide the graphical knowledge model based on the user input. [22] Workflow support system comprising: a device according to any one of claims 1 to 21; a work area; and an assistance facility. [23] Procedure (700) with: detecting (710) a work area in which a workflow can be executed, Providing (720) a sensor signal based on the detected working area; Obtaining (730) a description of a graphical knowledge model of the workflow, the graphical knowledge model having a plurality of activity nodes; Creating and parameterizing (740) a dynamic sequence model of the workflow based on the graphical knowledge model of the workflow, Determining (750) a state of the dynamic sequence model based on the sensor signal; Providing (760) a state signal based on the determined state of the dynamic sequence model, Mapping output pins (200a'-f') and objects (200a-f) of activity nodes (210, 215, 220, 225, 230, 235, 240, 245, 250) of the graphical knowledge model to nodes (Xt1,Xt2,…,Xt6) the dynamic sequence model, and Creating nodes of the dynamic sequence model based on activity nodes, based on V B0 = {v e |e = (e i ,e j ) ∈ DF,e i ∈ A,e j ∈ O} U {v r}, where v e a node of the set V B0 that represents an object (e i , ej ) an edge e or path between an object e j and an action e i of the graphical knowledge model, DF is a set of object flows of the graphical knowledge model, A is the set of actions or states of the graphical knowledge model and e i is an action or state, 0 is the set of object nodes of the graphical knowledge model and e j is an object node, v r is a root node, where an object flow connects an action of set A with an object of set 0 in the graphical knowledge model, or based on VB→={ve**|e*=(ei*, ej*)∈DF, ei*∈A, ej*∈0}∪VB0, where V B0 is an original set of nodes of the dynamic sequence model, vr* a root node from a current step, (ei*, ej*) an edge e* or path between an object ej* and an action egg* of the graphical knowledge model, DF is a set of object flows of the graphical knowledge model, A is the set of actions or states of the graphical knowledge model and egg* is an action or state, 0 is the set of object nodes of the graphical knowledge model and ej* is an object node, a star indicates a link to a current state of the graphical knowledge model, where an object flow connects an action of set A with an object of set 0 in the graphical knowledge model. [24] A computer program comprising a program code for carrying out the method according to claim 23 when the program runs on a computer.
Citation Information
Patent Citations
Computer-assisted surgery system and method
US9532848B2
Operating room management system and related methods
US20140067413A1