Generation of sensor data during artificial surgery

Abstract

A computer-implemented method of generating sensor data during an artificial procedure that can be used to train an artificial intelligence (AI) module is presented. The method comprises the steps of: providing a clinical data structure describing a plurality of clinical states of a clinical procedure and defining transitions between the plurality of clinical states, thereby allowing different pathways of the clinical procedure (step S1); providing at least one geometric model for modeling a surgical scene in one or more of said clinical states of said clinical procedure (step S2), said at least one geometric model comprising a plurality of geometric model parameters and boundary conditions of said geometric model parameters; wherein the geometric model parameters describe the surgical scenario in the clinical state of the clinical procedure, and wherein a specific combination of the geometric model parameter values defines a specific surgical scenario for the clinical state of the clinical procedure; the method further comprises the steps of defining a single path of the clinical procedure by selecting at least one transition between a first clinical state and a second clinical state of the clinical data structure (step S3); providing, as input to the geometric model, a set of geometric model parameter values for the at least one geometric model for the first and second clinical states of the defined single path, the first and second single surgical scenarios are modeled (step S4), and during-artificial surgical sensor data for the single path is generated based on the modeled first and second single surgical scenarios (step S5).

Description

TECHNICAL FIELD

[0001] The present invention relates to a computer-implemented method of generating artificial intra-operative sensor data usable for training an artificial intelligence (AI) module, training data for training an AI module for automatically analyzing intra-operative sensor data, a computer program, a program storage medium storing said program or said training data, and to an AI module for automatically analyzing intra-operative sensor data. BACKGROUND

[0002] Data during surgery, e.g. surgical data, in particular video data from real surgical procedures, is rare. For training an AI module or machine learning model that can analyze intra-operative data of real surgeries, a large amount of training data is required.

[0003] In the prior art, it has so far been suggested to use rendered scenes for learning, e.g. instrument segmentation, see e.g. Humic Surgical Platform, https: / / hutom.io / and https: / / hsdb-instrument.github.io / . Other technical fields also make use of artificial data for training AI modules. For example, artificial scenes are used for training AI modules of self-driving cars. In detail, NVIDIA’s Project DRIVE Sim – Synthetic Data Generation Powered by Omniverse Replicator (https: / / www.youtube.com / watch?v=gPaFgNEF82Q) suggests using synthetic 4D ground truth data for training AI modules or deep neural networks that make up the perception system in an autonomous vehicle (AV). DRIVE Sim synthetic data has proven effective in accelerating AV development, allowing developers to customize ground truth data to the specific needs of the model.

[0004] During research and development work in the field of digital operating rooms (ORs), the inventors of the present invention have recognized the need for accurately training an AI module that, after being trained, is to be configured for automatically recognizing or analyzing real clinical procedures by analyzing received intra-operative sensor data.

[0005] It is therefore an object of the present invention to provide an efficient method of providing artificial intra-operative sensor data.

[0006] Aspects, examples and exemplary steps of the present invention and embodiments thereof are disclosed in the following. Different exemplary features of the present invention can be combined according to the present invention as far as technically convenient and feasible.

[0007] The use of technical terms follows their common sense. If certain terms have a specific meaning, the definition of the term will be given below in connection with its context. SUMMARY

[0008] In this section, a description of the general features of the present invention is given, for example by referring to possible embodiments of the invention.

[0009] According to a first aspect of the present invention, a computer-implemented method of generating artificial intra-operative sensor data usable for training an artificial intelligence (AI) module is presented. The method comprises the following steps:

[0010] a) providing a clinical data structure describing a plurality of clinical states of a clinical procedure and defining transitions between the plurality of clinical states, thereby allowing different paths of the clinical procedure (step S1);

[0011] b) providing at least one geometric model for modeling a surgical scene in one or more of the clinical states of the clinical procedure (step S2),

[0012] wherein the at least one geometric model comprises a plurality of geometric model parameters and boundary conditions for the geometric model parameters;

[0013] wherein the geometric model parameters describe the surgical scene in the clinical states of the clinical procedure,

[0014] wherein a specific combination of the geometric model parameter values defines a specific surgical scene of a clinical state of the clinical procedure; the method further comprises the following steps:

[0015] c) defining a single path of the clinical procedure by selecting at least one transition between a first clinical state and a second clinical state of the clinical data structure (step S3);

[0016] d) providing a set of geometric model parameter values of the at least one geometric model for the first and second clinical states of the defined single path as input to the geometric model, thereby modeling a first and a second single surgical scene (step S4), and

[0017] e) generating artificial intra-operative sensor data for the single path based on the modeled first and second single surgical scene (step S5).

[0018] It is to be noted that the artificial intra-operative sensor data generated in step S5 is not generated by real measurements using corresponding one or more real sensors. The artificial intra-operative sensor data of the present application is generated by computation. Thus, the method step S5 can be understood as computing said artificial intra-operative sensor data. Thus, as will be understood by the skilled person, the present application makes use of "artificial intra-operative sensors" in that the method of the present application generates artificial intra-operative sensor data in a computational manner, which would be generated in a real-world scenario by sensors installed in e.g. a hospital operating room. Thus, data from intra-operative sensors is simulated with the method of the present application. This can be understood as modeling or simulating data generated by said sensors placed within e.g. a hospital's operating room. Thus, it should be understood by the skilled person that in step S5, artificial intra-operative sensor data of artificial intra-operative sensors is generated. This will be explained and elucidated in detail below in the context of the specific embodiments.

[0019] Preferably, the modeled or simulated sensor data has the same or substantially the same quality and properties and / or characteristics as the real data, e.g. noise, bandwidth, constraints and distribution.

[0020] As exemplary and non-limiting examples of such intra-operative sensors, the following sensors are mentioned: sound sensors, photo cameras, video cameras, tracking systems for tracking objects within an operating room, in particular electromagnetic tracking systems or infrared tracking systems, marker-based tracking systems, markerless tracking systems, temperature sensors, pressure sensors, depth cameras, lidar, brightness sensors, microphones, sound pressure sensors, x-ray sensors, magnetic field sensors, accelerometers, gyroscopes, ultrasound transducers. Furthermore, intra-operative sensors as used herein can be attached to or in contact with a patient, e.g. to measure vital signs, e.g. heart rate, one or more pulse parameters, blood pressure, temperature, one or more blood parameters (e.g. blood oxygen level, blood sugar, respiratory parameters), one or more tissue electrical properties (e.g. by electrodes implanted or attached on the brain or the skin), neuro-monitoring with one or more hand-held sensors, mechanical motion sensors, force sensors, strain sensors, bladder pressure sensors, rectal pressure sensors, see also https: / / en.wikipedia.org / wiki / Monitoring_(medicine).

[0021] Furthermore, step S3, i.e. defining the single path of the clinical procedure by selecting at least one transition between the first and the second clinical state of the clinical data structure, emphasizes that the method of the present application generates the required sensor data during the artificial surgery along the "path" of the clinical procedure defined by the clinical data structure, in particular when read in conjunction with steps S4 and S5. Thus, the method of the present application does not generate a single, isolated synthetic scenario of a clinical procedure, but rather takes into account transitions between two or more clinical states of a clinical data structure, wherein one scenario develops continuously into another. In other words, such scenarios are related or interlinked. Thus, compared to alternative solutions that generate a single synthetic scenario of a clinical procedure based on real scenarios only, but do not take into account transitions between two clinical states, the data generation according to the present application is done along one path within the clinical data structure.

[0022] It is to be noted that, in principle, the AI module described herein can be embodied as a hardware component / hardware module or a software component / software module. Preferably, the AI module is embodied as a software component / software module, i.e. a computer program. Thus, the AI software module can be trained with data generated with the methods presented herein. This includes training one or more machine learning models using these data. As will become clear from the following disclosure, the AI module disclosed herein can be regarded as a machine learning model.

[0023] An advantage resulting from this approach is that more complex clinical procedures comprising or consisting of different connected or related scenarios can be modeled or simulated. Thus, when using the AI module trained from these generated intra-operative sensor data to identify and / or classify a real clinical procedure monitored by the sensors, not only the scenarios and clinical states can be used to identify / classify the sensed clinical procedure, but also the transitions between the involved clinical states. Thus, also the approach using the transitions between these clinical states will result in a more robust decision. As will be understood by the skilled person, a particular clinical state or scenario only has significance or can be identified if placed in the context of a procedure and / or pathway, i.e. only if it occurs before step A, step B has a particular meaning. Thus, as will be understood by the skilled person, a state usually has a meaning even without context, but for a particular state there can be a difference if another state has occurred before. Furthermore, many different variations of a clinical procedure, i.e. its development over time, can be taken into account by varying the pathway. In this way, more transitions between two clinical states are generated, which cannot be generated based on real data describing a single scenario based on a single “base image”, for example. With the approach of the present application, the model thus has relatively more information available for the training of the AI module, as it not only uses the pure single scenario, but also takes one or more transitions into account. In addition, with the proposed approach rare transitions (e.g. special surgical cases) can be overbalanced, which will also make the solution more robust against edge cases in the real world. In this way, the proposed approach generates data along a pathway and all of said data can be used to describe the entire clinical procedure, i.e. the entire development of said clinical procedure over time. Thus, with the proposed approach, it is facilitated to present transitions / transitions phases between different clinical states of a clinical data structure. Thus, in a particular embodiment, the method comprises presenting at least one transition between a first clinical state and a second clinical state, e.g. generating a time-dependent “video”.

[0024] Furthermore, the “clinical data structure” as used in the present application and as exemplified in one of the embodiments Figure 1 The “clinical data structure” as should be understood in the context of the present application and as exemplified in one of the embodiments should be understood as a description of the possible course of a particular clinical procedure along a plurality of clinical states or nodes (cf. states 1 to 4 in the figure, which can be understood as nodes of said structure). Thus, the provided clinical data structure describes a plurality of clinical states of said clinical procedure and defines transitions between said plurality of clinical states, which in turn allows for different pathways of a clinical procedure. In an exemplary embodiment, the clinical data structure is embodied as a graph, wherein one pathway comprises a plurality of discrete clinical states or nodes of a clinical procedure. In a preferred embodiment, the clinical procedure is a surgical procedure, a diagnostic procedure and / or a therapeutic procedure, as will be detailed in the context of the specific embodiments below.

[0025] The "geometric model" used in the present application is to be understood as a geometric model with variable and / or fixed parameters and boundary conditions for these parameters to ensure reasonability and relevance. In one embodiment, the geometric model allows to model different operational processes of a specific surgical, diagnostic and / or therapeutic procedure. Changing these parameters of the geometric model can be used to change e.g. the scenario of a clinical procedure, e.g. like the movement of objects in the scenario, changing the objects present in such a scenario, their type, shape, position and / or orientation in the operating room. The geometric model can be embodied as a function e.g. y(ax) where the output y is a sensor value which depends on the variable input x.

[0026] With regard to Figure 1 the examples shown in and described in detail below, Figure 1 q1 and q2 shown in and used in are examples of possible parameters which define a state or attribute of a clinical state, x is a variable which will change during the simulation of a single clinical state (e.g. state 1 in ), e.g. time or position in a cycle (e.g. respiratory cycle). Figure 1

[0027] It has to be noted that the method of the present application provides at least one geometric model, but it is also possible to use two, three, four or more geometric models to perform the method presented herein. For example, in one specific embodiment, the same geometric model is used for each state of the clinical data. Thus, only one geometric model is used. In another specific embodiment, a different geometric model is used for each state. Thus, if the clinical data structure has N clinical states, N geometric models are used in this embodiment. As will be understood by the skilled person, any other possibility of using multiple geometric models and less than N geometric models is also an embodiment of the present application. In this case, for some clinical states, the same geometric model is used, while for other clinical states, a specific geometric model is used.

[0028] ​In a preferred embodiment, the geometric models of different clinical states are dependent on each other. For example, many parameters of the two models can be identical to ensure a constant and / or stable transition of one or more sensor values included in the sensor data during the artificial surgery generated with the present application in step S5. In another preferred embodiment, at least one geometric model of a first state used in the present application can have transition attributes that transition to a second geometric model used in the present application method in a second state. In a non-limiting example, ensuring a typical movement of a camera from one position in a first state to another position in a second state can be achieved by using the transition attributes of the geometric model of the first state. In this way, the transition attributes ensure that the parameter value "camera position" does not jump, i.e. there is no discontinuity or instability in the progress over time from the first state to the second state. As is apparent from the present disclosure, the present application can use spatial registration between the first geometric model and the second geometric model in this context. As is understood by the skilled person, it is desirable that the sensor data transitions smoothly between different clinical states. This applies to spatial and non-spatial clinical states. The result of ensuring such a smooth transition is that, for example, a "video" presented along a path is consistent. For example, the color of the objects is the same, the position of the objects does not "jump", the type of the objects remains the same. It should be noted that there are three types of parameters: constant parameters, continuous parameters, and intra-path free parameters. In the following, some examples are provided. A constant parameter can for example be the position of the patient relative to the ground in a clinical procedure and it is known beforehand that the position of the patient relative to the ground does not change at all in this particular clinical procedure. Further, a continuous parameter can for example be that the size of a tumor can only decrease in the course of a tumor resection procedure. With respect to "intra-path free" parameters, note that they can be chosen arbitrarily or randomly at each step and thus the model does not depend on them, e.g. the position of a door in a wall, or the position of a surgical room equipment, or a camera position, lighting. This will be explained in more detail in the context of Figure 3 .

[0029] The "path" of a clinical procedure used in the present application is to be understood as a specific, single sequence of / within a clinical data structure. In other words, in the provided clinical data structure, which describes a plurality of clinical states of this particular clinical procedure and defines transitions between said plurality of clinical states, different paths, i.e. different possibilities of how this clinical procedure can be realized, can be realized. Thus, the skilled person understands a path as one single course of action or development over time. This will be explained in more detail in the context of the non-limiting embodiment shown in Figure 1 , where one possible path is depicted that defines a clinical procedure along the depicted clinical states 1, 2 and 3 and not along the depicted clinical states 1, 2 and 3. Figure 1The development of clinical state 4 of the illustrated clinical procedure. Note, however, that the path can extend in both directions of the transition between two states, i.e. back and forth, Figure 1 One possible path in the illustrative example of a clinical procedure of Fig. 1 can be: from clinical state 1 to state 2 to state 3, back to state 2, and then to state 4. Other variations are possible. As will be clear to the skilled person, a clinical procedure has a generally number of steps or step repetitions. Typically, this number has to be finite; a clinical state can have a maximum number of occurrences, a minimum number of occurrences, and / or a probability of occurrence. Thus, the term "path" as used herein is to be understood as a specific, single sequence of clinical states in the clinical data structure.

[0030] A "clinical procedure" as used in the present invention and as will be clear to the skilled person is a general description of a kind of medical process or operation. Illustrative examples of different clinical procedures are e.g. hip implantation, heart valve implantation, insertion of a DBS (deep brain stimulation) electrode at a patient's skull, and appendectomy. Thus, it is to be understood as a "type", "category" or generic type of surgical, diagnostic and / or therapeutic procedure described by the provided clinical data structure, and for which artificial sensor data is desired to be obtained. Once such artificial sensor data is generated with respect to said clinical procedure using the method presented herein, said data can be used to train an AI module. This in turn facilitates that said trained AI module can automatically analyze real intraoperative sensor data collected during such real clinical procedure. In this way, the trained AI module can e.g. automatically identify the type of clinical procedure being performed in an operating room monitored by one or more sensors. Real sensor data of such sensors is thus provided to the AI module for said analysis. Thus, the term "clinical procedure" as used herein is to be understood in the context of the present invention as a general description of a type of medical operation simulated using a geometric model. As mentioned above, in a preferred embodiment, the clinical procedure is a surgical procedure, a diagnostic procedure and / or a therapeutic procedure. It is apparent from the present disclosure that the created artificial intraoperative sensor data at least partially describes the clinical procedure.

[0031] The term "clinical state" as used herein is to be understood as a clinically distinguishable state, which thus can be distinguished from another clinical state of the associated clinical procedure. It is thus a state that would be recognized by a clinician. Any clinical state used herein can have its own "state label", which can be displayed and / or reported to a user.

[0032] The term "surgical scene" or "scene" as used herein is to be understood as a specific intraoperative scene or single intraoperative situation which can be modeled using the geometric model suggested herein. A scene "occurs" or is associated with one clinical state of a clinical procedure described or defined by the clinical data structure suggested herein. Furthermore, the geometric model used in the present invention comprises a plurality of geometric model parameters and boundary conditions for said geometric model parameters, respectively. Thus, when each parameter is defined with one value, one scene is defined or individualized. For example, in the illustrative example of Figure 1 a "surgical scene" or "scene" of clinical state 1 is defined or determined when both parameters q1 and q2 are defined or determined. In Figure 1 a specific individualized scene of clinical state 1 is shown, wherein q1 is set to 0,9 and q2 is set to 0,7. q1 also fulfills the pre-defined boundary condition, wherein the minimum value of q1 is set to q 1,Min = 0,1 and the maximum value of q1 is set to q 1,Max = 14. Thus, with the sensor values q1 = 0,9 and q2 = 0,7, a surgical scene of clinical state 1 is defined. For example, the parameter q1 is "the position of the surgeon in the operating room coordinate system" and the parameter q2 is "the position of the tracked hip implant in the operating room coordinate system". As understood by the skilled person, the same applies for the surgical scene defined by the exemplary chosen parameter values, i.e. the exemplary chosen sensor values, as shown in Figure 1 for clinical states 2, 3 and 4.

[0033] As intraoperative data from real surgical procedures, like surgical data, especially video data, are rare, the method of the present invention provides a beneficial method of generating artificial intraoperative sensor data. This generated artificial intraoperative sensor data can be used to train AI modules and machine learning models which will analyze real intraoperative data. According to the method of the present invention, this data can be generated with the geometric model / parameter model which changes the scene or using the generated or extracted procedure of said artificial data generation. As apparent from the present disclosure, the method of the present invention can also use any additional data from other devices, for example data describing the type, shape, reflective properties of surgical instruments typically used in a specific surgical procedure.

[0034] However, the artificial intra-operative sensor data can be used for more purposes. For example, it can also be used to test an AI module that is configured to solve a given task on the intra-operative sensor data and that has been trained in any way to solve that task. The artificial intra-operative sensor data can also be used to test any technical system that it is configured to monitor the intra-operative sensor data and take some measures based on that intra-operative sensor data. For example, such a monitoring system can analyze the intra-operative sensor data for any anomalies and output an alarm if an anomaly is found. The artificial intra-operative sensor data can even be used for plausibility verification of real sensor data, i.e. to determine whether the intra-operative sensor data recorded during a real surgery is likely to be correct or whether the sensors that recorded that data have obviously failed.

[0035] Thus, the artificial intra-operative sensor data has many technical applications besides training an AI model for recognizing a surgical procedure, and is thus not limited to this application scenario.

[0036] Furthermore, independent of the specific application scenario, the proposed method has the technical effect of transforming the problem of generating (preferably physically valid) artificial intra-operative sensor data into a form that can be efficiently processed by a computer. The method is based on the technical consideration of which operations a computer can perform most efficiently, in particular by means of a GPU or other accelerator that is capable of efficiently processing geometric models. Thus, the technical effect is achieved regardless of what the generated artificial intra-operative sensor data is ultimately used for.

[0037] The provided clinical data structures and geometric models can ensure the plausibility and relevance of the clinical procedure scenarios that they describe. The use of input from e.g. experienced medical personnel like surgeons, the use of medical knowledge from medical databases and / or the use of suggestions of plausible scenarios of clinical procedures suggested by AI modules are part of specific implementations in which the clinical data structures and / or the geometric models are defined or computed. In other words, the geometric models can be modeled manually, e.g. following clinical guidelines or standard operating procedures. Or, the geometric models can be built automatically based on sensor data and / or based on machine learning, e.g. using hidden Markov models. Or, the geometric models can be built based on written or structured reports.

[0038] As will be explained in more details hereafter, at least one geometric model of the clinical procedure has parameters and boundary conditions for one or more of these parameters to ensure reasonableness and relevance. To create the required artificial sensor data, these parameters describing the scene are varied. For example, the movements of the objects in the scene and / or the types of the objects, their shapes and their positions are varied. In an exemplary implementation, the diameter of the screw used in the clinical procedure is varied. Other exemplary parameters of the geometric model that can be varied when performing the method of the application are the patient anatomy, the current medical staff, the background and / or the surrounding environment, the light parameters in the operating room, the detectable sounds in the operating room, the temperature in the operating room, etc.

[0039] As will be clear to the skilled person, different rendering conditions (e.g. artifacts, texture variations, lighting variations, camera position variations, viewing angles and presence of textures of objects in the operating room) are also examples of parameters that can be used by the geometric model of the application.

[0040] Thus, in the method of the application, based on different clinical states of the geometric model (i.e. sets of parameters), artificial intra-operative sensor data related to the modeled clinical procedure is created by varying the scene, e.g. including the movements of the objects in the scene (e.g. their types, shapes and positions), by varying the progress of the clinical procedure along a plurality of surgical states, and by varying the mentioned rendering conditions (e.g. artifacts, texture variations, lighting, camera position, textures of objects, etc.).

[0041] The method can include 2D or 3D rendering of the plausible video. In a first category, highly realistic renderings can be used that even an expert cannot distinguish from reality and that can involve style transfer. In a second category, high quality renderings based on physical models (e.g. light tracing models of materials and surfaces) can be used. In a third category, standard renderings can be used, i.e. geometry-based renderings using textures with heuristic lighting models.

[0042] The generation of sensor data during artificial surgery can also include modeling any other sensor data, such as tracking data or 3D surface point clouds from physical models or x-rays or ultrasound during surgery. The generation of data can be based on a machine learning model that is trained using real data, e.g. as DAL-E or DAL-E mini works, possibly using domain randomization to avoid training in the background. The model can also be referred to as a “generative adversarial network (GAN)” or “generative AI module”. It is noted that in embodiments of the invention also includes the use of training data from rare cases, also referred to as edge cases, in the context of other cases that are not part of the training data but are modeled. As will be clear to the skilled person, edge cases are important for use so that the product performs well in extreme situations.

[0043] As will be described in more detail below, annotations and / or labels can be generated from the model parameters used to generate artificial data by the method of the invention. The annotations and / or labels can be stored together with the generated artificial intraoperative sensor data. The annotations and / or labels used herein can be embodied in instances as pixel-level layers describing classes, metadata describing bounding boxes and / or labels describing images, scenes or entire videos.

[0044] With the artificial intraoperative sensor data thus generated, the data can be used to train AI modules / machine learning algorithms to later segment, label or annotate real intraoperative sensor data, i.e. the same or similar type of data, in space or time.

[0045] In the following, exemplary application scenarios are presented, i.e. how the possibilities of using AI modules trained with data generated by the method of the invention. Such AI modules are configured for automatically analyzing real intraoperative sensor data. Based on the analysis results, the following method steps described as application scenarios can be performed. Exemplary application scenarios of the method of the invention are the automatic creation of a surgical procedure report; the calculation of the remaining surgical procedure duration; the issuance of a warning; the control of equipment or instruments; the control of other hospital workflows, such as OR setup or patient transfer; the rescheduling of a surgery; the instrument and disposable check, tracking, counting; the recommendation of the next surgical / diagnostic / treatment step; the recommendation of the next surgical / diagnostic / treatment instrument; the general real-time, on-site or postoperative use of data; and the performance of a skill assessment or performance assessment.

[0046] Accordingly, the method presented herein can involve the following steps. First, a plurality of geometric model parameters and boundary conditions for the geometric model parameters are determined or specified. Second, the remaining, undetermined, i.e. free, parameters of the geometric model are displaced. As will be appreciated by the skilled person, this is a sampling of the parameter space. The results of these displacements / samplings can then be presented to provide, for example, a video simulating / modeling a clinical procedure.

[0047] It should be noted that some clinical states will look the same in sensor data, but are in fact different, which can be identified based on a history / status history, i.e. based on a list of states that have occurred previously. In other words, this means that the state in a procedure is best described by knowing not only all sensor data, but also the states that have occurred previously.

[0048] According to another exemplary embodiment of the present invention, the method further comprises the steps of:

[0049] storing the generated intraoperative sensor data of the single path together with the set of geometric model parameter values on which the data is based, wherein the set of geometric model parameter values on which the data is based is stored as a training label; and / or

[0050] storing the generated intraoperative sensor data of the single path together with the expected outcome that the AI module trained on the data should produce, wherein the expected outcome that the AI module trained on the data should produce is stored as a training label.

[0051] This embodiment requires that the generated intraoperative sensor data is stored together with the parameters of the geometric model or the expected outcome that the model trained on the data should produce, i.e. a training label, on which the data is generated. This data and parameters can be stored on the program storage medium of the present invention. As will be clear to the skilled person, this embodiment limits the generated intraoperative sensor data to data of “the single path”, thereby emphasizing that the method of the present invention creates data “along a path”. As detailed above, this distinguishes the present invention from alternative solutions that generate a single synthetic scenario of a clinical procedure based on a real scenario only, without considering the transition between two clinical states. The data according to the present invention is generated along a path within a clinical data structure.

[0052] According to another exemplary embodiment of the present invention, each clinical state of the clinical data structure comprises a label, wherein each label describes the respective clinical state as a characteristic phase of the clinical procedure and in a clinically distinguishable manner.

[0053] In this embodiment, labels of the clinical states described by the clinical data structure are detailed. In order to unambiguously distinguish the clinical states from each other, the labels characterize the respective clinical state such that a clinician can clearly see it as a state different from the others. Thus, in this embodiment, any clinical state has its own "state label" which can be displayed and / or reported to the user. In embodiments, the labels are displayed and / or reported to the user via a user interface.

[0054] With regard to the purpose and advantages of displaying / reporting labels to the user, the following should be noted. The labels have a clinical meaning, i.e. the user will gain additional information from it, or pay more attention or make comparisons. The states can also be defined by a set of organs, instruments or medical devices that play a role in this state, e.g. are moved, or touched or present. Typically, for a type of procedure, the medical literature will provide a set of states or steps, see e.g. the set of states defined in the paper mentioned on GitHub: https: / / github.com / CAMMA-public / Surgical-Phase-Recognition.

[0055] According to another exemplary embodiment, the clinical data structure is embodied as a directed and / or weighted graph, preferably with conditions on the directed edges or (pre) conditions on the nodes, defining what conditions have to be fulfilled in order to enter them. This embodiment more explicitly models which transitions between states are possible, preferably with weights defining the likelihood of this transition to occur.

[0056] This embodiment also allows to define whether backward transitions are possible (as used in the example), thereby defining whether a cycle can be made. For some procedures, states can be repeated, for others not, e.g. in the case of a tumor resection, it does not make sense or is not possible to return to the "prepare resection" state after the "tumor resection" state is completed.

[0057] According to another exemplary embodiment of the present application, for a first clinical state of the single path, a first geometric model is provided, and wherein for a second clinical state of the single path, a second geometric model is provided, and

[0058] wherein the first and second geometric models are different from each other, and / or wherein at least the boundary conditions of the first and second models are different.

[0059] In this implementation, at least two different models are required for the two different states. Using multiple geometric models provides users with greater flexibility to describe the required clinical procedures in detail using models. This implementation makes the two clinical states and two geometric models clear. As those skilled in the art will understand, more geometric models, or in particular all geometric models, can be different for all clinical states. In another implementation, not only are the boundary conditions of the first and second models different, but also the parameters of one, more, or all of the geometric models included by the first and second geometric models can be different.

[0060] According to another exemplary embodiment of the present invention, the second geometric model is spatially registered into the first geometric model.

[0061] In other words, for cases where the first and second geometric models are adjacent to each other (i.e., "nearby") within or along the path of the clinical procedure, such as in... Figure 1 In states 1 and 2, or states 2 and 3, or states 3 and 4, spatial registration provides a spatial link between the first and second models. This spatial registration / link between models ensures a smooth transition of the generated sensor data between adjacent states. Regarding the spatial registration of two geometric models, it is important to note that "registration" is limited to spatial descriptions, such as the positions of objects in state 1 and state 2 being continuous or identical. For color and other non-spatial attributes, this is defined only by "identical" or "continuous," not by registration.

[0062] According to another exemplary embodiment of the present invention, a second geometric model space is registered to the first geometric model and configured to ensure that first sensor data generated for the first clinical state from at least the first sensor is continuously transformed to second sensor data generated for the second clinical state from at least the first sensor.

[0063] Based on the foregoing implementation, this implementation clarifies that the required smooth transition applies to all adjacent states, where the corresponding models use or share at least one sensor of the same kind. Generally, for sensors sensing or measuring spatial parameters and sensors sensing or measuring non-spatial parameters, sensor data should transition smoothly between adjacent or neighboring clinical states. In this way, the following results can be achieved: for example, the video presented along the path describing the clinical data structure of the clinical procedure is consistent; for example, an object has the same color; the position of an object does not "jump," and the type of the object remains the same. In this context, it is reiterated that there are, in principle, three different parameter types: constant parameters, continuous parameters, and free parameters within the path.

[0064] According to another exemplary embodiment of the present application, for all geometric models of all adjacent clinical states in the clinical data structure, the spatial registration of the respective geometric model is configured to ensure the continuous transition of the generated sensor data of the adjacent clinical states for the same type of intra-operative sensor generating intra-operative sensor data.

[0065] In other words, this embodiment ensures that for each clinical state having an adjacent clinical state, wherein one or more same type of intra-operative sensors are used, a spatial registration between the respective geometric models of the two adjacent clinical states is provided. For example, in Figure 1 In the illustrated non-limiting embodiment, in both adjacent clinical states 1 and 2, the parameter q1 is sensed or measured by a respective intra-operative sensor. Similarly, in Figure 1 In the illustrated non-limiting embodiment, in both adjacent clinical states 2 and 4, the parameter r2 is sensed or measured by a respective intra-operative sensor. Thus, according to the above described embodiment, Figure 1 The geometric model of state 2 in is preferably spatially registered in terms of the parameter q1 into the geometric model of state 1 in Figure 1 Similarly, in the preferred embodiment, the geometric model of state 4 in is preferably spatially registered in terms of the parameter r2 into the geometric model of state 2. Figure 1 Similarly, in the preferred embodiment, the geometric model of state 4 in is preferably spatially registered in terms of the parameter r2 into the geometric model of state 2.

[0066] Thus, the data recorded / sensed by the same sensor has a smooth transition between adjacent clinical states.

[0067] According to another exemplary embodiment of the present application, the generated intra-operative sensor data comprises at least tracking data and image data; in particular tracking data and image data of a tracked intra-operative camera.

[0068] The tracking information can provide spatial information about placed objects in the operating room without any input from video or images. With this embodiment, it is possible to know exactly where the tools, the patient and / or the camera(s) are placed, even if these objects can not be visible to the camera at a certain moment. This embodiment also allows to use the tracked microscope video to know exactly where the focus of this camera is, even if the microscope video itself will be discarded completely. This embodiment relates to the generation of image data and tracking data. In contrast to alternative solutions using a "ground truth", the present invention uses in this embodiment the tracked position of the objects (preferably the tracked center position) to construct the scene. One big advantage is that with this method it is possible to simulate the position of the equipment and to know the position in the real world scene. So, for example, the exact position of the microscope relative to the patient, this can also help to identify specific procedural steps. It is relatively disadvantageous if only the video is available without any position data about the objects and the camera.

[0069] According to another exemplary embodiment of the present invention, artificial intra-operative sensor data is also generated, which is not used for and / or does not relate to the surgical scene of the clinical state of the clinical procedure.

[0070] In this embodiment, all sensor data is generated, i.e. also those sensors for any geometric model of any clinical state of the clinical procedure, which are silent (i.e. do not relate to this specific clinical state). For example, in Figure 1 In a non-limiting example, this would mean that when creating the artificial intra-operative data for clinical state 1, the sensor data for parameters r1 and r2 (used in clinical state 2, but not in clinical state 1), p1 and p2 (used in clinical state 3, but not in clinical state 1) and t1 and t2 (used in clinical state 4, but not in clinical state 1) would also create a value of 0, as they are not used in clinical state 1. However, sensor values of 0 can be advantageous and useful when training the AI module. For example, in case an endoscope is not used during the preparation of a specific clinical procedure, this can still be relevant data for training purposes. In other words, sensor data originating from any "unused" sensor, i.e. originating from those sensors which are not involved in the clinical procedure during this clinical state, is also presented.

[0071] According to another exemplary embodiment of the present invention, the generation of artificial intra-operative sensor data is performed in step S5 by presenting image data of intra-operative image sensors and / or intra-operative video sensors.

[0072] In this embodiment, the data of which the artificial intra-operative sensors are embodied in step S5 of the method described herein are intra-operative image sensors, intra-operative cameras, intra-operative video sensors and intra-operative video cameras.

[0073] According to another exemplary embodiment of the present application, the method further comprises the steps of repeating steps S3 and S4 for a plurality of clinical states and for a plurality of sets of said geometric model parameter values, and repeating step S5, thereby generating a plurality of artificial intra-operative sensor data for a plurality of different paths of said clinical data structure and for a plurality of surgical scenarios (step S6).

[0074] This embodiment describes that steps S3 to S5 can be iterated multiple times in order to generate complete artificial intra-operative sensor data not only along one path, but along multiple, preferably all possible paths allowed by the clinical data structure. In an example, all possible artificial intra-operative sensor data for all possible surgical scenarios of clinical states 1, 2 and 3 of the shown path will be created in a first iteration. Figure 1 In a second iteration, for example, the path starts from clinical state 1, goes to state 2, to state 3, back to state 2 and then to state 4.

[0075] According to another exemplary embodiment of the present application, artificial intra-operative sensor data is generated for all clinical states of said defined single path of said clinical procedure.

[0076] In an alternative of this embodiment, the method of the present application can also use real data introduced at one or more clinical states. Thus, according to this embodiment, a mix of artificial intra-operative sensor data and real data can be used.

[0077] According to another exemplary embodiment of the present application, the artificial intra-operative sensor data generated for all clinical states of said defined single path of said clinical procedure is provided in chronological order, such that one or more videos of said clinical procedure are provided.

[0078] In this embodiment, one or more videos are generated per single path. Since the method of the present application can generate data for multiple paths, also multiple videos can be generated (one or more per path). Each such video can then be used as training data for an AI module, as explained in more detail above and below.

[0079] According to another exemplary embodiment of the present application, all possible surgical scenarios under each clinical state are modelled.

[0080] According to another exemplary embodiment, the geometry parameters provided in step S4 are chosen such that the boundary conditions of the geometry model are respected to generate only physically valid configurations / cases, i.e. physically valid surgical scenarios.

[0081] However, generally, as will be clear to the skilled person, it is desirable that the AI module trained with the data generated by the method presented herein is robust. To increase robustness, during the training of the AI module with the data generated with any of the methods presented herein, the attention of the AI module can be distracted by providing as diverse cases as possible, i.e. surgical scenarios, including non-physically cases, i.e. physically invalid cases, and irrelevant cases.

[0082] A non-limiting example of a physically invalid case can for example be a surgical knife hovering in the operating room, typically placed on a table or similar, or held by the surgeon’s hand or by a robotic surgical device. This can be considered as physically invalid. However, it can be helpful to consider such a configuration, case or surgical scenario, as it can reflect a situation where the surgeon loses control of the knife, which then falls to the floor at that time.

[0083] Furthermore, in an embodiment, the AI module should not be trained on irrelevant features in the data, e.g. background color or irrelevant clinical objects. Some parts of the geometry model can be labeled as non-physical, which also depends on the clinical state. For example, if the patient is present or not is particularly relevant to the “surgery” state, then everything else preferably should change.

[0084] According to another exemplary embodiment of the present application, in step S4, the values of the geometry parameters are provided by a random generator, or by a mathematical function that selects the values of the geometry parameters.

[0085] This embodiment details how the geometry parameters can be chosen, and describes how the parameters can be arranged. The sampling strategy of the objects is all of this follows the boundary conditions of the geometry model to generate only physically valid configurations. In one example, all possible combinations of valid parameter values are used, and / or sampling from the parameter space (Monte Carlo), and / or using a machine learning model trained on real observations. The sampling strategy can weight configurations that are underrepresented in real data. This knowledge can be provided by the user and then used by the method of the present application. In a particular embodiment, the method uses the probabilities of transitions between clinical states / nodes. Thus, the selection of the path can be based on the probabilities (e.g. directly on the probabilities or using the inverse of the probabilities).

[0086] However, generally, there are multiple exemplary options for how the transition is chosen. First, the clinical probabilities can be used as probabilities, or the transitions can be chosen randomly, i.e. based on different distributions. In another embodiment, predetermined probabilities for the transitions can be used, which are not based on clinical probabilities but on clinical relevance / importance, i.e. if one wants to robustly detect edge cases, an over-representation of rare but important transitions can be used. In another embodiment, the choice of transition can be made based on the history of the states, e.g. a large incision implies a large surgery. In other words, certain states only make sense if certain other states pre-exist and this fact can be used when choosing the transition. In this way, impossible state transitions can be excluded. In another embodiment, the transition can be chosen based on state complexity or state type. In another embodiment, an AI-based approach can be used to choose the transition to find the best distribution in order to then train the most robust model. In this way, it can be checked how the model performs on a specific distribution and this distribution is automatically adjusted until the model with the best performance is found, e.g. on real scenarios including edge cases etc.

[0087] According to another exemplary embodiment of the present application, the sensor data is video data during surgery and the step S5 of generating artificial sensor data comprises rendering an artificial video of the clinical procedure.

[0088] According to another exemplary embodiment of the present application, the method further comprises the steps of:

[0089] g) using the generated artificial intra-operative sensor data and the set of geometric model parameter values used during generation of the artificial intra-operative sensor data, and preferably also using the labels derived therefrom and explained herein, to train an AI module that automatically analyzes real intra-operative sensor data.

[0090] This embodiment makes explicit the purpose of the artificial data generation, namely to train the AI module. It is to be noted that most parameters are likely to have no impact on the labels, or even on the generated sensor data (e.g. consider a change of a parameter of the geometric model at a location where the sensor is occluded).

[0091] According to another exemplary embodiment of the present application, the method further comprises the steps of:

[0092] h) providing real intra-operative sensor data describing a real clinical procedure to the trained AI module trained with the generated artificial intra-operative sensor data, and

[0093] i) automatically identifying the real clinical procedure by the trained AI module.

[0094] According to another exemplary embodiment of the present application, wherein each transition of the clinical data structure comprises a weight associated with a probability or frequency of occurrence in the clinical procedure.

[0095] According to another exemplary embodiment of the present application, the clinical procedure is a surgical procedure, a diagnostic procedure and / or a therapeutic procedure.

[0096] According to another exemplary embodiment of the present application, for at least two clinical states, preferably for all clinical states, the respective geometric model, the respective geometric model parameters and the respective boundary conditions depend on the clinical state.

[0097] According to another exemplary embodiment of the present application, the provided geometric model is configured for spatially modeling a surgical scene in one or more of the clinical states of the clinical procedure. According to another exemplary embodiment of the present application, wherein the geometric model parameters define at least a position, a shape and / or a type of one or more objects occurring in the surgical scene of the clinical state of the clinical procedure.

[0098] Unlike other embodiments, wherein also parameters of e.g. operating room lighting and / or object colors are used, this embodiment makes use of spatial parameters position, shape and / or type of one or more objects occurring in the surgical scene. It is to be noted that the type of objects can be taken from a variant database, e.g. also containing variants not yet observed in clinical data, e.g. new devices not yet published, e.g. new endoscopes. Thus, the method of the present application can make use of data available in such medical or surgical databases. It is also to be noted that the method of the present application does not require ground truth images from real data. Advantageously, the method of the present application can render scenes / situations without ground truth, i.e. without real images being present.

[0099] According to a second aspect of the present application, a program is presented which, when running on a computer or when loaded onto a computer, causes the computer to perform the method steps of the method according to any of the preceding claims.

[0100] The program should be understood to be a computer program element. It can be a part of a larger computer program, but it can also be an entire computer program in and of itself. For example, the program can be implemented as a computer program product that is tangibly embodied in a machine-readable storage medium, for execution by a processing means associated with a computer. The program can be implemented in a centralized fashion in one computer system or it can be implemented in a distributed fashion where different elements are spread across multiple computer systems. Any kind of computer system - or other apparatus - adapted for carrying out the described functionality can be used for

[0101] According to a third aspect of the present application, a training data for training an AI module for automatically analyzing sensor data during real surgery is presented, wherein the training data is generated according to the method of any of the aspects or embodiments presented herein.

[0102] According to a fourth aspect of the present application, a program storage medium and / or a download product is presented, on which the program and / or the training data is stored.

[0103] The program storage medium should be understood to be a computer readable medium. It can be a storage medium, such as a U disc, a CD, a DVD, a data storage device, a hard disk or any other medium that can store the program elements described above. The program of the present application can be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state storage medium supplied together with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. In particular, a product that is distributed online and that is implemented immediately upon being downloaded is referred to as a download product.

[0104] In this third aspect, the present application relates to a non-transitory computer readable program storage medium having stored thereon the program according to the second aspect and / or the training data according to the third aspect.

[0105] According to a fifth aspect of the present application, an AI module for automatically analyzing sensor data during real surgery is presented, wherein the AI module is trained with the training data according to the third aspect.

[0106] In a preferred embodiment, the AI module of the present application is selected from the group consisting of a generative adversarial network, a convolutional neural network and a neural network.

[0107] According to an exemplary embodiment, the AI module is configured to receive real surgery during sensor data describing a real clinical procedure as input, and wherein the AI module is configured to automatically identify the real clinical procedure by analyzing the received real surgery during sensor data.

[0108] For example, the present application does not involve or specifically include or encompass invasive steps that cause substantial physical disturbance to the body, require specialized medical expertise to perform, and carry significant health risks even when performed with the required specialized care and expertise. For example, the present application does not include steps of positioning a medical implant to secure it to an anatomical structure, or securing a medical implant to an anatomical structure, or preparing an anatomical structure for securing a medical implant to it. More specifically, the present application does not involve or specifically include or encompass any surgical, diagnostic or therapeutic activity. For this reason alone, no surgical, diagnostic or therapeutic activity, in particular no surgical, diagnostic or therapeutic step, is required or implied by the practice of the present application. Definitions

[0109] In this section, definitions of specific terms used in the present disclosure are provided, which also constitute part of the present disclosure. Artificial intelligence module

[0110] In the context of the present invention, the term "artificial intelligence module" shall include machine learning models. Thus, the method of the present invention of generating sensor data during a surgical procedure can be used to train an artificial intelligence (AI) module, i.e. to train a machine learning model. The terms artificial intelligence module and machine learning model are used synonymously herein. In particular, the AI module / machine learning model can be a model that contains a parameterized function with trainable parameters and that has a high generalization capability. The training of the parameters is usually done using training instances. This training can be done in a "supervised" way by comparing the output generated by the training instance with a "ground truth" that was used to "label" the respective training instance and deriving from this comparison a feedback about an adjustment of the trainable parameters. The training can also be done in an "unsupervised" way if it is not necessary to use a "ground truth" to determine whether the output of the AI module / machine learning model is good or bad. The training can also be done in a "semi-supervised" way by individual errors or inadequate outputs provided by the AI module / machine learning model that are corrected by an engineer. After training using a sufficiently large and sufficiently high-variability set of AI training instances, it can be expected that the model / machine learning model can also provide adequate outputs for inputs that were not seen during training. That is, the AI module / machine learning model can generalize to these unseen inputs. This generalization capability is particularly pronounced for inputs in the domain and / or distribution of the training instances. For example, an AI module / machine learning model trained on images of surgical procedures can be expected to generalize to images of unseen surgical procedures, but not to images of road traffic situations.

[0111] An artificial intelligence (AI) module is an entity that processes one or more inputs into one or more outputs by an internal processing chain that usually has a set of free parameters. The internal processing chain can be organized into layers that are connected to each other and are traversed consecutively in the process from input to output.

[0112] Many artificial intelligence modules are organized to process high-dimensional inputs into lower-dimensional outputs. For example, an HD image with a resolution of 1920 x 1080 pixels exists in a space with 1920 x 1080 = 2,073,600 dimensions. A common task of an artificial intelligence module is to classify an image into one or more classes based on, for example, whether the image contains certain objects or not. The output can then, for example, give for each object to be detected a probability that the object is present in the input image. This output exists in a space with as many dimensions as there are objects to be detected. Typically, there are about a few hundred or a few thousand objects to be detected.

[0113] Such a module is called "intelligent" because it can be "trained". The module can be trained using records of training data. A record of training data comprises training input data and corresponding training output data. The training output data of a training data record is the result that the module is expected to produce when given the training input data of the same training data record as input. The deviation between this expected result and the actual result produced by the module is observed and evaluated by means of a "loss function". This loss function is used as feedback for adjusting the parameters of the internal processing chain of the module. For example, the parameters can be adjusted with the optimization goal of minimizing the value of the loss function that is produced when all training input data is input into the module and the results are compared to the corresponding training output data.

[0114] As a result of this training, given a relatively small amount of training data records as "ground truth", the module is able to perform its work, e.g. classify images to determine which objects are contained therein, even if the number of input data records is orders of magnitude higher. For example, a set of about 100,000 training images, and which have been "labeled" with ground truth of which objects are present in each image, can be sufficient to train a module so that it can then recognize these objects in all possible input images, which can for example be more than 530 million, with a resolution of 1920x1080 pixels and a color depth of 8 bits.

[0115] It is noted that, in principle, the AI module described herein can be embodied as a hardware component / hardware module or a software component / software module. Preferably, the AI module is embodied as a software component / software module, i.e. a computer program. Thus, the AI software module can be trained with data generated with the methods presented herein. This includes training a machine learning model using these data. As mentioned above, the AI module disclosed herein can be regarded as a machine learning model. Neural networks

[0116] A neural network is a prime example of an internal processing chain of an artificial intelligence module. It consists of multiple layers, wherein each layer comprises one or more neurons. The neurons between adjacent layers are linked, the output of a first layer neuron being the input of one or more neurons in an adjacent second layer. Each such link is assigned a "weight" with which the respective input is entered into an "activation function" which gives the output of the neuron as a function of its input. The activation function is typically a non-linear function of its input. For example, the activation function can comprise a "pre-activation function" which is a linear function of its input, and a threshold function or other non-linear function which produces the final output of the neuron as a function of the value of the pre-activation function. Convolutional neural networks

[0117] A convolutional neural network is a neural network that comprises "convolutional layers". In a "convolutional layer", the output of a neuron is obtained by applying a convolution kernel to the input of these neurons. This greatly reduces the dimensionality of the data. Convolutional neural networks are often used for image processing. Generative adversarial network

[0118] A generative adversarial network is a combination of two neural networks, called "generator" and "discriminator". Such a network is used to artificially produce data records that are indistinguishable from records taken from a given set of training data records. The training objective of the generator network is to create, from input records containing random data, output records that are indistinguishable from records in the set of training data. That is, given only this output record, it is impossible to tell whether it was produced by the generator or contained in the set of training records. The discriminator, in turn, is specifically trained to classify given data records as either "real" training records or "fake" records generated by the generator. Thus, the generator and the discriminator compete with each other.

[0119] For example, a generative adversarial network can be used to create realistic images that are indistinguishable from a set of training images. From a limited number of training images (e.g. obtained by medical imaging), an almost unlimited number of fake images can be generated that can be mistaken for medical images. Its main application is to produce training data for other artificial intelligence modules (e.g. modules to be trained to classify whether certain features or objects are present in a medical image). Computer-implemented method

[0120] The method according to the present application is for example a computer-implemented method. For example, all steps or only part of the steps (i.e. less than the total number of steps) of the method according to the present application can be performed by a computer (e.g. at least one computer). An embodiment of a computer-implemented method is the use of a computer to carry out a data processing method. An embodiment of a computer-implemented method is a method in respect of which a computer operates such that the computer operates to carry out one, more or all steps of the method.

[0121] For example, the computer comprises at least one processor and, for example, at least one memory, in order to (technically) process data, for example, electronically and / or optically. The processor is, for example, made of a semiconductor substance or composition, for example, at least partially n- and / or p-doped semiconductor, for example, at least one of a group II, III, IV, V, VI semiconductor material, for example, (doped) silicon and / or gallium arsenide. The described computing or determining steps are, for example, carried out by a computer. The determining step or the computing step is, for example, a step of determining data within the framework of a technical method, for example, within the framework of a program. The computer is, for example, any kind of data processing device, for example, an electronic data processing device. The computer can be a device that is generally considered a computer, for example, a desktop PC, a laptop, a netbook, etc., but also any programmable apparatus, for example, a mobile phone or an embedded processor. The computer can, for example, comprise a system (network) of “sub-computers”, wherein each sub-computer represents a computer itself. The term “computer” comprises a cloud computer, for example, a cloud server. The term “cloud computer” comprises a cloud computer system, for example, a system comprising at least one cloud computer and, for example, a plurality of operatively interconnected cloud computers, for example, a server farm. Such a cloud computer is preferably connected to a wide area network, for example, the World Wide Web (WWW), and is located in a so-called computer cloud, which consists of all computers that are connected to the World Wide Web. This infrastructure is used for “cloud computing”, which describes computing, software, data access and storage services that do not require end-user knowledge of the physical location and / or configuration of the computer that provides a particular service. For example, the term “cloud” is used in this respect as a metaphor for the Internet (World Wide Web). For example, the cloud provides computing infrastructure as a service (IaaS). The cloud computer can act as a virtual host for an operating system and / or a data processing application for carrying out the method of the present application. The cloud computer is, for example, the Elastic Computing Cloud (EC2) provided by Amazon Web Services™. For example, the computer comprises an interface for receiving or outputting data and / or for carrying out an analog-digital conversion. The data is, for example, data representing physical properties and / or generated from a technical signal. The technical signal is, for example, generated by a (technical) detection device, for example, a device for detecting a marker device, and / or a (technical) analysis device, for example, a device for carrying out a (medical) imaging method, wherein the technical signal is, for example, an electrical or optical signal. The technical signal is, for example, representative of the data received or output by the computer. The computer is preferably operatively coupled to a display device that allows, for example, information output by the computer to be displayed to a user. One example of a display device is a virtual reality device or an augmented reality device (also called virtual reality glasses or augmented reality glasses), which can be used as “goggles” for navigation. A specific example of such augmented reality glasses is Google Glass (trademark of Google Inc.).An augmented reality device or a virtual reality device can be used both for inputting information into a computer by user interaction and for displaying information output by the computer. Another example of a display device is a standard computer monitor, including for example a liquid crystal display operatively coupled to a computer for receiving display control data from the computer to generate a signal for displaying image information content on the display device. A specific implementation of such a computer monitor is a digital lightbox. An example of such a digital lightbox is the product Buzz® by Brainlab AG. The monitor can also be a monitor of a portable device (e.g. handheld), such as a smart phone, a personal digital assistant or a digital media player.

[0122] The present application also relates to a program which, when running on a computer, causes the computer to perform one or more or all of the method steps described herein, and / or to a program storage medium (in particular in non-transient form) on which the program is stored, and / or to a computer comprising said program storage medium, and / or to a (physical, e.g. electrical, e.g. technically generated) signal wave, e.g. a digital signal wave, carrying information representative of the program (e.g. the aforementioned program), e.g. comprising code means adapted to perform any or all of the method steps described herein.

[0123] Within the framework of the present application, the computer program elements can be embodied in hardware and / or in software (including firmware, resident software, micro-code, etc.). Within the framework of the present application, the computer program elements can take the form of a computer program product which can be embodied by a computer-usable (for example, computer-readable) data storage medium comprising computer-usable (for example, computer-readable) program instructions, "code" or a "computer program" embodied in said data storage medium for use on or in connection with the instruction-executing system. Such a system can be a computer; the computer can be a data processing device comprising means for executing computer program elements and / or programs in accordance with the present application, for example comprising a digital processing processor (central processing unit or CPU), and optionally a volatile memory (for example, random access memory or RAM) for storing data and / or program code for executing the computer program elements and / or programs. Within the framework of the present application, the computer-usable (for example, computer-readable) data storage medium can be any data storage medium which can include, store, communicate, propagate or transport the program for use on or in connection with the instruction-executing system, apparatus or device. The computer-usable (for example, computer-readable) data storage medium can for example be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, apparatus or device or a propagation medium such as the Internet. The computer-usable or computer-readable data storage medium can even for example be paper or another suitable medium upon which the program is printed, since the program can be electronically captured, for example by optically scanning the paper or other suitable medium, then in suitable manner compiled, interpreted or processed. The data storage medium is preferably a non-volatile data storage medium. The computer program product described herein, and any software and / or hardware described herein form various means for performing the functions of the present application in example implementations. The computer and / or data processing device can for example comprise a guidance information device comprising means for outputting guidance information. The guidance information can for example be outputted visually to a user by visual indication means (for example, a monitor and / or a light), and / or acoustically to a user by acoustic indication means (for example, a loudspeaker and / or a digital speech output device), and / or haptically to a user by haptic indication means (for example, a vibrating element or a vibrating element incorporated into an instrument). For the purposes of the present document, a computer is a technical computer, for example comprising technical, for example tangible components, for example mechanical and / or electronic components. Any device mentioned herein is a technical and for example tangible device. BRIEF DESCRIPTION OF DRAWINGS

[0124] In the following, the present application is described in connection with the drawings, which give a background explanation and represent a specific embodiment of the present application. However, the scope of the present application is not limited to the specific features disclosed in the drawings, wherein:

[0125] Figure 2a to Figure 2c An embodiment of a clinical data structure is shown, which describes a plurality of clinical states of a clinical procedure and defines transitions between said plurality of clinical states, thereby allowing different paths of the clinical procedure;

[0126] Figure 3 Three different scenarios of a surgical procedure in an operating room are exemplarily shown in the form of three different images, which are generated by an embodiment of the method of the present application; and

[0127] Figure 1 is a schematic illustration of different transitions of three sensor values from a first clinical state to a second clinical state, which are used in the context of the present application. DETAILED DESCRIPTION

[0128] Figure 1 An embodiment of a clinical data structure 100 is shown, which contains a plurality of clinical states (states 1 to 4) of a specific clinical procedure, e.g. a specific part of a hip implantation. This is explained above in the context of method step S1. The clinical data structure 100 further defines transitions between said plurality of clinical states 1 to 4, thereby allowing different paths of the clinical procedure. One exemplary path is shown in Figure 1 with dashed lines. The shown path defines that the clinical procedure unfolds along the depicted clinical states 1, 2 and 3, but not along the unfolding of the clinical state 4 of the clinical procedure, which is shown. Figure 1 However, it is noted that another possible path can extend in both directions of the transition between two states, i.e. back and forth. Thus, Figure 1 One other possible path in the illustrative example of Figure 1 The geometric model used in this example of Figure 1 is defined or individualized. For example, in the illustrative example of Figure 1In the example, a specific individualized scenario of clinical state 1 is shown, wherein q1 is set to 0,9 and q2 is set to 0,7. q1 also fulfills the pre-defined boundary conditions: i.e. the minimum value of q1 is q 1,min =0,1 and the maximum value of q1 is q 1,max =14. Thus, with sensor values q1 =0,9 and q2 =0,7, a surgical scenario of clinical state 1 is defined. For example, parameter q1 is “position of the surgeon in the coordinate system of the operating room” and parameter q2 is “position of the tracked hip implant in the coordinate system of the operating room”. As understood by the skilled person, the same applies to the surgical scenarios defined by the exemplarily selected parameter values (i.e. the exemplarily selected sensor values) as shown for clinical states 2, 3 and 4 in Figure 1 .

[0129] Providing such a clinical data structure 100 as shown in Figure 2a to Figure 2c and one or more geometric models for modeling surgical scenarios allows to perform the method as presented herein. Thus, based on the clinical data structure 100 and the provided one or more geometric models, the following steps can be performed to:

[0130] a) defining a single path of the clinical procedure by selecting at least one transition between a first clinical state and a second clinical state of the clinical data structure (step S3);

[0131] b) providing a set of geometric model parameter values of at least one geometric model for the first, second and third clinical state (states 1 to 3) of the defined single path as input to the geometric model, thereby modeling the first, second and third single surgical scenario (step S4), and

[0132] c) generating artificial intra-operative sensor data for the single path from state 1 to state 2 to state 3 based on the modeled first, second and third single surgical scenario (step S5). In an exemplary embodiment, the generation of the artificial intra-operative sensor data in step S5 is performed by rendering image data of an intra-operative image sensor and / or an intra-operative video sensor.

[0133] Figure 2a - Figure 2cThree different scenarios of a surgical procedure in an operating room 200 are exemplarily shown in the form of three different images, which are generated using the method embodiment of the present application as described at least in the steps S1 to S5. Within the operating room 200, there is a first surgeon 201, a medical staff 202 and a patient 203 on a patient support. Furthermore, a robot arm 204 holding a surgical instrument 205 is shown, which can be tracked by markers 206 and a marker-based tracking system 207. Furthermore, as additional sensors, an intraoperative image sensor 209 as well as a temperature sensor 210, a sound sensor 211 and a vibration sensor 212 are shown. The tracking system 207 can track the spatial position of the surgical instrument 205 by emitting electromagnetic radiation 208, which is detected by the system 207 when reflected back. The system 207 can also track the position of the patient 203, the surgeon 201 and the medical staff 202 (the respective markers are not shown). The operating room 200 further comprises an intraoperative display 213 for showing medical images of the patient to the surgeon 201. The system 207 as well as the sensors 209 to 212 are intraoperative sensors, the data of which can be modeled / simulated with the present application. It is clear to the person skilled in the art that on the one hand these artificial sensors are used to generate the scenarios of Figure 2a to Figure 2c , on the other hand they are also shown in the presented scenarios. In other words, if a model that has been trained with the present application is later used for example to recognize a state, it uses the sensors 209 to 212 as input. As an example, the tracking data of the instrument when a surgeon performs a certain surgical step "A" is simulated, which can be recognized when later analyzing the tracking data from the sensors during a real procedure. When comparing Figure 2a , it can be seen that the spatial position of the surgeon 201 in the operating room as well as the spatial position of the tracked surgical instrument 206 has changed from the scenario shown in Figure 2b to the scenario shown in Figure 2c . Furthermore, in Figure 3 , the surgeon 201 is no longer present and the position of the surgical instrument 205 has changed again.

[0134] Figure 3 is a schematic illustration of different transitions of three sensor values from a first clinical state to a second clinical state used in the context of the present application. The x-axis can be called "x", which is in line with the above formula y(ax). Thus, in Figure 3In the middle, a geometric model is used, which is a function of e.g. y(ax), where the output y is a sensor value, which depends on the variable input variable x. It is shown that two states are described by two geometric models with potentially different parameters or boundary conditions; for at least some sensor values S (or y in the formula) a smooth transition between the states can be defined without the need for a dedicated geometric model to transition the states. For example, there can be a first state "preparation", in which the entire OR can be simulated as seen from a ceiling mounted camera, but without focusing on anatomical details. In a second state, the surgical site will be simulated, e.g. the position of organs and instruments; when switching between the states, the instruments (e.g. positions in the simulation) are not "jumping" or at least substantially staying in place. This is equally true for the patient. Thus, ​ It is shown that a spatial registration can be used from the second geometric model into the first geometric model, wherein the spatial registration is configured to ensure that first sensor data of at least a first sensor generated for a first clinical state has a continuous transition to second sensor data of the at least first sensor generated for a second clinical state. This embodiment clearly shows that the required smooth transition applies to all adjacent states, wherein the respective models use or share at least one sensor of the same kind. Typically, for sensors that sense or measure spatial parameters and for sensors that sense or measure non-spatial parameters, the sensor data should be smoothly transitioned between adjacent or neighboring clinical states. In this way, it is possible to achieve the result that e.g. a video presented along a path of a clinical data structure describing a clinical procedure is consistent, e.g. the color of one object is the same; the position of an object does not "jump", the type of an object remains the same. However, as already explained above, some parameters are also allowed to jump, to avoid training a model on the wrong parameters. Some parameters will also "jump" in reality, e.g. the light intensity when a light is switched off. The method of the invention can take this into account.

[0135] Other variations to the disclosed implementations can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit can fulfill the functions of several items or steps recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope of the claims.

Claims

1. A computer-implemented method of generating artificial intra-surgery sensor data usable for training an artificial intelligence (AI) module, the method comprising the steps of: a) providing a clinical data structure describing a plurality of clinical states of a clinical procedure and defining transitions between the plurality of clinical states, thereby allowing different paths of the clinical procedure (step S1); b) providing at least one geometric model for modeling a surgery scene in one or more of the clinical states of the clinical procedure (step S2), wherein the at least one geometric model comprises a plurality of geometric model parameters and boundary conditions for the geometric model parameters; wherein the geometric model parameters describe the surgery scene in the clinical states of the clinical procedure, wherein a specific combination of the geometric model parameter values defines a specific surgery scene of a clinical state of the clinical procedure; the method further comprising the steps of: c) defining a single path of the clinical procedure by selecting at least one transition between a first clinical state and a second clinical state of the clinical data structure (step S3); d) providing a set of geometric model parameter values of the at least one geometric model for the first and second clinical states of the defined single path as input to the geometric model, thereby modeling a first and a second single surgery scene (step S4); and e) generating artificial intra-surgery sensor data for the single path based on the modeled first and second single surgery scene (step S5).

2. The method according to claim 1, further comprising the steps of: storing the generated artificial intra-surgery sensor data for the single path together with the set of geometric model parameter values on which the data is based, wherein the set of geometric model parameter values on which the data is based is stored as a training label; and / or storing the generated artificial intra-surgery sensor data for the single path together with an expected outcome the AI module trained on the data should produce, wherein the expected outcome the AI module trained on the data should produce is stored as a training label.

3. The method according to claim 1 or 2, wherein each clinical state of the clinical data structure comprises a label, and wherein each label describes the respective clinical state as a characteristic phase of the clinical procedure and describes the respective clinical state in a clinically distinguishable manner.

4. The method according to any of the preceding claims, wherein for a first clinical state of the single path, a first geometric model is provided; and wherein for a second clinical state of the single path, a second geometric model is provided, and wherein the first and second geometric models differ from each other; and / or wherein at least the boundary conditions of the first and second models differ.

5. The method according to claim 4, wherein the second geometric model is spatially registered into the first geometric model.

6. The method according to claim 5, wherein, The second geometry model is spatially registered into the first geometry model, configured to ensure a continuous transition of first sensor data of at least a first sensor generated for the first clinical state to second sensor data of the at least first sensor generated for the second clinical state.

7. The method according to claim 6, wherein For all geometry models of all adjacent clinical states of the clinical data structure, for generating intraoperative sensor data for the same type of intraoperative sensor, the spatial registration of the respective geometry model is configured to ensure the continuous transition of the generated sensor data of the adjacent clinical states.

8. The method according to any one of the preceding claims, wherein, The generated intraoperative sensor data comprises at least tracking data and image data; in particular tracking data and image data of a tracked intraoperative camera.

9. The method according to any one of the preceding claims, wherein Further intraoperative sensor data of an intraoperative sensor is generated, which is not used for and / or does not relate to the surgical scene of the clinical state of the clinical procedure.

10. The method according to any one of the preceding claims, wherein In step S5, the generation of intraoperative sensor data is performed by rendering image data of an intraoperative image / video sensor.

11. The method according to any one of the preceding claims, further comprising the following steps: Steps S3 and S4 are repeated for a plurality of clinical states and for a plurality of sets of the geometry model parameter values, and step S5 is repeated, thereby generating a plurality of intraoperative sensor data for a plurality of different paths of the clinical data structure and for a plurality of surgical scenes (step S6).

12. The method according to any one of the preceding claims, wherein, Intraoperative sensor data is generated for all clinical states of the defined single path of the clinical procedure.

13. The method according to claims 11 and 12, wherein The generated intraoperative sensor data of all clinical states of the defined single path of the clinical procedure is provided in chronological order, such that one or more videos of the clinical procedure are provided.

14. The method according to any one of the preceding claims, wherein All possible surgical scenes under each clinical state are modelled.

15. The method according to any one of the preceding claims, wherein In step S4, the values of the geometry parameters are provided by a random generator or by a mathematical function selecting the values of the geometry parameters.

16. The method according to any one of the preceding claims, wherein The created intraoperative sensor data at least partially describes the clinical procedure.

17. The method according to any one of the preceding claims, wherein The sensor data is intraoperative video data, and wherein step S5 of generating intraoperative sensor data comprises rendering an artificial video of the clinical procedure.

18. The method according to any one of the preceding claims, further comprising the following steps: f) training an AI module for automatically analyzing real intra-operative sensor data using the generated artificial intra-operative sensor data and the set of geometric model parameter values used during generation of the artificial intra-operative sensor data.

19. The method according to claim 18, further comprising the steps of: g) providing real intra-operative sensor data describing a real clinical procedure to the trained AI module trained with the generated artificial intra-operative sensor data according to any of the preceding claims, and h) automatically identifying the real clinical procedure by the trained AI module.

20. The method according to any of the preceding claims, wherein, the clinical data structure is embodied as a graph, wherein a path comprises a plurality of discrete states of the clinical procedure.

21. The method according to any of the preceding claims, wherein, each transition of the clinical data structure comprises a weight associated with a frequency of occurrence in the clinical procedure.

22. The method according to any of the preceding claims, wherein the clinical procedure is a surgical procedure, a diagnostic procedure, and / or a therapeutic procedure.

23. The method according to any of the preceding claims, wherein, for at least two clinical states, preferably for all clinical states, the respective geometric model, the respective geometric model parameters, and the respective boundary conditions depend on the clinical state.

24. The method according to any of the preceding claims, wherein the provided geometric model is configured for spatially modeling an intra-operative scene in one or more of the clinical states of the clinical procedure.

25. The method according to any of the preceding claims, wherein, the geometric model parameters define at least a position, a shape, and / or a type of one or more objects occurring in the intra-operative scene of the clinical state of the clinical procedure.

26. A program which, when running on a computer or when loaded onto a computer, causes the computer to perform the method steps of the method according to any of the preceding claims.

27. Training data for training an AI module for automatically analyzing sensor data during a real surgery, wherein, the training data is generated according to the method of any of claims 1 to 25.

28. A program storage medium having stored thereon the program according to claim 26 and / or the training data according to claim 27.

29. An AI module for automatically analyzing real intra-operative sensor data, wherein the AI module is trained with the training data according to claim 27.

30. The AI module according to claim 29, wherein the AI module is configured to receive real intra-operative sensor data describing a real clinical procedure as input, and wherein the AI module is configured to automatically identify the real clinical procedure by analyzing the received real intra-operative sensor data.

31. A computer system for automatically analyzing real intra-operative sensor data, the computer system comprises an AI module according to claim 29 or 30.

32. A computer system according to claim 31, the computer system is configured to receive real intra-operative sensor data describing a real clinical procedure as input, and wherein the computer system is configured to automatically identify the real clinical procedure by analyzing the received real intra-operative sensor data.

Images