Surgical digital reproduction and assistance system and synchronous recording method

Abstract

The present disclosure relates to a surgical digital reproduction and assistance system (1) for time-dependent transmission of a surgical operation at a patient (P) or for virtual training or evaluation of a surgical operation at a patient (P), having a central data information system (2), at least one first data modality (6) and a second data modality (8), which is different from the first data modality (6), are stored or at least temporarily stored therein, both the first data modality (6) and the second data modality (8) having at least one position specification with respect to a unified common coordinate system (10) of the patient (P), the invention relates to a method for operating a display device, in particular a display device, comprising: a view creation unit (14) for creating a view to a display device (16); and a display device (16) for displaying the view, the view creation unit (14) being arranged on the display device (16). The view creation unit is adapted to generate, digitally provide, a visual view (18) having the first data modality (6) and the second data modality (8) at a preset time, based on a user preset (16) having at least one preset time, and at least one visual user interface (20), the created view (18) is digitally transmitted to the visual user interface and is output and displayed to the user by means of a presentation device (22) in order to evaluate, in particular, a surgical phase or to select a recorded section of the surgery and to reproduce at least the first and second data modalities in a spatially and temporally synchronized mode. Furthermore, the present disclosure relates to a synchronous recording method according to the paralleling claims as well as a computer-readable storage medium and a computer program.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a surgical digital reproduction and assistance system / virtual surgery system for the temporally related or temporally synchronous, in particular time real-time, transmission of a surgical operation at a patient or for virtually training or evaluating a surgical operation at a patient. It can also be said that the present disclosure relates to a virtual surgery training system and a virtual surgery assistance system. Furthermore, the present disclosure relates to a synchronous recording method, a computer-readable storage medium and a computer program according to the preamble of the independent claims. BACKGROUND

[0002] For surgical operations on patients, so-called video platforms or surgery platforms for the joint use (team platforms) or also platforms for the joint use of medical data are currently known. All of the above-mentioned solutions generally share medical data in the form of video streams / videos / video transmissions and enable the retrospective observation of an operation (e.g. filming of the operation and post-hoc reproduction). In particular embodiments, these also allow the joint use in real time during the operation. Here, different video recordings of video cameras arranged at different locations are detected by a central control unit and a view is created in which the different video streams are arranged side by side. The view is then output via a surgery display.

[0003] However, when different modalities are to be used together, that is to say when a system comprising a plurality of modalities which are also to interact with one another is to be used, problems arise. The current problem is therefore that when, in addition to video cameras, further medical data is to be accessed together, these only exist in isolation from one another and are not related to one another. This can for example be considered to be such that a first video is created by means of a surgery microscope (independently). Furthermore, a further (independent) video is created by means of an overview camera. Although the surgeon can later input these videos into a computer and play them back one after the other or side by side, they remain unrelated to one another at the time of recording. SUMMARY

[0004] The task of the present disclosure is therefore to avoid or at least reduce the disadvantages in the prior art and in particular to provide a surgical digital reproduction and assistance system, a synchronous recording method, a computer-readable storage medium and a computer program which enable the observation of a digitalized surgical operation / manipulation and provide the user with the possibility of intuitively switching and integrating different modalities. A further task can be to provide a central data system which can detect and store data from different modalities in a manner related to one another in order to then (in a data-connected manner) visually reproduce these data to the user as required. A sub-task is to interconnect the data of different modalities spatially and temporally with one another.

[0005] The task of the present disclosure is solved in accordance with the invention by the features of claim 1 with respect to a surgical digital reproduction and assistance system, by the features of claim 13 with respect to a method for synchronous recording, by the features of claim 14 with respect to a computer-readable storage medium, and by the features of claim 15 with respect to a computer program.

[0006] The basic idea of the present disclosure is therefore to provide at least two different (data) modalities for visualization, which are synchronized or related to each other in terms of time and location. This (surgical) system enables to observe a digitized surgery, i.e. a digitally recorded surgery, comprising a plurality of time-synchronized modalities positioned in a common coordinate system and thus data modalities, wherein the system in particular enables a user to switch between the modalities, to select sections (chapters) of the recorded surgery and to play all modalities in a synchronized mode (time and space).

[0007] The surgical digital reproduction and assistance system of the present disclosure thus not only synchronizes data in time, but also provides a spatial synchronization or correlation, i.e. the modalities are tracked in a common space or common coordinate system. The system and method thus enable a complete immersion in the surgery and thus in the subject of the surgery, wherein all modalities (data modalities) can be examined individually, but also in combination with each other (and in particular in a relationship). The user is thus able to experience the surgery as if he were in the operating room himself. In particular, an automatic time synchronization is provided.

[0008] In particular, a virtual surgical system (surgical digital reproduction and assistance system) is proposed, wherein two or more surgical modalities (data modalities) are imported from a central storage system, wherein the surgical modalities can be, for example, image data, volume data, position data, electrical signal data or voice data, but wherein at least two modalities are related or synchronized in time and space. Furthermore, a user interface is provided in the virtual surgical system, which enables to reproduce a surgery or case and to observe the at least two modalities. Thereby, a visual information about the surgical environment can be created by the data modalities, which enables the user to directly evaluate a specific surgical phase or step, more precisely, independently of the location, in particular not only independently of the location but also independently of the time.

[0009] The system of the present disclosure can be used, for example, by a trainee surgeon to remotely and virtually process a surgical or expert medical case. The system can also be used to exchange cases between experts, such as in the form of a surgical video analysis, offline (after the case) or online (during the case).

[0010] In other words, a surgical (virtual) digital reconstruction and assistance system for the time-dependent, in particular time-real-time transmission of a surgical procedure at a patient or for the virtual training or evaluation of a surgical procedure at a patient is disclosed and provided, having a central data information system / data information unit, in particular having a storage unit, in which at least one first (medical) data modality and a second (medical) data modality different from the first data modality are stored or at least temporarily stored, wherein not only the first data modality but also the second data modality have at least one (absolute) position specification, in particular orientation specification, with respect to a uniform common (global) coordinate system (KOS) of the patient, and are thus related (synchronized) to each other at least with respect to the position, in particular the orientation, and furthermore have a temporal specification (time stamp) in order to also be synchronized or related to each other with respect to a uniform absolute time, a view creation unit, which is adapted to generate a visual view having the first data modality and the second data modality at the (exactly) preset temporal specification on the basis of a user preset having at least one temporal specification, and at least one visual user interface, to which the created view is digitally transmitted and output and displayed to a user by means of a presentation device in order to evaluate a surgical procedure phase or to select a section of a recorded procedure and to reproduce at least the first and second data modality in a spatially and temporally synchronized mode. The transmission does not necessarily have to take place in real time, but can also take place in a so-called offline mode after the procedure. The data of the procedure only need to be related in time and space.

[0011] The term "position" refers to a geometric spatial position in three-dimensional space, which is specified, in particular, by means of coordinates of a Cartesian coordinate system. In particular, the position can be specified by three coordinates X, Y and Z.

[0012] The term "orientation" specifies a directional specification (approximately at the position) in space, too. It can also be said that the directional specification or rotational specification in three-dimensional space is specified by the orientation. In particular, the orientation can be specified by means of three angles.

[0013] The term "position" refers to a geometric spatial position in three-dimensional space, which is specified, in particular, by means of coordinates of a Cartesian coordinate system. In particular, the position can be specified by three coordinates X, Y and Z.

[0014] The term data modality here denotes that digital data is detected by a modality, such as a surgical microscope or an ultrasound device or a camera, in particular a shot that is provided as data modality. The position of the data modality here means that the position of the shot, for example, should be detected and stored. For example, a surgical microscope can be spatially tracked / traced in space relative to a uniform coordinate system of the patient, the orientation of the surgical microscope head can be determined with an optical system, and the position of the shot can be determined by the distance of the optical system from the shot, ultimately indirectly via the surgical microscope. It can even be sufficient in individual cases of the modality to approximately determine the position of the device and to approximately consider the position of the data modality. It can also be said that the shot device for the shot data almost always belongs to the data modality, wherein, of course, data can also be entered and the device does not have to belong to the system. It is important only that a spatial specification in the form of at least one position, in particular an orientation, for example of an instrument, in a uniform coordinate system in the coordinate system, together with a (synchronous) temporal specification, for example in the form of March 1, 2022, 17:53:33, is stored (in images with a time stamp, in videos that naturally have a continuous temporal specification).

[0015] The term time-related or synchronous here means that the data is synchronized with respect to a central absolute (system) time. This means that a uniform temporal specification is used when recording the data (in all modalities). For example, an internal clock can be used for each modality separately, wherein these internal clocks are coordinated with one another and display the same time on a second precision, in particular on a millisecond precision. For example, an internal radio clock can be used separately, which is set by radio, for example GPS, or a central clock generator can set the internal clocks via a data connection, for example via LAN or WLAN. Alternatively, for example, the data can also be transmitted to a central unit in real time with negligible delay, which then provides the data with a time and thus correlates with one another in time.

[0016] Advantageous embodiments are claimed in the dependent claims and in particular are set forth below.

[0017] According to embodiments, the surgical digital reproduction and assistance system can be adapted to run in a live mode, in which the first and second data modalities are provided and temporarily stored in real time with respect to time and the preset of the view creation unit is continuously run together in real time with respect to time in order to provide, inter alia, a video recording and a virtual reproduction of the surgery to the user independently of the location, in particular enabling a surgeon at a remote location to perform a surgery at a patient in an operating room, or the surgical digital reproduction and assistance system is adapted to run in a post-surgery mode, in which the recorded and stored surgery can be presented repeatedly at a later point in time in order to, for example, confirm a correct implementation of the surgery or to provide a virtual training to the user independently of the location and independently of the time. Thus, in the first mode, the surgical digital reproduction and assistance system can be adapted to transmit the surgery in real time with respect to time in the operating room to, for example, a further hospital, in which a surgeon assisting or performing a remote surgery is seated. By configuring the transmission to the user interface, the digital data can be forwarded at the time of transmission using different technologies, also securely transmitted to the client location via the Internet, and a secure and real-time-in-time visual data transmission can be achieved. The view data can also be "copied" to a certain extent in order to provide a respective user interface with, for example, a display for a plurality of users, so that a plurality of users can also virtually participate to a certain extent in the narrow operating room.

[0018] Preferably, the surgical digital reproduction and assistance system has a server with a central data information system and a plurality of clients with a respective user interface, wherein the surgical digital reproduction and assistance system is adapted to inquire individualized user presets from each user interface based on which the view is created and subsequently transmitted to the user interface individually adapted. Thus, each user, for example a surgeon with different expertise, can precisely find the data modalities and display or review them, for example, enlarged, which provides a particular advantage to the user. Thus, a network with a central server and a plurality of clients can be created. In particular, the data connection between the server and the clients is created via a secure intranet connection or a secure internet connection.

[0019] Preferably, at least the first data modality and / or the second data modality can have, in addition to a temporal specification and at least a location specification:

[0020] - a 2D recording (image or stream),

[0021] - a 3D recording (image or stream),

[0022] - a 3D point cloud recording (image or stream),

[0023] - an electrical / electrophysiological signal diagram,

[0024] - a voice note, and / or

[0025] - position information of a surgical instrument, in particular with instrument type. The data modality can thus in particular have a 2D recording (image or stream), a 3D recording (image or stream), a 3D point cloud recording (image or stream), an electrophysiological signal, a voice note and / or a position information of a surgical instrument.

[0026] The data modality preferably comprises a recording from a (surgical) staff camera, a recording from a patient camera, a microscope recording (in particular microscope images), an endoscope recording (in particular endoscope images), a fluorescence recording (in particular fluorescence images), a navigation recording (in particular navigation images), a histology recording (in particular histology images), an ultrasound recording (in particular ultrasound images), an electrophysiological signal and / or a voice note.

[0027] The surgical digital reproduction and assistance system has in particular a user input interface for processing the data stored in the central data information system, in particular supplemented or supplemented with annotations or supplemented with explanations in the (selected) modality with voice notes.

[0028] The surgical digital reproduction and assistance system can in particular have at least one computer and / or a portable mobile device, in particular a tablet, which forms at least one visual user interface and outputs the created views via a presentation device connected thereto, in particular a display. The user interface can thus in particular be run on a desktop computer or a mobile device such as a smartphone or a tablet.

[0029] According to a further embodiment, the presentation device can be a 2D display or a 3D display or a head-up display, in particular a virtual reality head-up display (VR) or an augmented reality head-up display (AR) or an MR head-up display. The visual information can thus be displayed via a 2D display such as a surgical monitor, a 3D display such as a monitor with the associated 3D glasses or a head-up display (VR, AR, MR).

[0030] According to an embodiment, the surgical digital reproduction and assistance system can have a digital surgical system with a visualization system, a navigation system and a robot with a robot arm, which is adapted to spatially correlate the first and second data modalities and to record the first and second data modalities in time synchronously or in mutual correlation and to provide them to the central data information system. The surgical digital reproduction and assistance system (virtual surgical system) obtains the data detected by the digital surgical system in particular via a cloud platform. The data of the data modalities are generated by the digital surgical system, which comprises the visualization system, the navigation system and the robot arm, wherein, of course, additional data can also be read or imported by computer technology.

[0031] In particular, the data information system may also store pre-operative treatment plans and / or post-operative treatment reports for the corresponding surgery, which are also included in the user's presets or views. In other words, the system can access the pre-operative treatment plan and / or post-operative treatment report for the corresponding case. Parts of this treatment plan information can be displayed together in the view, such as which phases or steps should be performed, when, and in what order. This allows the surgeon to centrally present all relevant information as needed.

[0032] Preferably, the view creation unit can insert a timeline with the surgical phases of the surgery recorded therein into the view, in order to enable quick access to each surgical step. The system particularly displays the phases of the surgery and enables quick access to each surgical step, wherein a preset time can be entered, for example, via a touch display, and the view is created at this (preset) time.

[0033] According to another embodiment, the view creation unit can be adapted to automatically determine the associated primary data modality from at least the first and second data modalities based on the current surgical phase and display this primary data modality in a view enlarged relative to the other data modality. In particular, the system automatically determines the primary modality for the current surgical phase and displays it enlarged. For example, the phase "Examination of tissue using a surgical microscope" can be stored in the treatment plan, and the view creation unit can be adapted to display the surgical microscope recording in an enlarged manner in the created view, with this recording occupying, for example, 50% of the view, while the remaining data from the remaining data modalities are displayed at other locations in the view.

[0034] In particular, three-dimensional navigation data can also be stored in the data information system as a 3D data set, in particular pre-operative CT and / or MRI recordings of at least one body segment of the patient, in order to preferably enable intraoperative reconstruction of the patient's tissue volume during the operation. The navigation data is thus available as a 3D data set and can be scrolled through or "virtually" navigated through the (tissue) volume at any time during the observation of the operation. This provides the surgeon with an additional important data modality.

[0035] Preferably, the surgical digital reproduction and assistance system may have a navigation system, and in order to provide the first data modality and / or the second data modality, may also have the following devices (as some kind of shooting equipment):

[0036] a digital surgical microscope for creating 2D or 3D recordings, in particular in the form of images or videos (e.g. in HD or 4K resolution), whose recording position, in particular the recording orientation (indirectly via a tracked microscope lens), is tracked by a navigation system;

[0037] - a (surgical) endoscope for creating 2D recordings or 3D recordings, in particular in the form of videos (e.g. HD or 4K resolution), whose recording position, in particular recording orientation (indirectly via the tracked endoscope), is tracked by the navigation system;

[0038] - a fluorescence image generator for creating fluorescence recordings, preferably coupled with microscope recordings and / or endoscope recordings (in particular videos), in particular 2D or 3D recordings (e.g. HD or 4K resolution), whose recording position, in particular recording orientation (indirectly via the tracked fluorescence image generator), is tracked by the navigation system

[0039] - at least one operating room staff camera (e.g. an operating room overview camera or a body camera tracked in space) related to the navigation system for providing a video and / or 3D point cloud image stream, in particular with the option of creating a representative symbol (avatar) for the respective operating room staff (for data protection reasons)

[0040] - a patient camera related to the navigation system (whose optical axis and field of view are directed at the patient) in order to detect and provide a video (image stream) and / or 3D point cloud image stream, in particular;

[0041] - a microphone for detecting, storing and later reproducing voice notes in time by means of an audio device;

[0042] - a navigation system with navigation data, in particular 3D volume data sets, preferably CT recordings and / or MRT recordings (e.g. in DICOM format) detected preoperatively and / or intraoperatively;

[0043] - a robot with a robot arm, with the coordinates of the robot arm, which are tracked together by the navigation system;

[0044] - surgical instruments, their instrument types and their instrument positions, especially orientations, are tracked with a navigation system. In other words, the surgical digital reproduction and assistance system and the central data information system preferably comprise the following modalities (e.g. formats): a digital surgical microscope, especially 2D or 3D recordings, especially videos (e.g. in HD or 4K resolution), are tracked by the navigation system; an endoscope, especially 2D or 3D recordings (videos) (e.g. in HD resolution or 4K resolution) are tracked by the navigation system; a fluorescence image generator for fluorescence recordings, preferably coupled with microscope recordings and / or endoscope recordings (videos), 2D or 3D recordings (e.g. in HD resolution or 4K resolution), are tracked by the navigation system; at least one operating room staff camera (e.g. overview camera), a video image stream and / or a 3D point cloud image stream is related to the navigation system, especially to the creation of a representational symbol (avatar) for the respective surgical staff member (for data protection reasons); a patient camera in order to detect and provide a video (stream / image stream) and / or a 3D point cloud image stream related to the navigation system, especially; a microphone for detecting, storing and later reproducing voice notes on an audio device; a navigation system with navigation data, especially 3D volume data sets, preferably CT recordings and / or MRT recordings (e.g. in DICOM format) detected preoperatively and / or intraoperatively; a robot with a robot arm, with coordinates of the robot arm, are tracked together by the navigation system; surgical instruments, their instrument types and their instrument positions, especially orientations, are tracked with a navigation system. These different (modal) recording devices can detect different modal data and store them synchronously in time and space and later visually provide them on a view.

[0045] According to the aspects claimed in the further application, which can be claimed completely independently according to the present disclosure, data (or data of the data modalities) can be detected and generated, inter alia, by a digital surgical system, which comprises a microscope and / or an endoscope and / or a navigation system and / or a robotically supported instrument guidance and / or a video camera and / or a microphone for voice notes and / or an interface to a third modality, such as a neuron monitor, histology or ultrasound (photography device). Furthermore, the digital surgical system can comprise a control unit, which is adapted to perform a spatial positioning with regard to at least one position, in particular an orientation, for each data modality, preferably by means of a navigation system, and to record a time specification (central absolute time) to which at least one of the data modalities belongs. All data are spatially related, inter alia, by means of a navigation system (with a tracking system, such as a navigation camera), by tracking the modalities in space during the surgery. All data are detected centrally in the digital surgical system, are automatically provided with a common time stamp or time specification and are stored, in particular in a server or cloud system. By processing a plurality of different modalities, the data can also be associated semantically, i.e. the recognition of surgical steps and the creation of a script of the recorded surgery. According to the present disclosure, two systems can be developed, each of which fulfils an independent function and is related to one another. The first system, the digital surgical system, creates data modalities and synchronizes them in time and space. The second system, the surgical digital reproduction and assistance system, in turn accesses these synchronized data and provides them to the user as required.

[0046] Preferably, the surgical digital reproduction and assistance system can have a digital twin, which presents all detected devices in the operating room, inter alia, in an avatar mode.

[0047] According to a further embodiment, the surgical digital reproduction and assistance system can also have the following means for providing at least one further (third) data modality:

[0048] a neuron monitoring device for detecting neuron monitoring data, in particular electrical signals or photographs (in particular images), the neuron monitoring position, in particular the orientation (indirectly via the neuron monitoring device), being tracked by means of a navigation system;

[0049] a biopsy device for creating tissue data, in particular microscope 2D photographs (images; for example in DICOM format), the histology position, in particular the orientation (indirectly via the tracked biopsy device), being tracked by means of a navigation system;

[0050] - an ultrasound device for creating ultrasound recordings, preferably 2D or 3D recordings (images), the recording position, in particular the recording orientation (indirectly via the tracked ultrasound device) of which is tracked by the navigation system. That is, the following data modalities can also optionally be stored: neuro-monitoring data, in particular electrophysiological signals or recordings (e.g. images), which are spatially localized by the navigation system; tissue data, in particular microscopic 2D recordings (images; e.g. in DICOM format), which are spatially localized by the navigation system; and / or ultrasound recordings, preferably 2D or 3D recordings (in particular images), which are spatially localized by the navigation system.

[0051] The view creation unit can in particular be adapted to arrange data modalities of the macroscopic (e.g. surgical staff camera) on the microscopic (e.g. microscope image) up to the nanoscopic (e.g. tissue image) in order to provide the user with an intuitive and structured visual view.

[0052] The user interface can preferably be a web-based software platform which enables the experience of a surgical case from almost anywhere. Through just one browser or software application as terminal device of the user interface a (secure) connection to the server can be built (through an intranet or the internet) and data can be provided accordingly.

[0053] The digital surgical system can in particular be provided with time-synchronized and spatially related multi-modal data through a data / video interface (e.g. wired Ethernet or HDMI or SDI).

[0054] The task regarding a synchronized recording method for creating a spatially and temporally synchronized first data modality and a second data modality of a surgical digital reconstruction and assistance system is solved in particular according to the present disclosure by a method having the steps:

[0055] Tracking at least one position, in particular orientation, of the first data modality in space by the navigation system relative to a common global coordinate system;

[0056] Detecting and storing provided data, in particular recordings, of the first data modality, together with the tracked position and together with a time specification of a uniform time, preferably together with a semantic of a surgical step;

[0057] Tracking at least one position, in particular orientation, of the second data modality in space by the navigation system relative to / in the common global coordinate system;

[0058] Detecting and storing provided data, in particular recordings, of the second data modality, together with the tracked position and together with a time specification of a uniform time;

[0059] The view creation unit creates a viewable view with the first data modality and the second data modality at the time specification of the preset based on the user preset with at least one time specification

[0060] The view is output by a presentation device.

[0061] The tasks in relation to the computer-readable storage medium and in relation to the computer program, that is to say by means of instructions which, when implemented by a computer, cause the computer to carry out the method steps according to the present disclosure.

[0062] The present disclosure in relation to the surgical digital reproduction and assistance system of the present disclosure is equally applicable to the synchronized recording method according to the present disclosure, and vice versa, in particular by analogous features of the device and the method. BRIEF DESCRIPTION OF DRAWINGS

[0063] The application is explained below in more detail with the aid of the drawings according to preferred embodiments. Shown are:

[0064] Figure 1 A schematic view of a surgical digital reproduction and assistance system according to a first preferred embodiment is shown, with viewable views of different data modalities generated by a digital surgical system and synchronized in space and time;

[0065] Figure 2 The working principle of storing data with spatial and temporal synchronization is shown schematically;

[0066] Figure 3 The exchange of data between a digital surgical system and a surgical digital reproduction and assistance system according to a further preferred embodiment is shown in a schematic view;

[0067] Figure 4 Possible applications of a surgical digital reproduction and assistance system are shown;

[0068] Figure 5 A user interface of a surgical digital reproduction and assistance system of a further preferred embodiment is shown; and

[0069] Figure 6 A flowchart of an embodiment of the synchronized recording method according to the present disclosure is shown.

[0070] The drawings are essentially schematic and are only for understanding the application. Identical elements are provided with identical reference signs. Features of the various embodiments can be replaced by one another. DETAILED DESCRIPTION

[0071] Figure 1schematic representation of a surgical digital reproduction and assistance system 1 for the transmission of a surgical operation at a patient P in time, here in real time of this time, or for the virtual training or evaluation of a surgical operation at a patient P according to a first preferred embodiment of the present disclosure.

[0072] The surgical digital reproduction and assistance system 1 has a central data information system 2 which has a storage unit 4 in which at least a first data modality 6 and a second data modality 8 different from the first data modality 6 are stored in a database. The first data modality 6 as well as the second data modality 8 have at least one position specification, in particular an orientation specification, with respect to a uniform, common coordinate system 10 of the patient P. In this embodiment, for the recording of the first data modality the belonging (X1, Y1, Z1) coordinates are stored in the entry of the database and for the recording of the second data modality also the belonging (X2, Y2, Z2) coordinates. Here, the database is also implemented with a column of position specifications. By means of this position specification, the two data modalities are related to each other with respect to the position and, if the orientation is also specified, for example in another data block column, in particular with respect to the orientation. Thus for each recording of each data modality there is (at least) a uniform position specification and the data modalities are spatially related or synchronized.

[0073] Furthermore, for each data modality (for each recording) also a time specification 12 is stored in order to be synchronized with each other also with respect to a uniform absolute time. In particular, recordings can be created at different points in time for each data modality 6, 8 and stored in a manner provided with a time stamp or time specification. For example, a surgical microscope can create a recording as a first data modality, a movement, at a first point in time tl in a (recorded) first position x_t1, y_t1, z_t1 and a further recording at a later second point in time t2 in a second position x_t2, y_t2, z_t2. The two recordings are stored. In particular, the surgical microscope can also record a video or video stream in which the position of the recording is continuously tracked (indirectly through the surgical microscope head). The video can for example be provided with a continuous time specification as usual and stored accordingly.

[0074] Important in the present surgical digital reproduction and assistance system 1 is that the data of the two different data modalities 6, 8 are entered and synchronized with each other not only spatially but also temporally and thus associated.

[0075] Based on this data information system 2 with stored data, a view creation unit 14 of the surgical digital reproduction and assistance system 1 can be digitally provided, which is adapted to generate a visual view 18 with the first and second data modality 6, 8 at a preset time based on a user preset 16 with at least one preset time. For example, when the user performs an input by touching the display and enters the preset time of the surgery as well as further parameters of the user preset, for example selects the data modality 6, 8 to be enlarged, the view creation unit 14 creates a view exactly at the time point of the surgery with the first and second data modality. Like a time machine or a time machine it is thus possible to jump to a determined time point and to generate a view. The created view 18 exists in digital form, in particular as a single static digital image or in the form of a moving view such as a video stream starting from the preset time.

[0076] The created digital view 18 is transmitted to the visual user interface 20 of the surgical digital reproduction and assistance system 1 and is visually output and displayed to the user by means of a presentation device 22. Based on this visual information, the user can be trained, or the user can evaluate a surgical phase or select a section of the recorded surgery and reproduce at least the first and second data modality in a spatially and temporally synchronized mode.

[0077] Figure 1 Here further different modalities are shown in addition to the first and second data modality, which are detected by the digital surgery system and which are temporally and spatially related according to the present disclosure.

[0078] Figure 2 In a schematic diagram it is shown how the data of the data modalities are stored, wherein the content of the (data) modalities is detected and the temporal and spatial correlation between the data.

[0079] In Figure 3 In a schematic diagram it is shown how the data of the data modalities are stored, wherein the content of the (data) modalities is detected and the temporal and spatial correlation between the data. Figure 3 The data exchange between the digital surgery system for data detection and the surgical digital reproduction and assistance system 1 for visual data presentation is shown.

[0080] Figure 4A digital surgical reproduction and assistance system 1 according to an embodiment of the present disclosure is shown. The digital surgical system is equipped with a monitor as a possible presentation device 22, a navigation system 104 for tracking instruments and for navigation, and a robot 106 with a robotic arm 108, to the end of which a surgical microscope 28 is connected as an end effector. Here, an experienced surgeon uses the digital surgical system to perform surgery in the real world, and the digital surgical system detects all data modalities together with the position or orientation in the real world and the time description. Data in the real world is thus collected. These data modalities are provided to the central information system 2 of the digital surgical reproduction and assistance system 1. The trainee surgeon can in turn access the data from any location, regardless of location, in particular via a website portal. The digital surgical reproduction and assistance system 1 creates a digital twin for the trainee surgeon, which presents a digital copy of the operating room (OP) and provides the trainee surgeon with the possibility of entering the operation. The trainee surgeon can observe the surgical operation via a 2D screen, a 3D screen or via VR or AR glasses / head-up display, as in Figure 4 , and enter the surgery. In particular, multiple trainee surgeons can access this user interface in the form of VR glasses, allowing multiple users to simultaneously track and train surgical procedures regardless of location. For example, during physician training, medical students or surgeons can be provided with this visual user interface 20 to enhance surgical techniques and subsequently provide personalized training.

[0081] Figure 5A user interface of a surgical digital reproduction and assistance system 1 showing a further embodiment. In this embodiment, the presentation device is configured in the form of a touch display and the view has a plurality of subfields with different data modalities. Through this user interface an impression of a cockpit is produced in which all modalities are displayed. The primary (data) modality of the different data modalities is shown enlarged centrally, wherein the user can of course change the layout through the touch display and can adapt it to the requirements, for example similar to a mobile phone by long-pressing a field of a data modality, then a change of the arrangement can be made by dragging the window. At the lower edge of the view 18 a control unit similar to a video player is visible, which indicates on a time scale what modality is active when and which enables the user to jump to a specific step in the surgical case or operation. The surgical case is likewise structured with a script and specific stages. Icons and sliders enable quick access to certain surgical steps. The preoperative plan (treatment plan) and the postoperative report (treatment report) can likewise be accessed. A virtually additionally added ring illustrates the spatial correlation between the data, for example the correlation between the focus of a microscope shot (in particular a microscope image) and the corresponding position in the navigation data set. The navigation data set has not only a video (stream) but always also a 3D data set. Thus at any time the surgical case can be paused and virtually "moved" through the 3D data set in order to explore for example structures that are not directly visible. The modalities are arranged from a macro view (for example an employee camera), a micro view (for example a microscope) to a nano view (for example histology). This enables the user to "zoom in" and "zoom out" when using different data modalities.

[0082] Figure 6 A flow chart showing a synchronization recording method for creating spatially and temporally synchronized first and second data modalities 6, 8 for a surgical digital reproduction and assistance system 1 according to the present disclosure.

[0083] In a first step S1 at least one spatial position, in particular a position, of the first data modality 6 is tracked S1 by the navigation system 104 relative to the common global coordinate system 10.

[0084] The digitally provided data, in particular a shot, of the first data modality 6 is then detected and stored S2 in the central data information system 2 together with the tracked position and together with a time indication of the uniform time in a second step S2.

[0085] In a step S3 at least one position, in particular a position, of the second data modality 8 in space relative to or in the common global coordinate system is tracked S3 by the navigation system 104, similar to step S1, wherein also the order of the first and second data modalities can be reversed.

[0086] In step S4, the provided data of the second data modality (8), in particular the taken recordings, are detected and stored in the central data information system (2) together with the tracked positions and together with the time specification of the uniform time, analogously to step S2.

[0087] Thereafter, in step S5, the created view 18 is transmitted to a user interface 20 in step S6.

[0088] Thereafter, in step S5, the created view 18 is transmitted to a user interface 20 in step S6.

[0089] Finally, in step S7, the output of the view 18 is performed by a presentation device 22.

[0090] By this synchronized recording method, data of different modalities or different data modalities can be detected and stored mutually synchronized in terms of space and time. At a later point in time, these data modalities can then be reproduced, in particular at a preset time of the user.

[0091] List of reference signs

[0092] 1 Surgical digital reproduction and assistance system

[0093] 2 Data information system

[0094] 4 Storage unit

[0095] 6 First data modality

[0096] 8 Second data modality

[0097] 10 Common global coordinate system

[0098] 12 Time specification

[0099] 14 View creation unit

[0100] 16 User preset

[0101] 18 Created view

[0102] 20 Visual user interface

[0103] 22 Presentation device

[0104] 24 Surgical instrument

[0105] 26 Timeline

[0106] 28 Surgical microscope

[0107] 30 Endoscope

[0108] 32 Fluorescence image generator

[0109] 34 Operating room staff camera

[0110] 36 Patient camera

[0111] 38 Microphone

[0112] 40 Neuromonitoring device

[0113] 42 Biopsy device

[0114] 100 Digital surgical system

[0115] 102 Visualization system

[0116] 104 Navigation system

[0117] 106 Robot

[0118] 108 Robotic arm

[0119] P Patient

[0120] S1 Step of tracking a first data modality position

[0121] S2 Step of detecting and storing data of the first data modality with position and time

[0122] S3 Step of tracking a position of a second data modality

[0123] S4 Step of detecting and storing data of the second data modality with position and time

[0124] S5 Step of creating a visual view

[0125] S6 Transmitting the visual view to a user interface

[0126] S7 Outputting the view by a presentation device

Claims

1. A surgical digital reproduction and assistance system (1) for the temporally correlated transmission of a surgical operation at a patient (P) or for the virtual training or evaluation of a surgical operation at a patient (P), having: The central data information system (2) has, inter alia, a storage unit (4) in which at least one first data modality (6) and a second data modality (8) different from the first data modality (6) are stored or at least temporarily stored, wherein both the first data modality (6) and the second data modality (8) have at least one position specification, in particular an orientation specification, with respect to a uniform, common coordinate system (10) of the patient (P) and are thus related to one another at least with respect to the position, in particular with respect to the orientation, and furthermore have a time specification (12) in order to be synchronized with one another with respect to a uniform absolute time, a view creation unit (14) which is adapted to produce, digitally provide a viewable view (18) having the first data modality (6) and the second data modality (8) at a user preset time on the basis of a user preset (16) having at least one preset time, and at least one viewable user interface (20) to which the created view (18) is digitally transmitted and output and displayed to a user by means of a presentation device (22) in order to evaluate a surgical operation phase or to select a section of a recorded operation and to reproduce at least the first and second data modalities in a spatially and temporally synchronized mode.

2. The surgical digital reproduction and assistance system (1) according to claim 1, characterized in that the surgical digital reproduction and assistance system (1) is adapted to be operated in a live mode in which the first data modality (6) and the second data modality (8) are provided and temporarily stored in time real-time and the preset time of the view creation unit (14) is operated together continuously in time real-time in order to provide, inter alia, a live video recording and to provide a virtual reproduction and assistance of the operation to a user in a position-independent manner, in particular a remote surgeon at a location remote from the operating room to perform a remote surgical operation at the patient (P), or the surgical digital reproduction and assistance system (1) is adapted to be operated in a postoperative mode in which a recorded and stored operation is visually reproduced postoperatively with at least the first data modality (6) and the second data modality (8) in order to confirm, inter alia, a correct implementation of the surgical operation or to provide a virtual surgical operation training to a user in a position-independent and time-independent manner.

3. Surgical digital re-creation and assistance system (1) according to claim 1 or 2, characterized in that at least the first data modality (6) and / or the second data modality (8) have, in addition to the time specification and at least the position specification, the following data as digital data, respectively: - a 2D recording in the form of an image or video stream, - a 3D recording in the form of an image or video stream, - a 3D point cloud recording in the form of an image or video stream, - a graph of at least one electrical signal, - a voice note, and / or - a position information of a surgical instrument (24), in particular together with an instrument type of the instrument.

4. Surgical digital re-pro and assistance system (1) according to any of the preceding claims, characterized in that, The surgical digital reproduction and assistance system (1) has at least one computer and / or portable mobile device, in particular a tablet, which forms at least one of the visual user interfaces (20) and outputs the created views (18) via a presentation device (22) connected there, in particular a display.

5. Surgical digital re-pro and assistance system (1) according to any of the preceding claims, characterized in that, The presentation device (22) is a 2D display or a 3D display or a head-up display, in particular a virtual reality head-up display or an augmented reality head-up display or a MR head-up display.

6. Surgical digital re-pro and assistance system (1) according to any of the preceding claims, characterized in that, The surgical digital reproduction and assistance system (1) has a digital surgical system (100) with a visualization system (102), a navigation system (104) and a robot (106) with a robot arm (108) which is adapted to record and provide the first data modality (6) and the second data modality (8) to the central data information system (2) spatially related and in time with each other.

7. Surgical digital re-pro and assistance system (1) according to any of the preceding claims, characterized in that, The surgical digital reproduction and assistance system (1) has a navigation system (104) and for providing the first data modality (6) and / or the second data modality (8) also: - a digital surgical microscope (28) for creating 2D recordings or 3D recordings, in particular in the form of videos, the position of the recordings of the videos, in particular the orientation of the recordings, being tracked by the navigation system (104); - a surgical endoscope (30) for creating 2D recordings or 3D recordings, in particular in the form of videos, the position of the recordings of the videos, in particular the orientation of the recordings, being tracked by the navigation system (104); - a fluorescence image generator (32) for creating fluorescence recordings in the form of 2D recordings or 3D recordings, the position of the recordings of the fluorescence recordings, in particular the orientation of the recordings, being tracked by the navigation system (104); - at least one staff camera (34), for example an operating room overview camera or a body camera tracked in space, for providing a video and / or a 3D point cloud image stream related to the navigation system (104); - a patient camera (36) whose optical axis and whose field of view are aligned to the patient (P) in order to detect and provide recordings, in particular a video and / or a 3D point cloud image stream, related to the navigation system (104); - a microphone (38) for detecting, storing and later reproducing recordings, in particular voice notes, in time via an audio device of the user interface (20); - a navigation system (104) with navigation data, in particular a 3D volume data set, preferably a preoperatively and / or intraoperatively detected CT recording and / or an MRT recording; - a robot (106) with a robot arm (108), wherein the coordinates of the robot arm (108) are tracked by the navigation system (104) or by a tracking system inside the robot; - tracking of a surgical instrument (24), of an instrument type of the surgical instrument and of an instrument position, preferably of an instrument tip, in particular of an instrument orientation, by means of the navigation system (104).

8. The surgical digital re-creation and assistance system (1) according to claim 7, characterized in that, The surgical digital reproduction and assistance system (1) has, in order to provide at least one further data modality, at least one of the following devices: - a nerve monitoring device (40) for detecting nerve monitoring data, in particular electrical signals or recordings, the position, in particular the orientation, of the nerve monitoring data of the nerve monitoring device being tracked by means of the navigation system (104), - a biopsy device (42) for creating tissue data, in particular microscopic 2D images, the position, in particular the orientation, of the histology of the tissue data being tracked by means of the navigation system (104), - an ultrasound device for creating ultrasound recordings, preferably 2D or 3D recordings, the position, in particular the orientation, of the recordings of the ultrasound device being tracked by means of the navigation system (104).

9. Surgical digital re-pro and assistance system (1) according to any of the preceding claims, characterized in that, In the data information system (2) the preoperative treatment plan and / or the postoperative treatment report of the surgery are also stored, which are included as a basis for the user presets (16) and / or the created views (18).

10. The surgical digital re-creation and assistance system (1) according to claim 9, characterized in that, The view creation unit (14) inserts in the views (18) a timeline (26) of the surgical stages of the treatment plan and / or the treatment report of the surgery recorded on the timeline (26) in order to be able to display each surgical step visually and quickly accessible.

11. Surgical digital re-creation and assistance system (1) according to any one of claims 9 or 10, characterized in that, The view creation unit (14) is adapted to automatically determine the main data modality (6, 8) belonging to the current surgical stage of the treatment plan and / or the treatment report from at least the first data modality (6) and the second data modality (8) and to display this main data modality (6, 8) enlarged in relation to the other data modality (8, 6) in the view (18).

12. Surgical digital re-pro and assistance system (1) according to any of the preceding claims, characterized in that, In the data information system (2) three-dimensional navigation data are also stored as a 3D data set, in particular a preoperative CT recording and / or MRT recording of at least one body section of the patient (P), in order to virtually penetrate through the tissue volume of the patient during the reproduction of the surgery, preferably intraoperatively.

13. A method for synchronously recording a first data modality (6) and a second data modality (8) for a surgical digital reproduction and assistance system, in particular according to any one of the preceding claims 1 to 12, characterized in that The following steps: tracking (S1) of at least one spatial position, in particular an orientation, of the first data modality (6) by means of a navigation system (104) relative to a common global coordinate system, detecting and storing (S2) of the digitally provided data, in particular recordings, of the first data modality (6) in a central data information system (2) together with the tracked position and with a time specification of a uniform time, tracking (S3) of at least one position, in particular an orientation, of the second data modality (8) in space relative to a common global coordinate system by means of a navigation system (104), detecting and storing (S4) of the provided data, in particular recordings, of the second data modality (8) in the central data information system (2) together with the tracked position and with a time specification of a uniform time, creating (S5), by a view creation unit (14), a visual view (18) having the first data modality (6) and the second data modality (8) at a preset time based on a user preset (16) having at least one preset time, transmitting (S6) the view (18) to a user interface (20), and outputting (S7) the view (18) by a presentation device (22).

14. A computer readable storage medium comprising instructions which, when implemented by the computer, cause the computer to carry out the method steps of claim 13.

15. A computer program comprising instructions which, when implemented by the computer, cause the computer to carry out the method steps of claim 13.

Images