Surgical digital playback- and assistance system, and synchronised recording method

Abstract

The present invention relates to a surgical digital playback- and assistance system (1) for temporally correlated transmission of a surgical procedure of a patient (P) or for virtual training or evaluation of a surgical procedure of a patient (P), comprising: a central data information system (2), in which at least one first data modality (6) and a second data modality (8), different from the first data modality (6), are stored or at least temporarily stored, both the first data modality (6) and the second data modality (8) comprising at least one piece of position information with regard to a standardised common coordinate system (10) of the patient (P), and thus being correlated with one another at least in respect of the position, and further comprising a timestamp (12), in order to also be synchronised with one another in respect of a standardised absolute time; digitally providing a view adjusting unit (14), which is adapted to generate, on the basis of a user specification (16) having at least one time specification, a visual view (18) having the first data modality (6) and the second data modality (8) at the time specification; and at least one visual user interface (20), to which the adjusted view (18) is digitally transmitted and is output and displayed to a user by means of a display device (22) in order to evaluate in particular surgical phases or in order to select segments of a recorded operation and to playback the at least first and second data modality in a mode synchronised with regard to space and time. The present invention also relates to a synchronised recording method and to a computer-readable storage medium and to a computer program according to the alternative independent claims.

Description

[0001] Surgical Digital Playback and Assistance System and Synchronized Recording Method

[0002] Description

[0003] Technical area

[0004] The present disclosure relates to a surgical digital playback and assistance system / virtual surgical system for the temporally correlated or temporally synchronized, in particular time-up-to-date, transmission of a surgical procedure on a patient or for the virtual training or evaluation of a surgical procedure on a patient. The present disclosure can also be said to relate to a virtual surgical training system and a virtual surgical assistance system. In addition, the present disclosure relates to a synchronized recording method, a computer-readable storage medium, and a computer program according to the preambles of the independent claims.

[0005] Technical background of the revelation

[0006] For surgical procedures on a patient, so-called surgical video platforms or surgical platforms for shared use (team platforms) or platforms for sharing medical data are currently known. All of these solutions mentioned above share medical data, usually in the form of video streams / videos / video streaming, and enable retrospective observation of procedures (e.g., recording the operation and subsequent playback). In special versions, they also allow shared use during a surgery in real time. In this case, the various video recordings from video cameras set up at different locations are recorded via a central control unit, and a view is created in which the various video streams are arranged side by side. This view is then displayed on an operating room display.However, it becomes problematic when different modalities of "recordings" are to be used together, i.e. when a system is to be used that comprises several modalities that are also intended to interact with one another. The current problem is that if other medical data is to be integrated in addition to video cameras, these are only available in isolation from one another and without any reference to one another. You can imagine this like this: an initial video is created using an operating microscope (in isolation). Next to it, a further (isolated) video is created using an overview camera. The surgeon can later import these videos onto a computer and play them back one after the other or side by side, but the recordings remain unrelated to one another.

[0007] Summary of the present disclosure

[0008] It is therefore the object of the present disclosure to avoid or at least mitigate the disadvantages of the prior art and, in particular, to provide a surgical digital playback and assistance system, a synchronized recording method, a computer-readable storage medium, and a computer program that enable a digitized surgical procedure / operation to be observed and provide a user with the option of intuitively switching between, and viewing, different modalities. A further object can be seen in providing a central data system that can capture and store data from different modalities with a correlation to one another in order to be able to later (data-linked) visually display this data for the user as needed.One subtask is to connect the data from the different modalities both in terms of space and time.

[0009] The objects of the present disclosure are achieved according to the invention by the features of claim 1 with regard to a surgical, digital reproduction and assistance system, by the features of claim 13 with regard to a synchronized recording method, by the features of claim 14 with regard to a computer-readable storage medium, and by the features of claim 15 with regard to a computer program.

[0010] A basic idea of ​​the present disclosure thus provides for providing at least two different (data) modalities for visualization, which are synchronized or correlated with each other in terms of time and position. Such a (surgical) system makes it possible to observe a digitized operation (i.e., a digitally recorded operation) that comprises several modalities, and thus data modalities, that are synchronized in time and located in a common coordinate system. The system, in particular, allows the user to switch between the modalities, select sections (chapters) of the recorded operation, and play all modalities in a synchronized mode (time and space).

[0011] The surgical digital rendering and assistance system of the present disclosure not only synchronizes the data temporally, but also provides spatial synchronization or correlation, meaning the modalities are tracked in a common space or a common coordinate system. Thus, the present system and method enables complete immersion in a surgical procedure and thus the subject of surgery by examining all modalities (data modalities) individually, but also in connection with one another (in a relationship). This allows the user to experience a surgery as if they were in the operating room themselves. In particular, automatic time synchronization is provided.

[0012] In particular, a virtual surgery system (surgical digital rendering and assistance system) is proposed in which two or more surgical modalities (data modalities) are imported from a central storage system. The surgical modalities may be, for example, image data, volumetric data, position data, electrical signal data, or speech data, but in which the at least two modalities are correlated or synchronized both temporally and spatially. Furthermore, the virtual surgery system provides a user interface that enables the surgical procedure or case to be reproduced and the at least two modalities to be viewed.This allows the data modalities to create visual information on the context of the surgical procedure, allowing the user to directly evaluate specific surgical phases or steps, regardless of location, in particular both location- and time-independent.

[0013] For example, the system of the present disclosure can be used by learning surgeons to remotely and virtually treat an expert's surgical procedure or medical case. The system can also be used to exchange cases between experts either offline (after the case) or online (during the case), such as in the form of a surgical video analysis.

[0014] In other words, a surgical (virtual) digital playback and assistance system for the temporally correlated, in particular time-updated, transmission of a surgical procedure on a patient or for the virtual training or evaluation of a surgical procedure on a patient is disclosed and provided, comprising: a central data information system / data information unit, in particular with a storage unit in which at least a first (medical) data modality and a second (medical) data modality different from the first data modality are stored or at least temporarily stored, wherein both the first data modality and the second data modality have at least one (absolute) position indication, in particular location indication, with respect to a uniform common (global) coordinate system (KOS) of the patient and thus at least with respect to the position,in particular with regard to position, are correlated (synchronized) with one another and further comprise a time stamp in order to be synchronized or correlated with one another with regard to a uniform absolute time, a view creation unit adapted to digitally generate a visual view with the first data modality and the second data modality at (exactly) the specified time based on a user specification with at least one time stamp, and at least one visual user interface to which the created view is digitally transmitted and output and displayed to a user by means of a display device in order to evaluate surgical phases or to select sections of a recorded operation and to reproduce the at least first and second data modalities in a mode synchronized with respect to space and time. The transmission does not necessarily have to be in real time,It is also possible to transfer all the data from the operation after the operation in a so-called offline mode. The data from the operation only needs to be temporally and spatially correlated.

[0015] The term "position" refers to a geometric, spatial position in three-dimensional space, which is specified in particular by means of coordinates in a Cartesian coordinate system. In particular, the position can be specified by three coordinates: X, Y, and Z.

[0016] The term "orientation," in turn, indicates an alignment (e.g., position) in space. One could also say that orientation indicates an orientation with a direction or rotation specification in three-dimensional space. In particular, orientation can be specified using three angles.

[0017] The term "attitude" encompasses both position and orientation. In particular, attitude can be specified using six coordinates: three position coordinates X, Y, and Z, and three angular coordinates for orientation.

[0018] The term data modality in this case means that digital data, in particular images that are provided as a data modality, is recorded using a modality such as a surgical microscope, an ultrasound device, or a camera. The position of the data modality in this case means that a position of, for example, the image is to be recorded and saved. For example, a surgical microscope can be spatially tracked / tracked relative to the patient's uniform coordinate system, a position of the surgical microscope head can be determined using the optical system, and the position of the image can finally be determined (indirectly via the surgical microscope) via a distance from the optical system to the image. In individual modality cases, it may even be sufficient to approximately determine the position of the device and use it as the approximate position of the data modality.One could also say that a data modality almost always includes a recording device for recording the data. Data can, of course, also be imported, and the device does not necessarily have to be part of the system. What is important is that both a spatial specification in the form of at least one position in the unified coordinate system, in particular a position, such as the position of an instrument, is stored in the coordinate system together with a (synchronized) time reference, such as March 1, 2022, 17:53:33 (with a timestamp for an image, and of course with a continuous time reference for a video).

[0019] The term temporally correlated or synchronized in this case means that the data is synchronized with a central, absolute (system) time. This means that a uniform time reference is used when recording the data (in all modalities). For example, an internal clock can be used for each modality, whereby these internal clocks are synchronized to the second, in particular to the millisecond, and display the same time. For example, an internal radio clock can be used in each case, which is set via radio such as GPS, or a central clock can set the internal clocks via a data connection (e.g. via LAN or WLAN). Alternatively, the data can be transmitted in real time (with negligible latency) to a central unit, which then assigns a time to the data and thus correlates it with each other in time.

[0020] Advantageous embodiments are claimed in the subclaims and are explained in particular below.

[0021] According to one embodiment, the surgical digital playback and assistance system can be adapted to be operated in a live mode, in which the first and second data modalities are provided and buffered in real time and the time specification for the view creation unit is continuously updated in order to provide a video recording and to provide a user with a location-independent virtual playback of the procedure, in particular to enable a surgeon at a remote location to perform an operation on the patient in the operating room, or the surgical digital playback and assistance system can be adapted to be operated in a postoperative mode, in which the recorded and stored procedure is visually repeated at a later time,For example, to confirm the correct execution of a procedure or to provide a user with virtual training independent of location and time. In a first mode, the surgical digital playback and assistance system can be adapted to transmit a real-time operation in an operating room to, for example, another hospital in which a surgeon is located who is assisting or performing the telesurgical operation. By configuring the transmission to the user interface, various technologies for forwarding digital data can be used, including secure transmission via the Internet to a client station, enabling secure and timely visual data transmission. The view data can also be "copied" to provide multiple users with a corresponding user interface, for example, with a display.so that even in a small operating room, several users can participate virtually.

[0022] Preferably, the surgical digital rendering and assistance system comprises a server with the central data information system and a plurality of clients, each with a user interface. The surgical digital rendering and assistance system is adapted to request an individual user specification from each user interface, create a view based on this user specification, and then send this customized view to the user interface. This allows each user, such as surgeons from different disciplines, to select precisely the data modalities that offer the user particular benefits, for example, to have them displayed in an enlarged format or to review them retrospectively. This allows a network to be established with a central server and a plurality of clients.

[0023] In particular, the data connection between the server and the clients is established via a secure intranet connection or a secure internet connection. Preferably, the at least first data modality and / or second data modality can include, in addition to the time information and at least the position information:

[0024] - a 2D recording (image or stream),

[0025] - a 3D recording (image or stream),

[0026] - a 3D point cloud image (image or stream),

[0027] - a graph of electrical / electrophysiological signals,

[0028] - a voice note, and / or

[0029] - Position information of a surgical instrument, in particular of an instrument type. The data modalities can therefore include, in particular, 2D recordings (images or streams), 3D recordings (images or streams), SD point cloud recordings (images or streams), electrophysiological signals, voice notes, and / or position information of surgical instruments.

[0030] The data modalities can preferably include recordings from a (surgical) personal camera, a recording from a patient camera, a microscope recording (in particular a microscope image), an endoscopy recording (in particular an endoscope image), a fluorescence recording (in particular a fluorescence image), navigation recordings (in particular a navigation image), a histology recording (in particular a histology image), an ultrasound recording (in particular an ultrasound image), an electrophysiological signal and / or a voice note.

[0031] The surgical digital playback and assistance system has, in particular, a user input interface for processing the data stored in the central data information system, in particular for supplementing with voice memos or for supplementing annotations or for supplementing explanations in the (selected) modalities.

[0032] In particular, the surgical digital playback and assistance system can comprise at least one computer and / or a portable, mobile device, in particular a tablet, which forms at least one of the visual user interfaces and outputs the created view via the presentation device connected thereto, in particular a display. The user interface can thus run in particular on a desktop computer or on a mobile device (such as a smartphone or tablet).

[0033] According to a further embodiment, the display device can be a 2D display, a 3D display, or a head-up display, in particular a virtual reality head-up display (VR), an augmented reality head-up display (AR), or an MR head-up display. The visual information can thus be presented via a 2D display such as a surgical monitor, a 3D display, such as a monitor with associated 3D glasses, or a head-up display (VR, AR, MR).

[0034] According to one embodiment, the surgical digital playback and assistance system can comprise a digital surgery system with a visualization system, a navigation system, and a robot with a robotic arm, which is adapted to record the first and second data modalities in a spatially correlated and temporally synchronized or correlated manner and to provide them to the central data information system. The surgical digital playback and assistance system (virtual surgery system) obtains the data acquired by the digital surgery system, in particular via a cloud platform. The data of the data modalities are generated by the digital surgery system, which comprises a visualization system, a navigation system, and a robotic arm; of course, additional data can also be read or imported via computer.

[0035] In particular, a preoperative treatment plan and / or a postoperative treatment report for the corresponding procedure can also be stored in the data information system, which is also included in the user specification or the view. In other words, the system can in particular have access to a preoperative treatment plan and / or a postoperative treatment report for the corresponding case. Parts of this information from the treatment plan can be displayed in the view, for example which phases or steps should be carried out when and in which order. This means that all relevant information is displayed to the surgeon centrally and as needed. Preferably, the view creation unit can insert a timeline into the view with the surgical phases of the surgical procedure plotted therein in order to enable quick access to each surgical step.In particular, the system displays the phases of the surgical procedure and enables quick access to each surgical step, for example by entering a time limit via a touch display and creating a view at that time (time limit).

[0036] According to a further embodiment, the view creation unit can be adapted to automatically determine an associated primary data modality from the at least first and second data modalities based on the current surgical phase and to display this primary data modality in the view enlarged compared to the other data modality. The system therefore automatically determines the primary modality for the current surgical phase and displays the primary modality in an enlarged size. For example, the phase "Examination of tissue with surgical microscope" can be stored in the treatment plan, and the view creation unit can be adapted to display the image from the surgical microscope enlarged in the created view, for example so that it takes up 50% of the view, and the remaining data from the remaining data modalities are displayed in a reduced size at other locations in the view.

[0037] In particular, three-dimensional navigation data can be stored in the data information system as a 3D dataset, in particular preoperative CT scans and / or MRI scans of at least one body section of the patient, in order to move through a tissue volume of the patient, preferably intraoperatively during the replay of the operation. The navigation data can thus be available as a 3D dataset and make it possible to scroll or "virtually" move through the (tissue) volume at any time while observing the operation. This provides the surgeon with another important data modality. Preferably, the surgical digital replay and assistance system can comprise a navigation system and, for providing the first data modality and / or second data modality, further comprise (as a recording device):

[0038] - a digital surgical microscope for creating 2D or 3D images, in particular in the form of images or videos (e.g. in HD or 4K resolution), the recording position, in particular the recording position (indirectly via a tracked microscope head) of which is tracked via the navigation system;

[0039] - a (surgical) endoscope for creating 2D or 3D images, in particular in the form of videos (e.g. HD or 4K resolution), the recording position, in particular the recording location (indirectly via the tracked endoscope) of which is tracked via the navigation system;

[0040] - a fluorescence imager for creating fluorescence images (preferably coupled with a microscope image and / or endoscope image (especially videos)), in particular 2D or 3D images (e.g. HD or 4K resolution), whose image position, in particular image orientation (indirectly via the tracked fluorescence imager) is tracked via the navigation system

[0041] - at least one operating room personal camera (such as an operating room overview camera or a body camera tracked in the room) to provide a video and / or a 3D point cloud image stream, correlated with the navigation system, in particular with an option to create representation symbols (avatars) for the respective operating room personnel (for data protection reasons)

[0042] - a patient camera (whose optical axis and field of view are directed towards the patient) to capture and provide, in particular, a video (image stream) and / or a 3D point cloud image stream, correlated with the navigation system;

[0043] - a microphone to record and save a voice note at a specific time and to play it back later via an audio device;

[0044] - a navigation system with navigation data, in particular 3D volume data sets, preferably preoperatively and / or intraoperatively acquired CT scans and / or MRI scans (e.g., in DICOM format); - a robot with a robot arm, with coordinates of the robot arm, tracked via the navigation system;

[0045] - Surgical instruments, the instrument type and position, in particular location, of which are tracked by the navigation system. In other words, the surgical digital playback and assistance system and the central data information system preferably comprise the following modalities (and formats): a digital surgical microscope, in particular 2D or 3D images, in particular videos (e.g., in HD or 4K resolution), tracked by the navigation system; an endoscope, in particular 2D or 3D images (videos) (e.g., HD or 4K resolution), tracked by the navigation system; a fluorescence imager for fluorescence images, preferably coupled with microscope and / or endoscope images (videos), 2D or 3D images (e.g., HD or 4K resolution), tracked by the navigation system;at least one operating room personal camera (such as an overview camera), video image stream and / or 3D point cloud image stream, correlated with the navigation system, in particular with a possibility for creating representation symbols (avatars) for the respective operating room personnel (for data protection reasons); a patient camera, in particular for capturing and providing a video (-streamZ image stream) and / or a 3D point cloud image stream, correlated with the navigation system; a microphone for capturing and storing voice memos and later playing them back via an audio device; a navigation system with navigation data, in particular 3D volume data sets, preferably preoperatively and / or intraoperatively acquired CT images and / or MRI images (for example in DICOM format); a robot with a robot arm, with coordinates of the robot arm, tracked via the navigation system;Surgical instruments, whose instrument type and position, especially orientation, are tracked using the navigation system. These various (modality) recording devices can capture data from different modalities and store them in a temporally and spatially synchronized manner, and later make them available visually via a single view.

[0046] According to a possibly completely independently claimable aspect of the present disclosure, which is claimable in a further application, the data (or (data of the) data modalities) can in particular be acquired and generated by a digital surgical system comprising a microscope and / or an endoscope and / or a navigation system and / or a robotic instrument guidance and / or video cameras that track and record the operating room staff and / or the patient and / or a microphone for voice annotations and / or an interface to third modalities such as neuromonitoring, histology or an ultrasound (recording device).Furthermore, the digital surgery system can comprise a control unit adapted to perform a spatial localization for each data modality with respect to at least one position, in particular a location, preferably via the navigation system, and to record at least one time reference associated with the data modality (a central absolute time). In particular, a navigation system (with a tracking system such as a navigation camera) spatially correlates all data by tracking the positions of the modalities in space during the operation. All data is recorded centrally in the digital surgery system, automatically provided with a common timestamp or time reference, and stored, in particular stored on a server or a cloud system.By processing multiple, different modalities, the data can also be linked with semantics, i.e., with the identification of the surgical step and the creation of a script of the recorded operation. According to the present disclosure, two systems can be identified, each fulfilling an independent function and interrelated with each other. The first system, the digital surgical system, creates the data modalities and synchronizes them in time and space. The second system, the surgical digital playback and assistance system, in turn accesses this synchronized data and makes it available to a user as needed.

[0047] Preferably, the surgical digital rendering and assistance system can comprise a digital twin that, particularly in an avatar mode, represents all recorded devices in the operating room. According to a further embodiment, the surgical digital rendering and assistance system can further comprise, for providing at least one further (third) data modality:

[0048] - a neuro-monitoring device for recording neuro-monitoring data, in particular electronic signals or recordings (in particular images), the neuro-monitoring position, in particular location (indirectly via the neuro-monitoring device) of which is tracked via the navigation system;

[0049] - a biopsy device for creating histology data, in particular microscopic 2D images (e.g. in DICOM format), the histology position, in particular location (indirectly via the tracked biopsy device) of which is tracked via the navigation system;

[0050] - an ultrasound device for creating ultrasound images, preferably 2D or 3D images, whose recording position, in particular recording location (indirectly via the tracked ultrasound device), is tracked via the navigation system. The following data modalities can also optionally be stored: neuromonitoring data, in particular electrophysiological signals or recordings (e.g., images) that are spatially localized via the navigation system; histology data, in particular microscopic 2D images (e.g., in DICOM format), that are spatially localized via the navigation system; and / or ultrasound images, preferably 2D or 3D images (e.g., images), that are spatially localized via the navigation system.

[0051] In particular, the view generation unit may be adapted to arrange the data modalities from macro, such as a surgical staff camera, to micro, such as a microscope image, to nano, such as a histology image, to provide a user with an intuitive and structured visual view.

[0052] The user interface can preferably be a web-based software platform that allows surgical cases to be experienced from virtually anywhere. Using only a browser or a software application on a device as the user interface, a (secure) connection to a server (via the intranet or internet) can be established, and the data can be provided accordingly. In particular, the digital surgical system can be fed with temporally synchronized and spatially correlated multimodal data via data / video interfaces, such as wired Ethernet, HDMI, or SDI.

[0053] The object is achieved with regard to the synchronized recording method for creating a spatially and temporally synchronized first data modality and second data modality for a surgical digital playback and assistance system, in particular according to the present disclosure, in that the method comprises the steps:

[0054] Tracking by a navigation system of at least one position, in particular location, of the first data modality in space relative to a common global coordinate system;

[0055] Capturing and storing the provided data of the first data modality, in particular a recording, together with the tracked position and together with a time indication of a uniform time, preferably together with a semantics of a surgical step;

[0056] Tracking by a navigation system of at least one position, in particular location, of the second data modality in space in / relative to the common global coordinate system;

[0057] Capturing and storing the provided data of the second data modality, in particular a recording, together with the tracked position and together with a time indication of a uniform time;

[0058] Creating, by a view creation unit based on a user specification with at least one time specification, a visual view with the first data modality and the second data modality at the specified time specification

[0059] Outputting the view through a display device.

[0060] The objects are achieved with respect to a computer-readable storage medium and with respect to a computer program, i.e., in that these comprise instructions which, when executed by a computer, cause the computer to perform the method steps according to the present disclosure. The disclosure regarding the surgical digital playback and assistance system of the present disclosure applies equally to the synchronized recording method according to the present disclosure and vice versa, in particular through analogous features of the device and method.

[0061] Short description of the characters

[0062] The invention is explained in more detail below using preferred embodiments with the aid of figures. They show:

[0063] Fig. 1 is a schematic view of a surgical digital playback and assistance system according to a first preferred embodiment, with a visual view of different data modalities generated by a digital surgical system and synchronized spatially and temporally;

[0064] Fig.2 shows schematically a functionality of a data storage with spatial and temporal synchronization;

[0065] Fig. 3 shows a schematic view of a data exchange between the digital surgery system and the surgical digital playback and assistance system according to another preferred embodiment;

[0066] Fig. 4 shows a possible application of the surgical digital playback and assistance system;

[0067] Fig. 5 shows a user interface of the surgical digital playback and assistance system of another preferred embodiment; and

[0068] Fig. 6 is a flowchart of an embodiment of a synchronized recording method according to the present disclosure.

[0069] The figures are schematic in nature and are intended only to aid understanding of the invention. Like elements are designated by the same reference numerals. The features of the various embodiments may be interchanged.

[0070] Detailed description of preferred embodiments

[0071] Fig. 1 shows a schematic view of a surgical digital playback and assistance system 1 for the time-correlated, here time-updated transmission of a surgical procedure on a patient P or for the virtual training or evaluation of a surgical procedure on a patient P, according to a first preferred embodiment of the present disclosure.

[0072] The surgical digital playback and assistance system 1 has a central data information system 2 with a storage unit 4, in which at least a first data modality 6 and a second data modality 8, which is different from the first data modality 6, are or will be stored in a database. Both the first data modality 6 and the second data modality 8 have at least one position specification, in particular a location specification, with respect to a uniform, common coordinate system 10 of the patient P. In this embodiment, an associated (X1, Y1, Z1) coordinate is stored in an entry in the database for the recording of the first data modality, and an associated (X2, Y2, Z2) coordinate is also stored for the second data modality. In the present case, the database is therefore designed with a column of position information.Through this position information, the two data modalities are correlated with each other in terms of position, and if an orientation is also specified, for example, in another Dantebank column, especially in terms of orientation. Thus, for each recording of each data modality, there is (at least) a uniform position information, and the data modalities are spatially correlated or synchronized.

[0073] Furthermore, a time specification 12 is stored for each data modality (for each recording) in order to be synchronized with each other with regard to a uniform absolute time. In particular, recordings can be created for each data modality 6, 8 at different times and saved with the time stamp or time specification. For example, a surgical microscope can create a recording as the first data modality at a first time t1 at a first position (of the recording) x_t1, y_t1, z_t1, be moved, and at a later, second time t2 create another recording at the second position x_t2, y_t2, z_t2. Both recordings are saved. In particular, the surgical microscope can also record a video or video stream, whereby a position of the recording is continuously tracked (indirectly via the surgical microscope head.The video can, as usual, be provided with a continuous time stamp and saved accordingly.

[0074] What is important in the present surgical digital playback and assistance system 1 is that the data of the two different data modalities 6, 8 are fed in and are synchronized and thus connected with each other both spatially and temporally.

[0075] Based on this data information system 2 with the stored data, a view creation unit 14 of the surgical digital playback and assistance system 1 can be digitally provided, which is adapted to generate a visual view 18 with the first data modality 6 and the second data modality 8 at the specified time based on a user specification 16 with at least one time specification. If, for example, the user performs an input via a touch display and enters a time specification for the surgical procedure, as well as further parameters of a user specification such as the selection of the data modality 6, 8 to be enlarged, the view creation unit 14 creates a view at precisely this point in time during the surgical procedure using the first and second data modalities. Similar to a fly-back machine or a time machine, it is thus possible to jump to a specific point in time and generate a view.This created view 18 is available digitally, in particular in the form of a single static digital image or a moving view, such as a video stream, starting from this time specification.

[0076] This created digital view 18 is transmitted to a visual user interface 20 of the surgical digital playback and assistance system 1 and visually output and displayed to a user by means of a display device 22. Based on this visual information, a user can be trained, or the user can evaluate surgical phases or select sections of a recorded operation and display at least the first and second data modalities in a mode synchronized with respect to space and time.

[0077] In addition to the first and second data modalities, Fig. 1 shows further, different modalities that are acquired by the digital surgical system and which are or are temporally and spatially correlated according to the present disclosure.

[0078] Fig. 2 shows a schematic view of how the data of the data modalities are stored, capturing the content of the (data) modality as well as the temporal and spatial correlation between the data.

[0079] Fig. 3 shows a schematic view of a possible data exchange. The digital surgery system records all data modalities during the operation. A treatment plan can be imported as input, for example, preoperative planning based on a CT and / or MRI dataset of patient P. The results of the operation are exported postoperatively in a treatment report. While the digital surgery system is positioned and arranged in an operating room, the data for the treatment plan and the treatment report are stored on a server, in particular a cloud system. This server, in particular the cloud system, then has a data connection to the surgical digital playback and assistance system 1 or the virtual surgical system according to the present disclosure. Fig.3 shows a data exchange between the digital surgical system for data acquisition and the surgical digital playback and assistance system 1 for visual data display.

[0080] Fig. 4 shows a surgical digital rendering and assistance system 1 according to an embodiment of the present disclosure. The digital surgical system is equipped with a monitor as a possible display device 22, a navigation system 104 for tracking instruments and for navigation, and a robot 106 with a robot arm 108, to the end of which a surgical microscope 28 is connected as an end effector. Here, an experienced surgeon performs an operation in the real world using the digital surgical system, and the digital surgical system records all data modalities along with positions or locations and a time indication in the real world. Thus, data is collected in the real world. These data modalities are provided to the central information system 2 of the surgical digital rendering and assistance system 1.A learning surgeon, in turn, can access the data from any location, in particular via a web portal. The surgical digital playback and assistance system 1 creates a digital twin for the learning surgeon, which represents a digital copy of the operating room (OR) and gives the learning surgeon the opportunity to immerse themselves in the operation. The learning surgeon can observe the surgical procedure either via a 2D screen, a 3D screen, or through VR or AR glasses / head-up display, as shown in Fig. 4, and immerse themselves in the operation. In particular, several learning surgeons can be provided with such a user interface in the form of VR glasses, so that several users can understand and be trained in a surgical procedure simultaneously and regardless of location.For example, such a visual user interface 20 can be provided to medical students or surgeons during their specialist training to consolidate their surgical skills and then train them individually.

[0081] Fig. 5 shows a user interface of a surgical digital playback and assistance system 1 of a further embodiment. In this embodiment, the display device is in the form of a touch display, and the view has several subfields with different data modalities. This user interface creates the impression of a cockpit in which all modalities are displayed. The primary (data) modality of the different data modalities is shown enlarged in the center, whereby the user can of course change the layout via the touch display and adapt it to their needs, for example, similar to a mobile phone by long-pressing the data modality field, after which the arrangement can be changed by dragging the window.At the bottom of view 18, a video player-like control unit is visible, which displays on a timescale when which modality was active and allows the user to jump to a specific step in the surgical case or procedure. The surgical case is also structured with a script and specific phases. Pictograms and a slider allow quick access to specific surgical steps. The preoperative plan (treatment plan) and the postoperative report (treatment report) are also accessible. A virtual, additionally inserted ring schematically indicates the spatial correlation between the data, such as the correlation between the focus of the microscope image (in particular the microscope image) and the corresponding position in the navigation dataset. The navigation dataset not only has a video (stream), but also always has a 3D dataset.This makes it possible to pause the surgical case at any time and virtually "move" through the 3D dataset, for example, to explore structures that are not directly visible. The modalities are arranged from macro views (e.g., staff camera) to micro views (e.g., microscope) to nano views (e.g., histology). This allows the user to "zoom in" and "zoom out" when using different data modalities.

[0082] Fig. 6 shows a flowchart of a synchronized recording method for creating a spatially and temporally synchronized first data modality 6 and second data modality 8 for a surgical digital playback and assistance system 1 according to the present disclosure.

[0083] In a first step S1, a tracking S1, by a navigation system 104, of at least one spatial position, in particular a location, of the first data modality 6 relative to a common global coordinate system 10 takes place.

[0084] In a second step S2, the digitally provided data of the first data modality 6, in particular a recording, is then captured and stored S2 in a central data information system 2, together with the tracked position and together with a time specification of a uniform time. In step S3, similar to step S1, whereby the order of the first and second data modalities can also be reversed, at least one position, in particular location, of the second data modality 8 in the spatially opposite or in the common global coordinate system is tracked by the navigation system 104.

[0085] In step S4, similar to step S2, the provided data of the second data modality (8), in particular a recording, are recorded and stored in the central data information system (2) together with the tracked position and together with the time specification of a uniform time.

[0086] Thereafter, in step S5, a view creation unit 14 creates a visual view 18 with the first data modality 6 and the second data modality 8 at the time specification based on a user specification 16 with at least one time specification.

[0087] In step S6, the created digital view 18 is then transmitted to a user interface 20.

[0088] Finally, in a step S7, the view 18 is output by a display device 22.

[0089] This synchronized recording process allows data from different modalities or different data modalities to be captured and stored in a synchronized manner in terms of space and time. These data modalities can then be played back at a later time, particularly at a user-specified time.

[0090] 1 Surgical digital playback and assistance system

[0091] 2 Data information system

[0092] 4 storage unit

[0093] 6 First data modality

[0094] 8 Second data modality

[0095] 10 Common global coordinate system

[0096] 12 Time indication

[0097] 14 View creation unit

[0098] 16 User default

[0099] 18 Created view

[0100] 20 visual user interface

[0101] 22 Display device

[0102] 24 Surgical instrument

[0103] 26 Timeline

[0104] 28 surgical microscope

[0105] 30 Endoscope

[0106] 32 fluorescence imagers

[0107] 34 Operating Room Personnel Camera

[0108] 36 patient cameras

[0109] 38 Microphone

[0110] 40 Neuro-monitoring device

[0111] 42 Biopsy device

[0112] 100 Digital Surgical System

[0113] 102 Visualization system

[0114] 104 Navigation system

[0115] 106 robots

[0116] 108 Robot arm

[0117] P Patient Step Track position of first data modality Step Acquire and save data of first data modality with position and time Step Track position of second data modality Step Acquire and save data of second data modality with position and time Step Create visual view Transfer visual view to user interface Output view through display device

Claims

Claims 1. Surgical digital playback and assistance system (1) for the temporally correlated transmission of a surgical procedure on a patient (P) or for the virtual training or evaluation of a surgical procedure on a patient (P), comprising: a central data information system (2), in particular with a storage unit (4) in which at least one first data modality (6) and a second data modality (8) different from the first data modality (6) are stored or at least temporarily stored, wherein both the first data modality (6) and the second data modality (8) have at least one position indication, in particular an orientation indication, with respect to a uniform common coordinate system (10) of the patient (P) and are thus correlated with one another at least with regard to the position, in particular with regard to the orientation, and further have a time indication (12),in order to be synchronized with each other with respect to a uniform absolute time, a view creation unit (14) which is adapted to digitally generate a visual view (18) with the first data modality (6) and the second data modality (8) at the time specified on the basis of a user specification (16) with at least one time specification, and at least one visual user interface (20) to which the created view (18) is digitally transmitted and output and displayed to a user by means of a display device (22), in particular to evaluate surgical phases or to select sections of a recorded operation and to display the at least first and second data modalities in a mode synchronized with respect to space and time.

2. Surgical digital reproduction and assistance system (1) according to claim 1, characterized in that the surgical digital reproduction and assistance system (1) is adapted to be operated in a live mode in which the first data modality (6) and second data modality (8) are provided and temporarily stored and the time specification for the view creation unit (14) runs continuously, in real time, in particular to provide a live video recording, and to provide a user with a location-independent virtual reproduction and assistance of the procedure, in particular to enable a surgeon at a location remote from an operating room to perform a telesurgical procedure on the patient (P), or the surgical digital reproduction and assistance system (1) is adapted to be operated in a postoperative mode in which the recorded and stored procedure with the at least first data modality (6) and the second data modality (8) can be visually reproduced postoperatively, in particular to confirm a correct execution of the surgical procedure or to provide a user with a virtual,to provide surgical training regardless of location and time.

3. Surgical digital reproduction and assistance system (1) according to claim 1 or 2, characterized in that the at least first data modality (6) and / or the second data modality (8) further comprise, as digital data, in addition to the time information and at least the position information: - a 2D recording in the form of an image or a video stream, - a 3D recording in the form of an image or a video stream, - a 3D point cloud recording in the form of an image or a video stream, - a graph of at least one electrical signal, - a voice note, and / or - position information of a surgical instrument (24), in particular together with an instrument type of the instrument.

4. Surgical digital reproduction and assistance system (1) according to one of the preceding claims, characterized in that the surgical digital reproduction and assistance system (1) has at least one computer and / or a portable, mobile device, in particular a tablet, which forms at least one of the visual user interfaces (20) and outputs the created view (18) via the presentation device (22) connected thereto, in particular a display.

5. Surgical digital playback and assistance system (1) according to one of the preceding claims, characterized in that the display device (22) is a 2D display or a 3D display or a head-up display, in particular a virtual reality head-up display or an augmented reality head-up display or an MR head-up display.

6. Surgical digital playback and assistance system (1) according to one of the preceding claims, characterized in that the surgical digital playback and assistance system (1) comprises a digital surgery system (100) with a visualization system (102), a navigation system (104) and a robot (106) with a robot arm (108), which is adapted to record the first data modality (6) and second data modality (8) spatially correlated and temporally synchronized to one another and to provide them to the central data information system (2).

7. Surgical digital playback and assistance system (1) according to one of the preceding claims, characterized in that the surgical digital playback and assistance system (1) has a navigation system (104) and, for providing the first data modality (6) and / or second data modality (8), further comprises: - a digital surgical microscope (28) for creating 2D images or 3D images, in particular in the form of videos, the position of the image, in particular the location of the image, of which is tracked via the navigation system (104); - a surgical endoscope (30) for creating 2D images or 3D images, in particular in the form of videos, the position of the image, in particular the location of the image, of which is tracked via the navigation system (104); - a fluorescence imager (32) for creating fluorescence images in the form of 2D images or 3D images, the position of the image, in particular the location of the image, of which image is tracked via the navigation system (104); - at least one operating room personal camera (34), such as an operating room overview camera or a body camera tracked in the room, for Providing a video and / or a 3D point cloud image stream that is correlated with the navigation system (104); - a patient camera (36) whose optical axis and field of view are directed towards the patient (P) in order to capture and provide an image, in particular a video and / or a 3D point cloud image stream, which is correlated with the navigation system 104; - a microphone (38) for capturing a recording, in particular a voice memo, in a time-bound manner, storing it and later playing it back via an audio device of the user interface (20); - the navigation system (104) with navigation data, in particular 3D volume data sets, preferably CT images and / or MRI images acquired preoperatively and / or intraoperatively; - a robot (106) with a robot arm (108), wherein the coordinates of the robot arm (108) are tracked via the navigation system (104) or via a robot-internal tracking system; - a surgical instrument (24), the instrument type and the instrument position, preferably an instrument tip, in particular instrument position, of which is tracked via the navigation system (104); 8. Surgical digital playback and assistance system (1) according to claim 7, characterized in that the surgical digital playback and assistance system (1) further comprises at least one of the following devices for providing at least one further data modality: - a neuro-monitoring device (40) for acquiring neuro-monitoring data, in particular electrical signals or recordings, the position of the neuro-monitoring data, in particular location, of which is tracked via the navigation system (104); - a biopsy device (42) for creating histology data, in particular microscopic 2D images, the position of the histology, in particular location, of which is tracked via the navigation system (104); - an ultrasound device for creating ultrasound images, preferably 2D or 3D images, the position of the image, in particular the location, of which is tracked via the navigation system (104).

9. Surgical digital playback and assistance system (1) according to one of the preceding claims, characterized in that a preoperative treatment plan and / or a postoperative treatment report of the surgical procedure is further stored in the data information system (2), which is included as a basis for the user specification (16) and / or the created view (18).

10. Surgical digital playback and assistance system (1) according to claim 9, characterized in that the view creation unit (14) inserts a timeline (26) in the view (18) with surgical phases of the treatment plan and / or treatment report of the surgical procedure entered on the timeline (26) in order to enable an intuitive visual display with accompanying rapid access to each surgical step.

11. Surgical digital playback and assistance system (1) according to one of claims 9 or 10, characterized in that the view creation unit (14) is adapted to automatically determine an associated primary data modality (6, 8) from the at least first data modality (6) and the second data modality (8) on the basis of the current surgical phase of the treatment plan and / or treatment report and to display this primary data modality (6, 8) in the view (18) in an enlarged manner compared to the other data modality (8, 6).

12. Surgical digital playback and assistance system (1) according to one of the preceding claims, characterized in that three-dimensional navigation data are also stored in the data information system (2) as a 3D data set, in particular preoperative CT images and / or MRI images of at least one body section of the patient (P), in order to virtually move through a tissue volume of the patient, preferably intraoperatively, during the playback of the operation.

13. Synchronized recording method for creating a spatially and temporally synchronized first data modality (6) and second data modality (8) for a surgical digital playback and assistance system, in particular a surgical digital playback and assistance system (1) according to one of the preceding claims 1 to 12, characterized by the steps: Tracking (S1), by a navigation system (104), at least one spatial position, in particular a location, of the first data modality (6) relative to a common global coordinate system, Capturing and storing (S2), in a central data information system (2), the digitally provided data of the first data modality (6), in particular a recording, together with the tracked position and together with a time indication of a uniform time, Tracking (S3), by a navigation system (104), at least one position, in particular a location, of the second data modality (8) in space relative to the common global coordinate system, Capturing and storing (S4) in the central data information system (2) the data provided by the second data modality (8), in particular a recording, together with the tracked position and together with the time indication of a uniform time, Creating (S5), by a view creation unit (14), based on a user specification (16) with at least one time specification, a visual view (18) with the first data modality (6) and the second data modality (8) at the time specification, Transmitting (S6) the view (18) to a user interface (20), and outputting (S7) the view (18) by a display device (22).

14. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the method steps of claim 13.

15. A computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method steps according to claim 13.

Images