Surgical digital playback support system and synchronized recording method

Abstract

The present invention relates to a surgical digital playback support system (1) for the temporally correlated transmission of a surgical procedure on a patient (P), or for virtual training or evaluation of a surgical procedure on a patient (P). The surgical digital playback support system comprises a central data information system (2), a view adjustment unit (14), and at least one visual user interface (20). The central data information system (2) stores, or at least temporarily stores, at least one first data modality (6) and a second data modality (8) different from the first data modality (6). Both the first data modality (6) and the second data modality (8) have at least one location information relating to a unified common coordinate system (10) of the patient (P), and therefore have timestamps (12) to be correlated with each other at least with respect to location, and further synchronized with each other with respect to standard absolute time. A digitally provided view adjustment unit (14) is configured to generate and digitally provide a visual view (18) including a first data modality (6) and a second data modality (8) at a specified time, based on a user specification (16) including at least one time specification. The adjusted view (18) is digitally transmitted to and output to at least one visual user interface (20) and displayed to the user by a display device (22), thereby allowing playback of at least the first and second data modalities in a spatially and temporally synchronized mode, particularly for evaluating surgical phases or selecting segments of recorded surgery. The present invention also relates to a synchronized recording method, a computer-readable recording medium, and a computer program according to alternative independent claims.

Description

Technical Field

[0001] The present disclosure relates to a surgical digital playback support system / virtual surgical system for temporally correlating or synchronizing surgical interventions on a patient, particularly for real-time transmission, or for virtual training or evaluation of surgical interventions on a patient. The present disclosure can also be said to relate to a virtual surgical training system and a virtual surgical support system. Furthermore, the present disclosure relates to a synchronization recording method, a computer-readable storage medium, and a computer program according to the beginning part of the independent claims.

Background Art

[0002] Currently, for surgical interventions on a patient, so-called surgical video platforms, surgical platforms for sharing (team platforms), or platforms for sharing medical data are known. All of these aforementioned solutions usually share medical data in the form of video streams / videos / video streaming, enabling retrospective observation of the intervention (e.g., recording and subsequent playback of the surgery). In special cases, real-time sharing during surgery is also possible. In this case, various images from video cameras installed at different locations are detected via a central control unit, and a view with various video streams arranged in parallel is created. This view is output via the operating room display.

[0003] However, problems arise when using a combination of different "recording" modalities, that is, when using a system that includes multiple modalities and coordinates with each other. Therefore, the current problem is that when integrating other medical data in addition to video cameras, they are only available separately and cannot be cross-referenced. This can be understood by assuming, for example, creating a first video via a (separate) surgical microscope. Furthermore, another (separate) video is created using a panoramic camera. The surgeon can later import these videos into a computer and play them sequentially or side by side, but the recordings remain unassociated with each other. [Overview of the project]

[0004] Accordingly, the object of this disclosure is to provide a surgical digital playback support system, a synchronized recording method, a computer-readable storage medium, and a computer program that avoid or at least mitigate the shortcomings of the prior art, in particular, that make digitized surgical interventions or surgeries observable and allow users to intuitively switch between different modalities and perform combined displays. Another object is to provide a central data system that can detect and store data from different modalities in an interconnected manner, so that they can later be visually displayed to the user (in a data-linked state) as needed. A partial object is to correlate data from modalities that are different in space and time.

[0005] The object of the disclosure of the present invention is achieved by the features of claim 1 with respect to the surgical digital regeneration support system according to the invention, by the features of claim 13 with respect to the synchronous recording method according to the invention, by the features of claim 14 with respect to the computer-readable storage medium according to the invention, and by the features of claim 15 with respect to the computer program according to the invention.

[0006] Therefore, the basic idea of ​​this disclosure is to provide for visualization at least two different (data) modalities that are synchronized or correlated in time and location.

[0007] Such a (surgical) system makes it possible to observe digitized surgery (i.e., digitally recorded surgery) that has multiple time-synchronized modalities and data modalities positioned in a common coordinate system. In particular, this system allows the user to switch between modalities, select segments (chapters) of recorded surgery, and play back all modalities in synchronized modes (time and space).

[0008] Therefore, the surgical digital regeneration support system of this disclosure not only synchronizes data temporally but also provides spatial synchronization or correlation. That is, modalities are tracked in a common space or coordinate system. In this way, the system and method allow all modalities (data modalities) to be examined individually and in relation to each other, enabling complete immersion in the surgical intervention and, consequently, the surgical object. This allows the user to experience the surgery as if they were in the operating room. In particular, automatic temporal synchronization is provided.

[0009] Therefore, a virtual surgery system (a digital surgical regeneration support system) is proposed. In this system, two or more surgical modalities (data modalities) are imported from a central memory system. The surgical modalities may be image data, volume data, positional data, electrical signal data, or audio data, but at least two modalities are temporally and spatially correlated or synchronized. Furthermore, the virtual surgery system is provided with a user interface that recreates a surgical intervention or case and makes at least two modalities visible. This allows the data modalities to provide visual information about the surgical intervention-related information (context), enabling the user to directly evaluate a specific surgical phase or step in a portable manner (particularly in a portable and time-independent manner).

[0010] For example, the system described herein can be used for trainee surgeons to remotely and virtually handle surgical interventions or medical cases performed by experienced surgeons. The system can also be used to exchange cases between experienced surgeons offline (after the case is completed) or online (during the case), for example, in the form of surgical video analysis.

[0011] In other words, a surgical (virtual) digital replay support system is disclosed and provided for the time-correlated (especially real-time) transmission of surgical interventions to patients, or for the virtual training and evaluation of surgical interventions to patients. The system comprises a central data information system / data information unit, a view adjustment unit, and at least one visual user interface. The central data information system / data information unit comprises, in particular, a storage unit, which stores, or at least caches, at least a first (medical) data modality and a second (medical) data modality distinct from the first data modality. Here, both the first and second data modalities have at least one (absolute) position index (especially a posture index) with respect to the patient's unified common (global) coordinate system (COS), thereby correlating (synchronizing) with respect to at least position (especially posture), and have time indexes (timestamps) for additional synchronization or correlation with respect to unified absolute time. The view adjustment unit is configured to generate and digitally provide a visual view at a (precisely) specified time index, using a first and second data modality, based on user specifications including at least one time index. At least one visual user interface is configured to receive, output, and display the generated view digitally to the user via a display device. This allows for evaluation of surgical phases or selection of recorded surgical segments, and playback of at least the first and second data modalities in a spatially and temporally synchronized mode. Transmission does not necessarily have to be in real time; it is also possible to transmit all surgical data in a so-called offline mode after the surgery. The surgical data only needs to be correlated in terms of time and space.

[0012] The term "position" refers to a geometric, spatial location in three-dimensional space, specifically identified using coordinates in a Cartesian coordinate system. In particular, a position is specified by three coordinates: X, Y, and Z.

[0013] On the other hand, the term "orientation" specifies the orientation in space (for example, the orientation in terms of position). Orientation can also be described as indicating alignment or rotation in three-dimensional space. In particular, orientation can be indicated by three angles.

[0014] The term "pose" encompasses both position and orientation. In particular, pose can be defined using six coordinates: three position coordinates X, Y, and Z, and three orientation coordinates.

[0015] In this specification, the term "data modality" means digital data detected via a modality such as a surgical microscope, ultrasound device, or camera, and in particular, recordings provided as a data modality. In this embodiment, the location of the data modality means the location of the object being detected and recorded. For example, a surgical microscope is spatially tracked / followed in space with respect to the patient's unified coordinate system, the orientation of the surgical microscope head is determined using the optical system, and ultimately the location of the object being recorded is determined by the distance from the optical system to the object being recorded (indirectly via the surgical microscope). In the case of individual modalities, it may be sufficient to simply determine the position of the device roughly and use that roughly as the location of the data modality. Also, it can be said that a data modality almost always includes a recording device for data recording. Of course, data can also be imported, and the device does not need to be part of the system. The important point is that a spatial representation of at least one position within the unified coordinate system (particularly the orientation of an instrument, etc.) is stored in the coordinate system along with a (synchronized) time index (for example, in the form of March 1, 2022, 17:53:33) within the unified coordinate system (of course, this is done using a timestamp in the case of images and a continuous time index in the case of videos).

[0016] In this context, "temporally correlated" or "synchronized" means that the data is synchronized with a central absolute (system) time. This means that a unified time index is used (across all modalities) when data is recorded. For example, each modality could use its own internal clock, which would be synchronized to the second (especially the millisecond) and display the same time. For example, an internal radio-controlled clock could be set via radio waves such as GPS, or a central clock generator could set the internal clock via a data connection (e.g., LAN or WLAN). Alternatively, for example, data could be transmitted to a central location in real time (with negligible delay), and the central location could then assign a time to this data to make it temporally correlated.

[0017] Advantageous embodiments are asserted in the dependent claims and are described in particular below.

[0018] According to one embodiment, the surgical digital playback support system is configured to operate in live mode, in which the first and second data modalities are provided and cached in real time, and the time setting of the view adjustment unit is continuously performed in real time. This is particularly to provide users with video recordings and virtual playback of interventions regardless of location, and in particular to enable remote surgeons to perform interventions on patients in the operating room. Alternatively, the surgical digital playback support system is configured to operate in postoperative mode, in which the recorded and saved interventions are visually redisplayed later, for example, to verify the accurate execution of the intervention or to provide users with virtual training in a location- and time-independent manner. In the first mode, the surgical digital playback support system may be configured to transmit operating room surgeries in real time to another hospital where a surgeon assisting or performing remote surgery is located. By configuring transmission to a user interface, digital data can be transferred using various technologies, including secure transmission over the internet to client sites, enabling secure and real-time visual data transmission. Furthermore, it is possible to "duplicate" the view data to provide a user interface that accommodates multiple users, for example, one equipped with a display, allowing multiple users to virtually participate in a confined operating room.

[0019] Preferably, the surgical digital reconstructive support system comprises a server having a central data information system and a plurality of clients, each having its own user interface. The surgical digital reconstructive support system is configured to request individual user specifications for each user interface, create a view based on these specifications, and transmit it to the individually adapted user interface. This allows each user (e.g., surgeons of different specialties) to precisely select data modalities that offer specific advantages, such as zooming in or reviewing them later. In this way, a network including a central server and multiple clients can be constructed. In particular, the data connection between the server and clients is established via a secure intranet connection or a secure internet connection.

[0020] Preferably, at least the first data modality and / or the second data modality may include, in addition to time and at least positional indices, 2D recordings (images or streams), 3D recordings (images or streams), 3D point cloud recordings (images or streams), graphs of electrical / electrophysiological signals, voice memos, and / or positional information of surgical instruments (particularly instrument type). Thus, the data modality may, in particular, include 2D recordings (images or streams), 3D recordings (images or streams), 3D point cloud recordings (images or streams), electrophysiological signals, voice memos, and / or positional information of surgical instruments.

[0021] The data modalities may preferably include (surgical) staff camera recordings, patient camera recordings, microscopic images (especially microscopic images), endoscopic recordings (especially endoscopic recordings), fluorescence recordings (especially fluorescence recordings), navigation images (especially navigation images), histological images (especially histological images), ultrasound images (especially ultrasound images), electrophysiological signals, and / or voice memos.

[0022] The surgical digital playback support system includes, in particular, a user input interface for editing data stored in a central data information system, and specifically, an interface for adding voice memos, adding annotations, or adding explanations in (selected) modalities.

[0023] In particular, a surgical digital regeneration support system may include at least one computer and / or a portable mobile device (especially a tablet) which forms at least one visual user interface and outputs the created view via a connected display device (especially a display). Thus, the user interface can operate on a desktop computer or a mobile device (such as a smartphone or tablet).

[0024] In further embodiments, the display device may be a 2D display, a 3D display, or a head-up display (particularly a virtual reality head-up display (VR), an augmented reality head-up display (AR), or an MR head-up display). Thus, visual information may be displayed via a 2D display such as a surgical monitor, a 3D display such as a monitor equipped with associated 3D glasses, or a head-up display (VR, AR, MR).

[0025] According to one embodiment, the surgical digital regeneration support system may include a digital surgical system having a robot equipped with a visualization system, a navigation system, and a robotic arm, the robot being configured to record first and second data modalities in a spatially correlated and temporally synchronized or correlated manner and provide them to a central data information system. The surgical digital regeneration support system (virtual surgery system) acquires the data detected by the digital surgical system, particularly via a cloud platform. The data modality data is generated by the digital surgical system equipped with a visualization system, a navigation system, and a robotic arm. Of course, additional data can also be read or imported by computer technology.

[0026] In particular, preoperative treatment plans and / or postoperative treatment reports for corresponding interventions can also be stored in a user-specified or view-based data information system. That is, the system can access preoperative treatment plans and / or postoperative treatment reports for corresponding cases. Some of this treatment plan information (such as which phases and steps should be performed, when, and in what order) may be included in the view. In this way, all necessary relevant information is displayed centrally to the surgeon.

[0027] Preferably, the view adjustment unit may insert a timeline into the view that displays the surgical phases of the surgical intervention to enable quick access to each surgical step. The system specifically displays the phases of the surgical intervention and enables quick access to each surgical step by allowing time specifications to be entered, for example, via a touch display, and creating a view corresponding to this time (specification).

[0028] According to a further embodiment, the view adjustment unit can be configured to automatically determine a primary data modality associated from at least the first and second data modalities based on the current surgical phase, and expand this primary data modality for display in the view relative to other data modalities. Thus, the system in particular automatically determines the primary modality at the current surgical stage and expands the display of the primary modality. For example, a phase of "tissue examination with a surgical microscope" may be stored in the treatment plan, and the view adjustment unit may be configured to expand and display the recording of the surgical microscope within the created view (e.g., to occupy 50% of the view, and the remaining data of the remaining data modalities are displayed in a reduced size at other positions within the view).

[0029] In particular, the three-dimensional navigation data can be stored in the data information system as a 3D data set (in particular a preoperative CT recording and / or an MRI recording of at least one part of the patient's body) for moving within the patient's tissue volume (preferably intraoperatively during the playback of the surgery). Thus, the navigation data may be available as a 3D data set and can be scrolled or "virtually" moved within the (tissue) volume at any time during the observation of the surgery. Thereby, the surgeon can utilize another important data modality.

[0030] Preferably, the surgical digital playback support system may include a navigation system and further comprises a digital surgical microscope, a (surgical) endoscope, a fluorescence imaging system, at least one operating room staff camera (such as a panoramic operating room camera or a body camera tracked within the room), a patient camera (with its optical axis and field of view directed towards the patient), a microphone, a navigation system, a robot, and surgical instruments to provide a first data modality and / or a second data modality (as one type of recording device). The digital surgical microscope produces 2D or 3D recordings (particularly in image or video format, e.g., HD or 4K resolution), and its recording position (particularly recording orientation) is tracked by the navigation system (indirectly via a tracked microscope head). The (surgical) endoscope is for generating 2D or 3D recordings (particularly in video format, e.g., HD or 4K resolution), and its recording position (particularly recording orientation) is tracked by the navigation system (indirectly via a tracked endoscope). A fluorescence imaging system is for creating fluorescence recordings (preferably combined with microscopic images and / or endoscopic recordings (particularly video)) (particularly 2D or 3D recordings (e.g., HD or 4K resolution)), the recording location (particularly recording posture) is tracked (indirectly via the tracked fluorescence imaging system) by the navigation system. At least one operating room personnel camera is for providing video and / or 3D point cloud image streams correlated with the navigation system (particularly, optionally creating representation symbols (avatars) for each operating room personnel (for data protection reasons)). A patient camera detects and provides video (image streams) and / or 3D point cloud image streams correlated with the navigation system. A microphone is for detecting, saving, and later playing back voice memos chronologically via audio equipment. The navigation system comprises navigation data (particularly 3D volumetric datasets (preferably CT and / or MRI recordings in DICOM format, preferably detected preoperatively and / or intraoperatively)).The robot has a robotic arm, and the coordinates of the robotic arm are tracked via a navigation system. The surgical instrument is tracked in the navigation system with respect to its instrument type and instrument position (especially the pose). In other words, the surgical digital playback assistance system and the central data information system preferably include a digital surgical microscope (especially 2D or 3D recording (especially, for example, a video with HD or 4K resolution)) tracked via the navigation system, an endoscope (especially 2D or 3D recording (video) (for example, with HD or 4K resolution)) tracked via the navigation system, a fluorescence imaging system for fluorescence recording (preferably a 2D or 3D recording (for example, with HD or 4K resolution) combined with a microscope and / or endoscope recording (video)) tracked via the navigation system, at least one operating room staff camera (for example, a panoramic camera) video stream and / or 3D point cloud image stream (especially with a display symbol (avatar) creation function for each operating room staff (for data protection)) associated with the navigation system, a patient camera associated with the navigation system and especially detecting and providing a video (stream or image stream) and / or 3D point cloud image stream, a microphone for detecting and storing voice memos and later playing them back via an audio device, a navigation system with navigation data (especially a 3D volume data set (preferably detected preoperatively and / or intraoperatively, for example, a CT recording and / or an MRI recording in DICOM format)), a robot with a robotic arm whose coordinates are tracked via the navigation system, and a surgical instrument whose instrument type and instrument position (especially the pose) are tracked in the navigation system. The modalities (and examples of formats) include these different recording devices. These different (modality) recording devices can record data of different modalities, store them synchronously in time and space, and later provide them visually via a view.

[0031] With respect to aspects of this disclosure that can be asserted entirely independently as needed (and asserted in a separate application), data (or data modalities (data)) may be detected and generated by a digital surgical system that has interfaces to third-party modalities such as microscopes, and / or endoscopes, and / or navigation systems, and / or guidance by robotic instruments, and / or video cameras for tracking and detecting surgical staff and / or patients, and / or microphones for voice memos, and / or neuromonitoring, histological imaging, ultrasound (recording devices). Furthermore, the digital surgical system may include a control unit configured to perform spatial localization for each data modality with respect to at least one position (particularly posture), preferably via a navigation system, and to record at least one time index (central absolute time) associated with the data modality. All data are spatially correlated by tracking the position of the modality in space during surgery, particularly via a navigation system (with a tracking system such as a navigation camera). All data are centrally detected within the digital surgical system, automatically assigned a common timestamp or time index, and stored, in particular, on a server or cloud system. By processing multiple different modalities, the data can also be associated with semantics, namely the identification of surgical steps and the scripting of recorded surgeries. According to this disclosure, two systems can be constructed that perform independent functions but are interconnected. The first system, a digital surgical system, generates data modalities and synchronizes them with respect to time and space. The second system, a surgical digital playback support system, accesses this synchronized data and provides it to the user as needed.

[0032] Preferably, the surgical digital regeneration support system may have a digital twin representing all devices detected in the operating room, particularly in avatar mode.

[0033] According to another embodiment, a surgical digital regeneration support system providing at least one additional (third) data modality further includes a neuromonitoring device for detecting neuromonitoring data (particularly electronic signals or recordings (particularly images), the neuromonitoring position, particularly orientation, being tracked (indirectly via the neuromonitoring device) via a navigation system), a biopsy device for creating tissue image data (particularly 2D microscopic recordings (e.g., images in DICOM format), the biopsy device whose tissue image position (particularly orientation) is tracked (indirectly via the tracked biopsy device) via a navigation system, and ultrasound images (preferably 2D or The ultrasound apparatus for creating 3D recordings (images) may include an ultrasound apparatus whose recording position (particularly the recording orientation) is tracked (indirectly via a tracked ultrasound apparatus) by a navigation system. Optionally, data modalities such as spatially localized neuromonitoring data (particularly electrophysiological signals or recordings (images, etc.)) via a navigation system, spatially localized tissue image data (particularly 2D microscopic recordings (images in DICOM format, etc.)) and / or spatially localized ultrasound images (preferably 2D or 3D recordings (particularly images)) via a navigation system may also be stored.

[0034] In particular, the view adjustment unit may be configured to arrange data modalities ranging from macroscopic images such as personal surgical cameras to microscopic images such as microscope images, and nanoscopic images such as tissue images, in order to provide the user with an intuitive and structured visual view.

[0035] The user interface is preferably a web-based software platform that allows users to experience surgical cases from virtually anywhere. The user interface may also consist solely of a browser or software application on a terminal device, establishing a (secure) connection to the server (via an intranet or the internet) and providing data accordingly.

[0036] In particular, digital surgical systems can input time-synchronized and spatially correlated multimodal data via data / video interfaces such as wired Ethernet, HDMI®, and SDI.

[0037] In particular, the present disclosure aims to achieve the objective of a synchronized recording method for generating a spatially and temporally synchronized first data modality and a second data modality for a surgical digital regeneration support system by including the following steps: the steps include: tracking at least one location (in particular orientation) of the first data modality in space with respect to a common global coordinate system by a navigation system; detecting and storing data (in particular recording) of the provided first data modality together with the tracked location and a time index of unified time (preferably together with the semantics of the surgical step); tracking at least one location (in particular orientation) of the second data modality in space by the navigation system within or with respect to a common global coordinate system; detecting and storing data (in particular recording) of the provided second data modality together with the tracked location and a time index of unified time; creating a visual view of the first data modality and the second data modality at a given time index by a view adjustment unit based on a user specification including at least a time index; and outputting the view by a display device.

[0038] Problems are solved with respect to computer-readable storage media and computer programs, namely, instructions that cause a computer to perform method steps according to this disclosure when executed by a computer.

[0039] The disclosure relating to the surgical digital playback support system in this disclosure also applies to the synchronized recording method in this disclosure, and vice versa. In particular, it applies due to the similar features of the apparatus and method. [Brief explanation of the drawing]

[0040] The present invention will be described in more detail below with reference to the drawings, using preferred embodiments. [Figure 1] A schematic diagram of a surgical digital regeneration support system according to a first embodiment is shown, including a visual view of different dynamic modalities generated by a digital surgical system and synchronized spatially and temporally. [Figure 2] The operating modes for spatially and temporally synchronized data storage are outlined below. [Figure 3] A schematic diagram of data exchange between a digital surgical system and a surgical digital regeneration support system, according to another preferred embodiment, is shown. [Figure 4] This document presents possible applications of a digital regeneration support system for surgical procedures. [Figure 5] A user interface for a surgical digital regeneration support system according to another preferred embodiment is shown. [Figure 6] A flowchart of an embodiment of the synchronous recording method according to this disclosure is shown. The drawings are schematic and are intended only to aid in understanding the present invention. Identical components are denoted by the same reference numerals. Features of the various embodiments are interchangeable. [Modes for carrying out the invention]

[0041] Figure 1 shows a schematic diagram of a surgical digital replay support system 1 according to a first embodiment of the present disclosure, for the temporally correlated (here, real-time) transmission of a surgical intervention to patient P, or for virtual training or evaluation of a surgical intervention to patient P.

[0042] The surgical digital regeneration support system 1 includes a central data information system 2 equipped with a storage unit 4, in which at least a first data modality 6 and a second data modality 8 distinct from the first data modality 6 are stored or to be stored in a database. Both the first data modality 6 and the second data modality 8 have at least one position index (particularly a posture index) with respect to the unified common coordinate system 10 of the patient P. In this embodiment, the (X1, Y1, Z1) coordinates associated with the database entry for recording the first data modality are stored, and similarly, the (X2, Y2, Z2) coordinates associated with the second data modality are stored. Thus, in this embodiment, the database is configured to include a position index column. This position index is used to correlate the two data modalities with respect to position, and further, if orientation is specified in another column of the database, for example, it is also used to correlate them particularly with respect to posture. Thus, there is (at least) one unified position index for each record of each data modality, and the data modalities are spatially correlated or synchronized.

[0043] Furthermore, for each data modality (each record), a time index 12 is stored to synchronize them with respect to a unified absolute time. In particular, records may be created at different times for each data modality 6, 8 and stored with a timestamp or time index. For example, a surgical microscope may, as the first data modality, create a record at the first time t1 at the first position x_t1, y_t1, z_t1, move, and then create yet another record at the second time t2 at the second position x_t2, y_t2, z_t2. Both records are stored. In particular, a surgical microscope may also record a video or video stream in which the position of the object being recorded is continuously tracked (indirectly via the surgical microscope head). For example, the video may be assigned a continuous time index and stored accordingly.

[0044] In this surgical digital regeneration support system 1, the key feature is that data from two different data modalities 6 and 8 are supplied, spatially and temporally synchronized, and interconnected.

[0045] Based on the data information system 2 containing this stored data, the view adjustment unit 14 of the surgical digital playback support system 1 is configured to generate a visual view 18 including a first data modality 6 and a second data modality 8, corresponding to a time specification based on a user specification 16 that includes at least one time specification, and this view can be provided digitally. For example, when a user inputs a time specification for a surgical intervention via a touch display, along with other user-specified parameters such as the selection of data modalities 6 and 8 to zoom in on, the view adjustment unit 14 creates a view of the surgical intervention using the first and second data modalities at that exact point in time. Similar to a flyback device or time machine, it is possible to jump to a specific time and generate a view. This generated view 18 is available digitally based on this time specification, in particular in the form of a single still digital image or a dynamic view (e.g., a video stream).

[0046] The generated digital view 18 is transmitted to the visual user interface 20 of the surgical digital playback support system 1 and visually output via the display device 22 for display to the user. Based on this visual information, the user can be trained, evaluate surgical phases, or select segments of recorded surgery to play back at least the first and second data modalities in a spatially and temporally synchronized mode.

[0047] Figure 1 shows, in addition to the first and second data modalities, further different modalities detected by the digital surgical system and correlated temporally and spatially in accordance with this disclosure.

[0048] Figure 2 schematically illustrates how data from data modalities is stored. Here, not only the content of the (data) modality but also the temporal and spatial correlations between data are detected.

[0049] Figure 3 shows a schematic diagram of possible data exchange. The digital surgical system detects all data modalities during surgery. A treatment plan (e.g., a preoperative plan based on patient P's CT and / or MRI datasets) can be imported as input. The results of the surgery are exported postoperatively in a treatment report. The digital surgical system is installed and positioned in the operating room, while the treatment plan and treatment report data are stored on a server (particularly a cloud system). This server (particularly a cloud system) has data connectivity with the surgical digital regeneration support system 1 or virtual surgical system as described herein. Thus, Figure 3 illustrates data exchange between the digital surgical system for data acquisition and the surgical digital regeneration support system 1 for visual data display.

[0050] Figure 4 shows a surgical digital reconstructive support system 1 according to an embodiment of the present disclosure. The digital surgical system includes a monitor 22 as an example of a display device, a navigation system 104 for instrument tracking and navigation, and a robot 106 equipped with a robotic arm 108, the tip of which is connected to a surgical microscope 28 as an end effector. Here, an experienced surgeon performs surgery in the real world using the digital surgical system, and the digital surgical system detects all data modalities along with position or orientation and time indicators in the real world. Thus, data is collected in the real world. These data modalities are provided to the central information system 2 of the surgical digital reconstructive support system 1. This allows surgical residents to access this data from anywhere, particularly via a web portal. The surgical digital reconstructive support system 1 creates a digital twin for residents. The digital twin represents a digital copy of the operating room (OR), enabling residents to immerse themselves in the surgery. As shown in Figure 4, residents can observe surgical interventions and become immersed in the surgery through 2D screens, 3D screens, or VR or AR glasses or head-up displays. In particular, providing multiple resident surgeons with a VR goggle-type user interface allows multiple users to understand and train in surgical interventions simultaneously and in a portable manner. For example, such a visual user interface 20 can be provided to medical students and surgeons undergoing specialist training to solidify surgical skills and then to provide individual training.

[0051] Figure 5 shows the user interface of the surgical digital regeneration support system 1 in a further embodiment. In this embodiment, the display device is in the form of a touch display, and the view has multiple subfields with different data modalities. This user interface creates the impression of a cockpit where all modalities are displayed. Of the different data modalities, the primary (data) modality is displayed enlarged in the center. The user can, of course, change the layout via the touch display, adapting it to their needs, for example, by long-pressing the data modality field and dragging the window to change its position, similar to a mobile phone. At the bottom of view 18 is a control unit, similar to a video player, which displays which modality was active and when on a time scale, allowing the user to jump to specific steps of surgical cases or interventions. Surgical cases consist of scripts and specific phases. Pictograms and sliders allow quick access to specific surgical steps. Preoperative plans (treatment plans) and postoperative reports (treatment reports) are also accessible. The virtually inserted ring schematically illustrates spatial correlations between data (such as the correlation between the focus of a microscopic image (especially a microscopic image) and its corresponding location in the navigation dataset). The navigation dataset always includes 3D datasets as well as video (streams). This allows the user to pause the surgical case at any time and virtually "move" the 3D dataset, for example, to explore structures that are not directly visible. Modalities are arranged from macroview (e.g., staff camera) to microview (e.g., microscope) and nanoview (e.g., tissue image). This allows the user to "zoom in" and "zoom out" when using different data modalities.

[0052] Figure 6 shows a flowchart of a synchronized recording method for creating spatially and temporally synchronized first data modality 6 and second data modality 8 for the surgical digital regeneration support system 1 according to this disclosure.

[0053] In the first step S1, the navigation system 104 tracks S1 at least one spatial position (particularly orientation) of the first data modality 6 with respect to the common global coordinate system 10.

[0054] In the second step S2, the digitally provided data of the first data modality 6, in particular records, are detected and stored in the central data information system 2, along with time indicators by tracked location and unified time.

[0055] In step S3, similar to step S1 (the order of the first and second data modalities may be reversed), the navigation system 104 tracks at least one position (in particular orientation) of the second data modality 8 in space with respect to the common global coordinate system.

[0056] In step S4, similar to step S2, the data (particularly the records) provided by the second data modality (8) is detected and stored in the central data information system (2) along with the tracked location and unified time indices.

[0057] Next, in step S5, the view adjustment unit 14 generates a visual view 18 that includes a first data modality 6 and a second data modality 8, corresponding to a time specification based on a user specification 16 which includes at least one time specification.

[0058] In step S6, the created digital view 18 is transmitted to the user interface 20.

[0059] Finally, in step S7, view 18 is output by the display device 22.

[0060] This synchronized recording method makes it possible to detect and store data from different modalities, or different data modalities, in a spatially and temporally synchronized manner. These data modalities can then be played back, particularly at times specified by the user. [Explanation of symbols]

[0061] 1 Surgical digital playback support system 2 Data information system 4 Storage unit 6 First data modality 8 Second data modality 10 Common global coordinate system 12 Time index 14 View adjustment unit 16 User specified 18 Created view 20 Visual user interface 22 Display device 24 Surgical instruments 26 Timeline 28 Surgical microscope 30 Endoscope 32 Fluorescence imaging system 34 Operating room staff camera 36 Patient camera 38 Microphone 40 Neuromonitoring device 42 Biopsy device 100 Digital surgical system 102 Visualization system 104 Navigation system 106 Robot 108 Robot arm P Patient S1 Step S2 Tracking the position of the first data modality Step S3 Detecting and saving data of the first data modality along with position and time Step S4 Tracking the position of the second data modality Step S5 Detecting and saving data of the second data modality along with position and time Step S6 Creating a visual view Step S6 Transmitting the visual view to the user interface Step S7 Outputting the view through the display device

Claims

1. A surgical digital playback support system (1) for the temporally correlated transmission of surgical interventions to a patient (P), or for virtual training or evaluation of surgical interventions to a patient (P), comprising a central data information system (2), a view adjustment unit (14), and at least one visual user interface (20), The central data information system (2) includes, in particular, a storage unit (4), The storage unit (4) stores or at least caches a first data modality (6) and a second data modality (8) that is different from the first data modality (6). Both the first data modality (6) and the second data modality (8) are, The patient (P) has a unified common coordinate system (10) in which at least one positional index is a postural index, and therefore correlates with each other at least with respect to position, particularly posture. Furthermore, it has a time index (12) that further synchronizes with each other with respect to unified absolute time, The view adjustment unit (14) is configured to create and digitally provide a visual view (18) including the first data modality (6) and the second data modality (8) at a time specified based on a user specification (16) which includes at least one time specification. A surgical digital playback support system (1) is provided, wherein the created view (18) is digitally transmitted to and output to at least one visual user interface (20) and displayed to the user via a display device (22), thereby enabling the user to evaluate surgical phases or select recorded segments of surgery, and to play back at least the first and second data modalities in a spatially and temporally synchronized mode.

2. In the surgical digital regeneration support system (1) according to claim 1, The surgical digital regeneration support system (1) is configured to operate in live mode. In the aforementioned live mode, The first data modality (6) and the second data modality (8) are provided and cached in real time. The time setting of the view adjustment unit (14) is continuously executed in real time, providing users with live video recording and virtual playback and support of interventions regardless of location, and enabling surgeons located away from the operating room to perform remote surgical interventions on the patient (P), or The surgical digital regeneration support system (1) is configured to operate in postoperative mode. In the postoperative mode, the surgical digital replay support system (1) makes recorded and stored interventions, including at least the first data modality (6) and the second data modality (8), visually reproducible postoperatively, thereby particularly confirming the correct execution of the surgical intervention or providing the user with virtual surgical training in a place and time independent manner.

3. In the surgical digital regeneration support system (1) according to claim 1 or 2, At least the first data modality (6) and / or the second data modality (8) each further, as digital data, in addition to the time index and at least the position index, 2D recording in image or video stream format, 3D recording in image or video stream format, 3D point cloud recording in image or video stream format, A graph of at least one electrical signal, Voice memos, and / or Location information of surgical instruments (24), particularly location information including the instrument type of the instrument. A surgical digital regeneration support system (1), including the above.

4. In the surgical digital regeneration support system (1) described in any of the preceding paragraphs, The surgical digital regeneration support system (1) comprises at least one portable mobile device which is a computer and / or in particular a tablet, The apparatus is a surgical digital regeneration support system (1) which forms at least one of the visual user interfaces (20) and outputs the created view (18) via a display device (22), which is in particular a display, connected thereto.

5. In the surgical digital regeneration support system (1) described in any of the preceding paragraphs, The display device (22) is 2D display or 3D display, or A surgical digital regeneration support system (1), which is a head-up display, particularly a virtual reality head-up display, an augmented reality head-up display, or an MR head-up display.

6. In the surgical digital regeneration support system (1) described in any of the preceding paragraphs, The aforementioned surgical digital regeneration support system (1) is The system comprises a digital surgical system (100) equipped with a visualization system (102), a navigation system (104), and a robot (106) equipped with a robotic arm (108). The robot (106) is configured to record the first data modality (6) and the second data modality (8) in a spatially correlated and temporally synchronized manner, and to provide them to the central data information system (2), in a surgical digital regeneration support system (1).

7. In the surgical digital regeneration support system (1) described in any of the preceding paragraphs, The aforementioned surgical digital regeneration support system (1) is Equipped with a navigation system (104), Furthermore, in order to provide the first data modality (6) and / or the second data modality (8), the system comprises a digital surgical microscope (28), a surgical endoscope (30), a fluorescence imaging system (32), at least one operating room staff camera (34), a patient camera (36), a microphone (38), the navigation system (104), a robot (106), and surgical instruments (24), The digital surgical microscope (28) creates 2D or 3D recordings in video format, and its recording position, in particular, is tracked via the navigation system (104). The surgical endoscope (30) creates 2D or 3D recordings in video format, and its recording position, in particular, is tracked via the navigation system (104). The fluorescence imaging system (32) creates fluorescence recordings in the form of 2D or 3D recordings, and in particular, its recording orientation, or recording position, is tracked via the navigation system (104). The at least one operating room personnel camera (34) is an operating room panoramic camera or a body camera tracked within the operating room, and provides a video and / or 3D point cloud image stream correlated with the navigation system (104). The patient camera (36) has its optical axis and field of view directed toward the patient (P) and detects and provides recordings, particularly video and / or 3D point cloud image streams, that correlate with the navigation system 104. The microphone (38) detects, saves, and later plays back recordings, particularly voice memos, along the timeline via the audio device of the user interface (20). The navigation system (104) has navigation data which is a 3D volume dataset, preferably a CT recording and / or MRI recording detected preoperatively and / or intraoperatively. The robot (106) is equipped with a robot arm (108), and the coordinates of the robot arm (108) are tracked via the navigation system (104) or the robot tracking system. The surgical instrument (24) is tracked via the navigation system (104) in a surgical digital playback support system (1), where the instrument type and, preferably, the instrument position, which is the tip of the instrument, are tracked.

8. In the surgical digital regeneration support system (1) according to claim 7, The surgical digital regeneration support system (1) further comprises at least one of the following devices to provide at least one additional data modality: a neuromonitoring device (40), a biopsy device (42), and an ultrasound device. The neuromonitoring device (40) detects neuromonitoring data, which is in particular electrical signals or recordings, and the position of the neuromonitoring data, in particular posture, is tracked via the navigation system (104). The biopsy device (42) creates data of a tissue image, which is in particular a 2D microscope image, and the position, which is in particular the orientation of the tissue image, is tracked by the navigation system (104). The ultrasound device is for creating ultrasound images, preferably in 2D or 3D, and in particular, its recording position, which is the recording posture, is tracked via the navigation system (104), in a surgical digital playback support system (1).

9. In the surgical digital regeneration support system (1) described in any of the preceding paragraphs, The data information system (2) further stores preoperative treatment plans and / or postoperative treatment reports for the surgical intervention, and the preoperative treatment plans and / or preoperative and postoperative treatment reports are included as the basis for the user-specified (16) and / or created views (18) of the surgical digital playback support system (1).

10. In the surgical digital regeneration support system (1) according to claim 9, The view adjustment unit (14) inserts a timeline (26) displaying treatment reports of the surgical phases and / or surgical interventions of the treatment plan into the view (18), enabling intuitive visual display with quick access to each surgical step, in a surgical digital playback support system (1).

11. In the surgical digital regeneration support system (1) according to claim 9 or 10, A surgical digital regeneration support system (1) is configured such that the view adjustment unit (14) automatically determines the relevant primary data modality (6, 8) from at least the first data modality (6) and the second data modality (8) based on the current surgical phase of the treatment plan and / or the treatment report, and displays the primary data modality (6, 8) in the view (18) in relation to the other data modalities (8, 6).

12. In the surgical digital regeneration support system (1) described in any of the preceding paragraphs, The data information system (2) also stores three-dimensional navigation data as a 3D dataset, which in particular is preoperative CT and / or MRI records of at least one body part of the patient, thereby enabling a surgical digital regeneration support system (1) to virtually move within the patient's tissue volume, preferably during the regeneration of the surgery.

13. A synchronized recording method for creating spatially and temporally synchronized first data modality (6) and second data modality (8) for a surgical digital regeneration support system, particularly a surgical digital regeneration support system (1) according to any one of the preceding claims 1 to 12, Step (S1) of tracking the spatial position, in particular the attitude, of at least one of the first data modalities (6) with respect to a common global coordinate system, using the navigation system (104), Step (S2) of detecting and storing the digitally provided data, particularly the record, of the first data modality (6) in the central data information system (2), along with the tracked location and unified time indicators. Step (S3) of tracking the position of at least one of the second data modalities (8), particularly its orientation, in space related to the common global coordinate system, using the navigation system (104), Step (S4) of detecting and storing the provided data, in particular the record, of the second data modality (8) in the central data information system (2), along with the tracked location and unified time time indicators, Step (S5) of the view adjustment unit (14) creating a visual view (18) including the first data modality (6) and the second data modality (8) at the specified time, based on user specifications (16) including at least a time specification, Step (S6) of transmitting the view (18) to the user interface (20), A method comprising the step (S7) of outputting the view (18) using a display device (22).

14. A computer-readable storage medium, A storage medium that, when executed by a computer, includes instructions causing the computer to perform the steps of the method according to claim 13.

15. It is a computer program, A computer program that, when executed by a computer, includes instructions causing the computer to perform the steps of the method according to claim 13.

Images