System for pre-operative surgical planning based on virtual reality

Abstract

The present invention relates to a system (100) for visualizing an organ for planning tumor resection prior to surgery and for providing an immersive view of a patient's anatomy.

Description

[0001] The present invention relates to a system for preoperative surgical planning in a virtual reality environment, and more particularly to a system for visualizing organs to plan tumor resection prior to surgery and for providing an immersive view of a patient's anatomy.

[0002] Virtual reality (VR) devices and software tools have grown in popularity over the past few years and have proven to be valuable tools for improving surgeons' understanding of patient anatomy. VR assessment platforms incorporate more features and functionality than existing 2D or 3D planning software, facilitating immersive and interactive manipulation, realistic in-depth perception, and visualization of the complex relationships of anatomical structures. As a result, surgeons gain a more realistic and detailed understanding of the patient's anatomy.

[0003] Furthermore, automated imaging algorithms can potentially create more efficient planning by enabling visualization of anatomical structures of interest. This could be further improved by developing combined artificial intelligence (AI)-based and immersive 3D-VR platforms as preoperative planning tools that complement traditional CT imaging.

[0004] In particular, in procedures such as thoracoscopic segmentectomy, which are generally more complex than thoracoscopic lobectomy, identifying intraparenchymal planes and ligating target branches of the pulmonary artery, vein, and bronchi can be more challenging. This is partly due to the many anatomical variations and anomalies in the pulmonary vascular anatomy.

[0005] Therefore, the success of segmentectomy may depend largely on the surgeon's preoperative understanding of the patient's anatomy. Although conventional computed tomography (CT) imaging is the gold standard for preoperative planning of anatomical (sub)lobectomy, identifying the segmental boundaries and segmental branches of arteries, veins, and bronchi becomes more challenging when conventional 2-dimensional (2D) CT techniques are employed.

[0006] It is therefore an object of the present invention to provide a system for visualizing an organ for planning a segmentectomy and / or lobectomy procedure, which system provides accurate and patient-specific insights into the organ.

[0007] This object is achieved by a system according to independent claim 1. Further advantageous embodiments can be achieved by a system according to the dependent claims.

[0008] According to the present invention, a system for visualizing an organ for planning tumor resection is provided, including details of segmentation and other data related to the patient. In addition, the system provides an immersive view of the patient's anatomy. The system includes:

[0009] - a control module comprising data storage and processing resources configured to store, access and execute computer-readable program instructions;

[0010] - a server communicating with the control module via a communication network;

[0011] - a user interaction module, including a multi-player module configured to facilitate real-time interaction between a user and the system;

[0012] a user input module configured to control and receive input information from a user, wherein the user input module includes a plurality of input devices, a virtual reality input module for receiving input in a virtual reality environment, and a virtual reality control module for performing operations based on information received from the virtual reality input module; and

[0013] - Input / output module, which includes a rendering module, motion tracking device, 3D display, 2D display and head-mounted display for immersive viewing;

[0014] - the user interaction module, the user input module and the input / output module communicate with each other and with the control module via a communication interface;

[0015] - a control module configured to receive image data and processed data from a server and to store the image data and the processed data in a data storage device, wherein the image data comprises a 2D computed tomography image of an anatomical structure, and wherein the processed data comprises segmented data representing segments of the anatomical structure;

[0016] - a user interaction module configured to create visualizations of the anatomical structure in 2D and 3D based on the image data and the processed data received from the control module, and to edit the created 2D and 3D visualizations based on input information received from the user input module;

[0017] - the input / output module is configured to display visualizations on corresponding 2D and 3D displays and to provide an immersive view of the patient's anatomy via a head-mounted display, wherein the immersive view is generated from the image data and the processed data, preferably using a ray-stepping technique;

[0018] - The input / output module is further configured to edit the 2D visualization, 3D visualization and immersive view using the rendering module based on information received from the user interaction module, the user input device, the virtual reality input module and the motion tracking device.

[0019] The system advantageously enables a better understanding of the patient's anatomy and provides a detailed, realistic, immersive view of the anatomy and its segments. The system further advantageously allows real-time interaction with the displayed image (e.g., modifying image size, color, orientation, and viewing angle). Furthermore, the system enables real-time manipulation of the displayed image while also facilitating dynamic interaction with other users, for example, through voice chat capabilities that simulate a live conversation with other remote users. Specifically, the system allows users to immersively visualize anatomical structures such as the lungs, as well as other tissue or organ structures. By mastering these immersive and "deep dive" visualizations before surgery, one or more surgeons can more confidently plan surgeries, thereby performing them in a more accurate and controlled manner. By providing immersive visualizations of 3D renderings (or 3D models) of the subject's anatomy during preoperative planning, surgeons can better understand the specific patient's anatomy when planning surgery. This, in turn, reduces the risk of unexpected errors or problems while performing the surgery and provides better orientation and fewer surprises during surgery. As a result, resections planned using the system according to the present invention will likely be more accurate and less invasive due to more advanced and accurate planning, meaning, for example, less organ tissue must be removed. This is particularly beneficial in the removal of tumors in lung tissue, but also in other organs (e.g., liver) or tissue structures (e.g., brain). Generally speaking, the system is applicable to all types of tissue in a patient.

[0020] In particular, the system includes a module for viewing 2D and / or 3D visualizations on a mobile device, such as a mobile phone, tablet computer and / or web browser. In this way, the system advantageously provides mobile visualization of 3D models of anatomical structures.

[0021] The system may further include a remote display device (eg, including a 2D display and a 3D display) configured to display 2D and / or 3D visualizations, thereby enabling a surgeon to view the 2D scan and / or 3D model during surgery.

[0022] In particular, the system includes a control module configured to communicate with a server via a communication network to receive medical images, scan data, and other information from the server. The scan data may include 2D computed tomography scan data, manually created 3D segmentation data, and 3D segmentation data created by an artificial intelligence (AI)-based algorithm.

[0023] The control module includes data storage for storing data and information received from the server and processing resources configured to store, access and execute computer-readable program instructions.

[0024] The user input module is configured to control and receive input information from the user. Therefore, the user input module includes multiple input devices, a virtual reality input module and a virtual reality control module.

[0025] The system further includes a user interaction module including a multi-player module configured to facilitate real-time interaction between a user and the system.

[0026] The user interaction module also includes an input / output module configured to facilitate further interaction with the system (eg, via a motion tracking device or head mounted display) and output 2D and 3D views of the subject anatomy.

[0027] In particular, the input / output module includes a rendering module, a motion tracking device, a 3D display, a 2D display, and a head-mounted display.

[0028] Thus, the input / output module is configured to display visualizations on 2D and 3D displays contained therein, and to provide an immersive view of the patient's anatomy via a head-mounted display. In particular, the immersive view is generated from the image data and the processed data, preferably using a ray-stepping technique.

[0029] Additionally, the user interaction module is configured to create visualizations of the anatomical structure in 2D and 3D based on the image data and processed data received from the control module. The user interaction module is further configured to edit the created 2D and 3D visualizations based on input information received from the user input module.

[0030] The user interaction module, the user input module and the input / output module are configured to communicate with each other and with the control module through a communication interface. The communication can be wireless communication or wired communication.

[0031] The rendering module of the input / output module is configured to edit the 2D visualization, 3D visualization, and immersive view in real time based on information received from the user interaction module, the user input device, the virtual reality input module, and the motion tracking device.

[0032] In particular, the rendering module comprises a rendering algorithm, such as a ray-marching algorithm, configured to perform solid rendering, volume rendering and / or transparent mesh rendering of a segment of the anatomical structure. The rendering algorithm is further configured to provide a stereoscopic view.

[0033] In particular, the plurality of input devices include at least one of a mouse, a joystick, a controller, a motion sensor, a keyboard, and a microphone. The plurality of input devices advantageously facilitates inputting, receiving, and adding various types of data and information into the system in an efficient manner.

[0034] The user input module may further include a shortcut key (or hot key) module configured to control a list of keyboard keys for general purposes rather than specific purposes. In particular, the user input module is configured to provide a separation layer between some objects (e.g., 3D models) and the components of the user input module. The hot key module is further configured to provide a location for adding code for keyboard-driven events that do not belong to any specific object.

[0035] In particular, the user input module also includes a mouse control module configured to use the mouse and keyboard to perform predetermined functions. In this way, the input devices work together to quickly perform predetermined functions, such as editing the opacity of an image.

[0036] The user input module may further include a tool manager module configured to manage a virtual reality (VR) tool (user tool), wherein the VR tool determines the state of the VR controller. In particular, the VR tool is attached to the VR controller and is configured to perform actions based on input received from the VR input module. Advantageously, the tool manager module is configured to convert actions / instructions defined by the VR controller into real actions within the VR environment.

[0037] Alternatively or additionally, the user input module further comprises a transparency plane module configured to create a clipping plane to determine the camera visible scene, for example in the viewing area. This advantageously provides a clear (and less busy) view of the subject organ.

[0038] The user input module may further include a scan slice module configured to control the 2D visualization of the CT scan in the VR environment.

[0039] In particular, the user interaction module includes a fragment manager module configured to load fragments of the anatomical structure into the system, i.e., into the active workspace. In particular, the fragment manager module is further configured to store the state of each fragment of the anatomical structure and create a list thereof. In this way, new fragments of the analysis structure can be loaded (added) into the system, and the active workspace can be dynamically modified. The state refers to the transparency level of the rendered fragment, i.e., shell, solid, or another percentage of transparency.

[0040] The user interaction module further comprises a path manager module configured to create a list of paths to be followed by the camera in the virtual reality environment and / or the 2D environment. Advantageously, the path manager module provides a reliable pre-operative planning procedure based on predefined visualization steps defined by the camera's path.

[0041] In particular, the user interaction module further includes a brush manager module configured to create colored segments in the virtual reality environment based on input information. In this way, the segments created in the VR environment can be easily controlled and distinguished by the user by using different colors. Preferably, the brush manager module is further configured to provide a list of colored (brushed) segments and save the list of colored segments.

[0042] In particular, the user interaction module further includes a preset manager module, which contains a preconfigured array of segment settings for delimiting different viewpoints of the available segments (or segmented data). Advantageously, in this way, a specific viewpoint of the available segments can be quickly and automatically delimited for the user.

[0043] Preferably, the preconfigured array of segment settings includes image data, automatic segmentation, bronchial vessels, arteries, veins, airways, segments (solid), segments (shell), complete anatomy.

[0044] Furthermore, the user interaction module may further comprise an automatic segmentation module configured to create new segments of the analysis structure and add them to the active workspace. Advantageously, the automatic segmentation manager module automatically creates the new segments using a software tool.

[0045] In particular, alternatively or additionally, the user interaction module comprises a workspace control module configured to create a list of files and paths linkable to the new workspace.

[0046] In particular, the workspace control module is configured to control and modify the workspace within the system before loading the workspace. In this way, files can be quickly loaded into the corresponding workspace, and the paths associated therewith can be reliably linked to the workspace.

[0047] For example, a workspace may contain a combination of a patient's CT scan, segmentation, measurements that may be automatically saved. The system is configured to create a new workspace by loading a new CT scan (or, if preferred, a new CT scan with additional segmentation).

[0048] In particular, the input / output module may further include a scan container module configured to control features or characteristics of the scanned object. In particular, the scan container module is configured to control the positioning, scaling, and rotation of the displayed anatomical structure. In this way, the features of the displayed anatomical structure (i.e., the scanned object) can be individually modified as needed.

[0049] Additionally or alternatively, the input / output module may further include a screenshot module configured to save a screenshot of the workspace. Advantageously, the properties of the captured frame may be defined to create an output image with high resolution. Metadata may also be added to the output image.

[0050] In particular, the data storage device of the control module includes video memory, RAM, ROM and flash memory.

[0051] In particular, the control module may further include an authorization module configured to implement authorization functionality so that only authorized users can access. For example, a unique code for software installation contained (stored) in the system. Based on the code of the internal system, an access key is provided, which is unique to the authorized user and grants access through the control module.

[0052] In particular, the authorization module is advantageously used to prevent users from copying and pasting software built on another computer. By having each computer have a unique code connected, unauthorized copying of the technology is prevented.

[0053] Alternatively or additionally, the control module may comprise an automatic saving module configured to automatically save the workspace. Advantageously, the saved workspace may be easily controlled by the workspace control module.

[0054] In particular, the user interaction module may further include an event control module configured to convert or translate mouse clicks into button actions in the active workspace.

[0055] Additionally, the user interaction module may further include a measurement control module configured to control and create measurements and procedures to be performed by the system (e.g., using 2D tools or 3D (VR) tools). Preferably, each measurement includes a name, a value, a 2D and 3D visualization, and a list of coordinates. This, in turn, generates a line between two determined coordinates, with a quantitative measurement in millimeters.

[0056] In particular, the measurement control module is configured to calculate a value of a created measurement based on its current coordinates.Alternatively or additionally, the measurement control module is configured to edit a measurement and / or trigger its saving / deletion.

[0057] The measurements include 2D or 3D measurements configured to be created via the user input module.

[0058] In particular, the measurement may further comprise a sphere tool and / or a marker tool configured to edit the CT scan along coordinates provided thereon. The sphere and marker tools are configured to be applied in VR and on a 2D screen.

[0059] In particular, the measurement control module is configured to keep track of (saved) measurements and active measurements in the current workspace.

[0060] In particular, the measurement control module is configured to handle the saving and deletion of measurements. Preferably, the measurement control module is configured to trigger the workspace manager module to update the workspace file. Furthermore, the measurement control module may trigger the measurement panel to update the displayed measurement list.

[0061] For example, preferred CT scan settings, slice colors, events and combinations of measurement or sphere tools are configured to be saved in the workspace so that when the user closes the program and restarts the application, the preferred settings are automatically present.

[0062] Additionally, once the workspace is loaded, the workspace control module is configured to notify the measurement control module to load corresponding measurements from the loaded workspace.

[0063] The above-mentioned interaction and communication between the measurement control module and the workspace control module provides reliable transmission of image data and information within the system and a unified visualization program.

[0064] In particular, the system may further include a module for visualizing the 3D model on a phone and a website through the cloud.

[0065] In particular, the system is configured to perform a dynamic modeling method for making 3D patient images interactive.

[0066] The dynamic modeling method describes a software pipeline that processes 3D images into interactive software. Specifically, a 3D image of the patient is acquired from a CT scan or MRI scan as a prerequisite. The dynamic modeling method then enables the user to perform simulated surgical procedures specific to the patient's anatomy based on the acquired 3D image. The method is intended to be used as a tool for planning procedures both preoperatively and intraoperatively.

[0067] In particular, the method comprises the following steps:

[0068] - (optionally) acquiring a 3D image of the patient from a CT scan or MRI scan as a prerequisite,

[0069] - labeling of anatomical structures in 3D images to define which voxels of the original 3D patient image belong to a specific anatomical structure,

[0070] - creating a 3D surface for each of the anatomical structures based on the previously created labels, said 3D surface preferably being represented by a triangular mesh;

[0071] - define the volume of each anatomical structure;

[0072] - assigning physical properties to each anatomical structure;

[0073] - Simulate the transition of patients to surgical status; and

[0074] -Perform interactive simulations on patients.

[0075] In particular, in the step of creating a 3D surface, the created surface mesh is used to represent the physical boundaries of each anatomical structure, and these physical boundaries are visually represented in the interactive 3D graphics.

[0076] In particular, the step of defining the volume of each anatomical structure creates a volume representation of each of the anatomical structures. The volume representation is composed of a tetrahedral mesh, where a tetrahedron is a 3D primitive shape. The volume mesh defines the object being simulated.

[0077] Specifically, the allocation step is based on a set of preset attributes stored in the software. The user then selects one of the presets that best matches the health status of the patient in question. Thereafter, the patient's transition to the surgical state is initiated.

[0078] Specifically, the surgical state involves the patient's new posture during surgery and the effects of the incision. Finite element methods are used to simulate the effects of the initial surgical incision. Finite elements thus conformally deform the anatomical structure to the predicted forces during the transition to the surgical state. Advantageously, the stimulation is performed interactively and accurately.

[0079] In particular, in the step of performing stimulation, the simulation is run in an interactive 3D environment where the user can interact and the results are visually presented in real time. Finite element methods can further be used to simulate the patient's flexible tissue.

[0080] In particular, the method facilitates user interaction by performing surgical manipulations via a computer mouse or motion tracking controller.A simulation circuit then simulates the forces of these manipulations on the anatomy and presents the deformation state of the anatomy to a display in real time.

[0081] The invention will be better understood with the aid of the description of an embodiment given by way of example and illustrated.

[0082] In the figures it is shown:

[0083] Figure 1 Schematic diagram of a system according to the present invention;

[0084] Figure 2 Schematic diagram of the user input module;

[0085] Figure 3 Schematic diagram of the user interaction module;

[0086] Figure 4 Schematic diagram of the user input / output module;

[0087] Figure 5 Illustrative view of the performance of the sphere tool;

[0088] Figure 6 an illustrative view of the performance of the marking tool;

[0089] Figure 7 A flow chart of a dynamic modeling method for making a 3D patient image interactive; and

[0090] Figure 8 Flowchart of a possible algorithm for the system.

[0091] Figure 1 A system 100 for visualizing organs to plan tumor resection prior to surgery and for providing an immersive view of a patient's anatomy is schematically shown. The system 100 includes a control module 102 in communication with a cloud server 170 via a communication network 180.

[0092] The server 170 includes medical imaging (eg, with DICOM, NIFTI, or OBJ extensions), such as 2D CT scan images and processed images, ie, 3D images generated from the 2D CT scan images, for example, using AI-based algorithms.

[0093] The system 100 further includes a user interaction module 114, a user input module 114, and an input / output module 118 that communicate with each other and with the control module 102 via a communication interface 112. The communications may include wireless or wired communications.

[0094] The control module 102 includes data storage and processing resources 106 .

[0095] The data storage device 104 includes, for example, video memory, RAM, ROM, and flash memory.

[0096] Processing resource 106 includes, for example, an Intel core i7 or faster processor configured to execute instructions stored in memory.

[0097] Additionally, the control module 102 may further include an authorization module 108 configured to implement authorization checking functionality to verify that the correct authorization key is in place so that only authorized users can access the system.

[0098] It is also contemplated that the control module 102 includes an automatic save module 110 configured to automatically save a workspace. For example, the automatic save of a workspace can be saved every 3, 5, 7, and / or 5 seconds. Advantageously, the saved workspace can be controlled using a "quick load" option described below.

[0099] The system 100 may further include a module (not shown) configured to transmit the 3D rendering (3D model of the anatomical structure) on, for example, a mobile phone, tablet computer, and / or web browser. In this way, the system advantageously provides mobile 2D and / or 3D visualization.

[0100] Additionally, the system 100 may include a remote display device, preferably a 2D display and / or a 3D display, for viewing 3D models and 3D renderings during a surgical procedure.

[0101] Figure 2 Components of the user input module 116 are shown schematically. Thus, the user input module 116 includes a plurality of input devices 120, a virtual reality (VR) input module 122, a VR control module 124, and a mouse control module 126.

[0102] The user input module 116 may further include a tool manager module 128 configured to manage or control the VR tools (or user tools) of the VR input module 122. Each VR tool is attached to a VR control module 124 (VR controller) and is configured to perform actions based on user input received from the VR input module 122. The tool manager module 128 is preferably configured to initialize a (predefined) list of user tools. In this way, the actions / commands defined by the VR controller 124 can be effectively converted into real actions within the VR environment (e.g., MedicalVR software).

[0103] The mouse control module 126 is configured to control the special behavior of the mouse cursor in the software application when the mouse cursor does not behave like a cursor.For example, the user can hold down a keyboard key and move the mouse to directly control the opacity of an image.

[0104] The user input module 116 may further include a hotkey module (not shown). This module is configured to provide a layer of separation between some objects and the user input module 116, as well as a place to add code for keyboard-driven events that do not belong to any particular object.

[0105] For example, shortcut keys can be predefined to trigger the following actions:

[0106] -Select a scan preset

[0107] - Reset position and all options

[0108] - Reset Position

[0109] - Reset fragment transparency and reset color

[0110] - Toggle instant rendering updates

[0111] -Change stereo mode

[0112] - Toggle fragment shading

[0113] -Rotate 3D model

[0114] -Zoom in or out.

[0115] Additionally, the user input module 116 may further include a transparency plane module 129, which includes, for example, a slicer tool for creating a clipping plane. The transparency plane module 129 is configured to determine how much of the scene in the viewport is visible to the camera. Segments of the analysis structure and scan data obscured by the transparency plane module 129 are not displayed, allowing the user to easily view the subject's organ.

[0116] A scan slice module (not shown) may also be included in the user input module 116 , which is configured to control the 2D visualization of the CT scan in the VR environment (eg, MedicalVR software). Figure 3 Components of the user interaction module 114 are schematically shown. The user interaction module 114 includes a multi-player module 150, a segment manager module 152, a path manager module 154, and a brush manager module 156.

[0117] The multiplayer module 150 is configured to allow real-time interaction with (and / or modification of) scanned images and views. The multiplayer module is further configured to facilitate real-time manipulation and changes to scanned views and drawings, while enabling, for example, voice chat capabilities to simulate a live session with other remote users.

[0118] Thus, the multiplayer module 150 may be configured to perform the following functions in real time between users, for example by transmitting:

[0119] -User's head and hand controller position data

[0120] - Live voice audio (not recorded, only streamed)

[0121] - Scan translation / rotation / zoom

[0122] - Scan preset status

[0123] - Scan Vision Settings

[0124] - Pen and brush drawing

[0125] -Any other essential workspace data or settings

[0126] The user identifier and the host / guest's permissions in controlling the session. The snippet manager module 152 is configured to manage and control the snippets of the analysis structure that can be loaded into the workspace. This module is further configured to store the status of each snippet and create a list of (loaded) snippets. The snippet manager module 152 advantageously facilitates adding new snippets or removing snippets from the current (active) workspace.

[0127] In addition, the path manager module 154 of the user interaction module 114 is configured to create a list of paths for the camera to follow in the virtual reality environment and / or in the 2D environment. Therefore, the path of the camera can be predetermined and / or adjusted by the user.

[0128] Specifically, the brush manager module 156 is configured to create colored segments in the virtual reality environment based on user input information. The so-called brush is actually a fragment coloring created by the user in the virtual environment. In this way, the user can create a brush segment by adding color to the CT scan.

[0129] The brush manager module 156 may be further configured to provide a list of brush segments and save the list.

[0130] A preset manager module 158 is also included in the user interaction module 114 for controlling presets in the workspace. These presets are preconfigured arrays of segment settings used to delimit different viewpoints of available scan segments for the user in a quick and easy manner.

[0131] Presets include, for example, Original CT Scan, Automatic Segmentation, Bronchial Vessels, Arteries, Veins, Airways, Segments (Solid), Segments (Shell), and Complete Anatomy. Solid segments are opaque, and shell segments are transparent with approximately 20-30% of the transparency of solid segments.

[0132] Additionally or alternatively, the user interaction module 114 may include an automatic segmentation module 160 configured to (automatically) create new segments of the analysis structure using a software tool. The new segments are configured to be loaded into the active workspace.

[0133] In particular, the auto-segmentation module 160 is further configured to store the status of each new segment and create a list of new segments.

[0134] The user interaction module 114 may further include a workspace control module 162 configured to control workspaces before they are loaded. Preferably, the workspace control module 162 is configured to create a list of files and paths that can be linked to a new workspace. For example, the workspace control module 162 is configured to control the loading of files and create a list of recent files that can be used for a so-called "quick load" option in the MedicalVR software.

[0135] The user interaction module 114 may further include a measurement control module 166 configured to control and create measurements and procedures to be performed using the system 100. Each measurement contains a name, a value, a visualization in 2D and 3D, and a list of coordinates. Each type of measurement (e.g., distance, sphere tool, or marker tool) is a subcategory of measurement. Measurements, spheres, or markers are saved in the workspace along with the accompanying coordinates. When creating or editing a measurement, the measurement control module 166 is configured to calculate the value of the measurement based on the current coordinates of the measurement and create (or update) the corresponding visualization accordingly. The measurement control module 166 is further configured to edit the measurement and / or trigger its saving / deletion.

[0136] The measurements include 2D or 3D measurements configured to be created via the user input module 116. For example, a 2D measurement is created based on a user's mouse input.

[0137] The measurement control module 166 is further configured to keep track of the (saved) measurements in the current workspace and the measurement currently loaded by the user (ie, the active measurement).

[0138] In particular, the measurement control module 166 is configured to handle the saving and deletion of measurements. For example, when a measurement is saved or deleted, the measurement control module 166 is configured to trigger the workspace manager module 162 to update the workspace file and trigger the measurement panel to update its displayed measurement list.

[0139] In particular, when a user loads a workspace, the workspace control module 162 is configured to notify the measurement control module 166 to load corresponding measurements from the loaded workspace.

[0140] Figure 4 Components of the input / output module 118 are shown schematically. The scan container module includes a rendering module 130, a motion tracking device 132, and 2D and 3D displays 136, 138.

[0141] The rendering module 130 includes a rendering algorithm, such as a ray marching algorithm, configured to perform solid rendering, volume rendering, and / or transparent mesh rendering of fragments. The rendering module is further configured to provide stereoscopic display.

[0142] The motion tracking device 132 includes sensors, for example, to capture data regarding the position, orientation, and velocity of the hand or body.

[0143] The 2D display 136 and the 3D display 138 include, for example, a standard computer monitor and a stereoscopic 3D HDMI display, respectively.

[0144] The input / output module 118 may alternatively or additionally include a scan container module 140. In particular, the scan container module 140 is configured to control the positioning, scaling, and rotation of the scanned object. The scan container module 140 is further configured to automatically convert between different units, such as voxels, scene units, and millimeters.

[0145] In particular, the input / output module 118 may also include a screenshot module 142 configured to save a screenshot of the workspace. Preferably, before making a given screenshot, the settings are set to high for one frame to create an image with high resolution. Metadata may also be optionally inserted into the output image.

[0146] It is further contemplated that the system includes a module that enables a surgeon to view a rendered, 3D model of the anatomical structure during a given surgical procedure.

[0147] Figure 5 and 6 The performance of the sphere tool and the marking tool of the system 100 are illustratively shown. The sphere tool is configured to allow the user to create a 3D sphere around a specific anatomical region of interest. Once the 2D or 3D view of the anatomical structure ( Figure 5 By triggering the sphere tool in the top panel of the , the user can point and click on the area of interest. The user can then click and drag to the preferred size to define the sphere ( Figure 5 The middle panel in the figure is used to view the area in the 3D view. Since the center of the sphere is calculated as the starting point, the size is based on the radius. A circle is drawn in the 2D view, and a 3D sphere is visualized in the 3D view. Similarly, the marking tool is configured so that the user can mark an area of interest in the 2D view, for example, and then view the marked area in the 3D view from the top panel to the bottom panel, respectively, as shown in FIG. Figure 6 As shown in .

[0148] Figure 7 A dynamic modeling method 200 for making a 3D patient image interactive is shown. The method is advantageously performed by the system 100.

[0149] 3D patient images can optionally be viewed via Figure 7 The pipeline feed shown in .

[0150] In step 201, anatomical structures in a 3D image are labeled. This step defines which voxels in the original 3D patient image belong to specific anatomical structures. In step 202, a 3D surface is created for each anatomical structure based on the previously created labels. The 3D surface is represented by a triangular mesh. This surface mesh is used to represent the physical boundaries of each anatomical structure and visually represent these physical boundaries in the interactive 3D graphics.

[0151] Then, in step 203, the volume of each anatomical structure is defined. The volume representation consists of a tetrahedral mesh, where a tetrahedron is a 3D primitive shape. The volume mesh defines the object being simulated.

[0152] At step 204, physical attributes are assigned to each anatomical structure. This is based on a set of preset attributes stored in the software. The user then selects one of the presets that best matches the health condition of the patient in question.

[0153] At step 205, the patient is stimulated to transition to the surgical state. The surgical state involves the patient's new position during surgery and the effects of the incision. The effects of the initial surgical incision are simulated using the finite element method. The finite element method in this software causes the anatomical structure to conformally deform with the predicted forces during the transition to the surgical state.

[0154] Finally, in step 206, an accurate and interactive simulation of the patient is performed. The simulation is run in an interactive 3D environment where the user can interact and the results are visually presented in real time. Finite element methods can also be used to simulate the patient's flexible tissue.

[0155] Advantageously, step 206 can be run on a local computer with high-end graphics capabilities. The user interacts by performing surgical manipulations via a computer mouse or motion tracking controller. The simulation circuit then simulates the forces of these manipulations on the anatomical structure and presents the deformed anatomical state to a display in real time.

[0156] The rendering algorithm can be as follows (see Figure 8 ):

[0157] Rendering algorithm used for the main image:

[0158] Volume Data Baking

[0159] Refresh cached voxel bitmap if settings or data have changed (compute shader)

[0160] (This should be done as infrequently as possible, as it is expensive.)

[0161] Clears the voxel and fragment data texture buffers.

[0162] Pass 1

[0163] The data texture is reconstructed by parsing the fragments in order from least significant to most significant.

[0164] Fragments can overlap, but the highest fragment overwrites voxels from lower priority fragments.

[0165] User-created brushes are processed like snippets, but are stored differently for efficiency.

[0166] Mesh-based fragments cannot be handled here, they are drawn as unrelated 3D objects.

[0167] This step creates a 3D bitmap with indices instead of colors, referencing different arrays containing color / opacity / fill etc.

[0168] Pass 2

[0169] Combine the processed fragment data with the scan data.

[0170] This parses each original voxel and outputs it to another voxel buffer.

[0171] The algorithms used for this are fairly detailed, but still change frequently.

[0172] This determines which voxels should be filtered out completely, and the transparency of those that remain.

[0173] Output voxels get their color from the fragment or from a shading texture if provided.

[0174] If the clip has Fill enabled, the above transparency selection will be overridden.

[0175] Pass 3

[0176] Applies an edge detection filter to the entire volume based on per-fragment settings.

[0177] For applicable fragments, the transparency of edge pixels is increased and that of reduced voxels is reduced.

[0178] Note: This step may be moved to pass 1, depending on which behavior is preferred.

[0179] Solid Rendering

[0180] Draw each physical object that will be affected by the scanned image.

[0181] NOTE: The depth buffer from this stage needs to be accessible to the volume rendering stage.

[0182] Volume Rendering

[0183] For each visible screen pixel, process a ray through the voxel bitmap. (Fragment shader)

[0184] Take transparency planes into account when calculating ray start / end points.

[0185] Calculates the number of ray samples to take based on the settings (usually 500-1000).

[0186] Each sampled voxel is combined using front-to-back sorting.

[0187] When a ray collides with a solid object other than the scan content, processing of that ray is stopped.

[0188] This is currently done by using the depth buffer and assuming all other objects are opaque.

[0189] Adjusts the saturation, contrast, and brightness of pixels based on your settings.

[0190] Transparent Mesh Rendering (Fragments)

[0191] Draw all meshes, storing the frontmost alpha and depth position

[0192] Use the frontmost mesh at the correct alpha and blend one-to-one to draw each fragment

[0193] Use each non-frontmost mesh, approximate alpha and blend OneMinusDstAlpha / One to add to each fragment

[0194] NOTE: Considering the slicing tool at this stage is difficult and will need to be rewritten.

[0195] Special entity rendering

[0196] Draw any objects that must always be visible, such as controllers

[0197] The system now renders the image as normal (VR or 2D). The stereo option is also available for 2D, but does not affect the above rendering.

[0198] Reference numerals

[0199] 100 systems

[0200] 102 Control Module

[0201] 104 Data Storage Device

[0202] 106 Processing Resources

[0203] 108 Authorization Module

[0204] 110 Autosave Module

[0205] 112 Communication Interface

[0206] 114 User Interaction Module

[0207] 116 User Input Module

[0208] 118 Input / Output Module

[0209] 120 Input device

[0210] 122 VR input module

[0211] 124 VR control module

[0212] 126 Mouse Control Module

[0213] 128 Tool Manager Module

[0214] 129 Transparency Module

[0215] 130 Rendering Module

[0216] 132 Motion Tracking Module

[0217] 134 3D display

[0218] 136 2D Display

[0219] 138 Head-mounted display

[0220] 140 Scan Container Module

[0221] 142 Screenshot Module

[0222] 150 Multiplayer Module

[0223] 152 Fragment Manager Module

[0224] 154 Path Manager Module

[0225] 156 Brush Manager Module

[0226] 158 Preset Manager Module

[0227] 160 Automatic segmentation module

[0228] 162 Workspace Control Module

[0229] 164 Scan Container Module

[0230] 166 Measurement Control Module

[0231] 170 (cloud) servers

[0232] 180 Communication Network

[0233] 200 Methods

[0234] 201 Methods and Steps

[0235] 202 Methods and Steps

[0236] 203 Methods and Steps

[0237] 204 Methods and Steps

[0238] 205 Methods and Steps

[0239] 206 Methods and Steps

Claims

1. A system (100) for visualizing an organ to plan tumor resection before surgery and for providing an immersive view of a patient's anatomy, the system (100) comprising: a control module (102) comprising a data storage device (104) and processing resources (106) configured to store, access, and execute computer-readable program instructions; a server (170) communicating with the control module (102) via a communication network (180); a user interaction module (114) comprising a multiplayer module (150) configured to facilitate real-time interaction between a user and the system (100); a user input module (116) configured to control and receive input information from the user, wherein the user input module (116) includes a plurality of input devices (120), a virtual reality input module (122) for receiving input in a virtual reality environment, and a virtual reality control module (124) for performing operations based on information received from the virtual reality input module (122); and an input / output module (118) comprising a rendering module (130), a motion tracking device (132), a 3D display (134), a 2D display (136), and a head-mounted display (138) for immersive viewing; The user interaction module (114), the user input module (116), and the input / output module (118) communicate with each other and with the control module (102) via a communication interface (112); The control module (102) is configured to receive image data and processed data from the server (170) and store the image data and the processed data in the data storage device (104), wherein the image data comprises a 2D computed tomography image of the anatomical structure, and wherein the processed data comprises segmented data representing segments of the anatomical structure; The user interaction module (114) is configured to create visualizations of the anatomical structure in 2D and 3D based on the image data and the processed data received from the control module (102), and to edit the created 2D and 3D visualizations based on the input information received from the user input module (116); The input / output module (118) is configured to display the visualization on the corresponding 2D display (136) and the 3D display (134), and to provide an immersive view of the anatomical structure of the patient via the head mounted display (138), wherein the immersive view is generated from the image data and the processed data, preferably using a ray marching technique; The input / output module (118) is further configured to edit the 2D visualization, the 3D visualization, and the immersive view using the rendering module (130) based on the information received from the user interaction module (114), the user input device (120), the virtual reality input module (122), and the motion tracking device (132).

2. The system (100) according to claim 1, It is characterized by The system (100) further comprises: a remote display configured to display the 2D and 3D visualizations, and / or A module configured to deliver 2D and 3D visualizations on a mobile device.

3. The system (100) according to claim 1 or 2, It is characterized by The plurality of input devices (120) include at least one of a mouse, a joystick, a controller, a motion sensor, a keyboard, and a microphone. Preferably, the plurality of input devices (120) include a mouse, and the user input module (116) further includes a mouse control module (126) configured to perform predetermined functions using the mouse and the keyboard.

4. The system (100) according to any one of claims 1 to 3, It is characterized by The user input module (116) further includes a tool manager module (128) configured to manage virtual reality tools and / or a transparency plane module (129) configured to create a clipping plane to determine a camera-viewable scene.

5. The system (100) according to any one of the preceding claims, It is characterized by The user interaction module (114) further includes a segment manager module (152) configured to load segments of the analysis structure into the system. Preferably, the segment manager module (152) is further configured to store the status of each segment of the anatomical structure and create a list thereof.

6. The system (100) according to any one of the preceding claims, It is characterized by The user interaction module (114) further includes a path manager module (154) configured to create a list of paths for a camera to follow in the virtual reality environment and / or the 2D environment.

7. The system (100) according to any one of the preceding claims, It is characterized by The user interaction module (114) further includes a brush manager module (156) configured to create color segments in the virtual reality environment based on the input information.

8. The system (100) according to any one of the preceding claims, It is characterized by The user interaction module (114) further comprises a preset manager module (158), the preset manager module comprising a preconfigured array of segment settings for delineating viewpoints of the segmented data, preferably the preconfigured array of segment settings comprising the image data, automatic segmentation, bronchial vessels, arteries, veins, airways, solid segments, shell segments, complete anatomical structures.

9. The system (100) according to any one of the preceding claims, It is characterized by The user interaction module (114) further includes an automatic segmentation module (160) configured to create new segments of the analysis structure.

10. The system (100) according to any one of the preceding claims, It is characterized by The user interaction module (114) further includes a workspace manager module (162) configured to create a list of files and paths that can be linked to the new workspace.

11. The system (100) according to any one of the preceding claims, It is characterized by The input / output module (118) further includes a scan container module (140) configured to control positioning, scaling, and rotation of the displayed anatomical structure.

12. The system (100) according to any one of the preceding claims, It is characterized by The input / output module (118) further includes a screenshot module (142) configured to save a screenshot of the workspace.

13. The system (100) according to any one of the preceding claims, It is characterized by The data storage device (104) includes video memory, RAM, ROM and flash memory.

14. The system (100) according to any one of the preceding claims, It is characterized by The control module (102) further includes an authorization module (108) configured to authorize a user to access software installation and / or an automatic saving module (110) configured to automatically save a workspace.

15. The system (100) according to any one of the preceding claims, It is characterized by The user interaction module (114) further includes a measurement control module (166) configured to create measurements to be taken by the system (100).

Images