Method and system for improving visualization interaction with medical 2d and 3D

Abstract

Some embodiments relate to 3D visualization of medical image data. An overlay may be generated from a particular 2D image slice of the plurality of 2D image slices and anatomical entities segmented therein. In the overlay, the segmented anatomical entities are interactive for selection or deselection. A 3D visualization is displayed in which the selected one or more anatomical entities are visually distinguishable.

Description

TECHNICAL FIELD

[0001] The presently disclosed subject matter relates to a computer-implemented method, system and computer readable medium for medical 3D visualization of medical image data. BACKGROUND

[0002] Medical visualization in oncology plays an important role in modern medicine, which uses various types of scans, such as computed tomography scans (CT) or magnetic resonance imaging (MRI). Surgeons and other medical professionals are trained to understand these scan results in order to accurately identify tumors and determine the most effective strategy for patient treatment.

[0003] 3D visualization of these scans further helps surgeons to develop a surgical plan, especially in complex cancer cases where traditional scans are difficult to interpret. For example, in complex pancreatic cancer, where the tumor grows into a vein, it is difficult to assess its extent by scanning. With 3D visualization, surgeons get a direct overview of the patient's specific anatomy, enabling them to identify key anatomical landmarks and potential abnormalities, thereby minimizing the risk of injury during surgery.

[0004] Medical 3D visualization is still an area with room for improvement. SUMMARY

[0005] A method for medical 3D visualization of medical image data is provided herein. The medical imaging data comprises a plurality of 2D image slices, such as obtained by scanning a volume of a patient's body with a medical imaging device. In the method, an overlay is displayed for a particular 2D image slice of the plurality of 2D image slices, wherein segmented anatomical entities are interactable for selection or deselection in the overlay.

[0006] In addition to the overlay, a 3D visualization of the same anatomical entities is shown. The 3D visualization is generated from the plurality of 2D image slices and visually distinguishes the anatomical entities selected at least by the interactable segmented anatomical entities shown in the overlay. In a typical example, deselected anatomical entities are not shown, while selected anatomical entities are shown. Other options are possible, however, such as changing the transparency of, for example, deselected anatomical entities.

[0007] The method has the advantage that it allows the user (e.g. a medical practitioner) to study the overlay and / or the 2D image slices related thereto and select the anatomical entity therein as desired, without having to shift attention to other interfaces. Especially when analyzing medical 2D images (such as X-ray or similar scan images), it is important to maintain continuous focus. Shifting attention causes a disruption in the cognitive process, which can result in missed observations or inaccurate observations. By providing a lean interface, the method ensures that the practitioner remains immersed in the assessment.

[0008] Known systems for generating 3D visualizations of medical images have not established themselves in the medical field as such. In fact, medical practitioners mostly stick to 2D medical images. This can be due to the complexity of existing systems, as well as the skepticism of medical practitioners. Generally, new technologies involving 3D visualizations are complex to learn and often require special input and interaction systems and interfaces. Embodiments further address this problem by building on interfaces that medical practitioners are already familiar with: they can operate and handle the 2D viewer in the usual way and interact with the 3D viewer through this as well, as the anatomical entity is interactable.

[0009] The method can be implemented in an electronic device or system, such as a computer or a non-transitory computer readable medium.

[0010] It will be appreciated by the skilled person that the method can be applied to multi-dimensional image data acquired by various acquisition modalities, such as but not limited to standard X-ray imaging, computed tomography (CT), magnetic resonance imaging (MRI), ultrasound (US), positron emission tomography (PET), single photon emission computed tomography (SPECT), and nuclear medicine (NM), or any combination of these.

[0011] Embodiments of the method can be implemented on a computer as a computer implemented method, or as dedicated hardware, or as a combination of both. Executable code of embodiments of the method can be stored on a computer program product. Examples of computer program products include storage devices, optical storage devices, integrated circuits, servers, online software, etc. Preferably, the computer program product comprises non-transitory program code stored on a computer readable medium for executing an embodiment of the method when the program product is executed on a computer.

[0012] In one embodiment, the computer program comprises computer program code adapted to perform all or part of the steps of an embodiment of the method when the computer program is run on a computer. Preferably, the computer program is embodied on a computer readable medium.

[0013] Another aspect of the currently disclosed subject matter is a method for making a computer program downloadable. This aspect is used both when uploading a computer program and when making a computer program downloadable. By downloading the software according to the embodiments, a computer can be transformed into a device according to the embodiments. Attached Figure Description

[0014] Further details, aspects, and embodiments will be described by way of example only with reference to the accompanying drawings. Elements in the drawings are shown for simplicity and clarity and are not necessarily drawn to scale. In the drawings, elements corresponding to those already described may have the same reference numerals. In the drawings:

[0015] Figure 1 An example embodiment of a system for medical 3D visualization of medical image data is illustrated schematically.

[0016] Figure 2 An example embodiment of a system for medical 3D visualization of medical image data is illustrated schematically.

[0017] Figure 3 An example embodiment of a medical 3D visualization system for medical image data, including pre-computed 3D objects, is illustrated schematically.

[0018] Figure 4 An example of an embodiment of an arrangement for medical 3D visualization of medical image data is illustrated schematically.

[0019] Figure 5 An example of an embodiment of 3D visualization of medical image data is illustrated schematically.

[0020] Figure 6 Examples of embodiments for 3D visualization, overlay, and receiving selections are illustrated schematically.

[0021] Figure 7 An example of an embodiment of 3D visualization and overlay is illustrated schematically.

[0022] Figure 8 An example embodiment of a system for 3D visualization is illustrated schematically.

[0023] Figure 9 An example of an embodiment of a method for medical 3D visualization of medical image data is illustrated schematically.

[0024] Figure 10a A computer-readable medium having a writable portion according to an embodiment is illustrated schematically, the writable portion including a computer program.

[0025] Figure 10b A representation of a processor system according to an embodiment is shown schematically. List of reference numerals The following list of figure labels and abbreviations is for ease of interpretation. Figures 1-10b The accompanying drawings are provided for informational purposes only and should not be construed as limiting the scope of the claims. 100 Medical 3D Visualization Systems for Medical Image Data 101 processor system 102 Storage devices 103 Communication Interface 200 Medical 3D Visualization Systems for Medical Image Data 201 medical imaging data, including multiple 2D image slices 202 Overlay for specific 2D image slices 210 Medical Imaging Data Repository 220 segmentation model 230 Stack Generator 240 3D Visualization Tools 241 3D Visualization 250 User Interface 260 3D display 261 2D monitor 300 Medical 3D Visualization Systems for Medical Image Data 310 Pre-computer modeling of 3D objects for anatomical entities 321-323 3D objects for anatomical entities 331-333 Visualization of 3D Objects for Anatomical Entities 340 Combination 3D Visualization 400 layouts for medical 3D visualization of medical image data 410 2D Monitor 420 pointing device 430 Keyboard 440 3D Display 3D representation of 500 anatomical entities 510 Location of a specific 2D image slice in a 3D model 520 Position of 2D segments in 3D representation 530 Anatomical Entities 600 Medical 3D Visualization System for Medical Image Data 610 stacking 611 cursor Interactive fragments 612, 613, and 614 620 display of 3D visualization 700 Medical 3D Visualization Systems for Medical Image Data 710 stacked The segments selected in 712 and 713 720 display of 3D visualization The selected elements in 3D models 722 and 723. 800 Track control system for medical 3D visualization Cameras 801 and 802 810, 820 Orbital spheres surrounding the 3D model 3D visualization corresponding to 830 and 840 900 Methods for 3D Visualization of Medical Image Data 901 Retrieve medical imaging data from the medical imaging data repository 902 Segmenting anatomical entities from multiple 2D image slices 903 Generates and displays an overlay based on a specific 2D image slice and its segmented anatomical entities. 904 Receives user interaction to select one or more anatomical entities in the displayed overlay. 905 generates 3D visualizations from multiple 2D image slices 906 Display 3D Visualization 1000, 1001 Computer-readable media 1010 Store data 1020 Computer Program 1110 Processor System 1120 Processing Unit 1122 Memory 1124 Application-Specific Integrated Circuit 1126 Communication Components 1130 Interconnection 1140 Processor System Detailed Implementation

[0026] While the subject matter disclosed herein allows for many different embodiments, one or more specific embodiments are shown in the accompanying drawings and will be described in detail herein. It should be understood that this disclosure should be regarded as illustrative of the principles of the subject matter of this disclosure and is not intended to limit it to the specific embodiments shown and described.

[0027] In the following text, for ease of understanding, the units of the embodiment in the operational state are described in operation. However, it is clear that the various units are arranged to perform the functions described as being performed by them.

[0028] Furthermore, the subject matter disclosed herein is not limited to the embodiments, but also includes all other combinations of features described herein or recited in mutually different dependent claims.

[0029] In the context of tumor medical imaging, surgeons can greatly benefit from visualizing 3D versions of routine scans. Medical practitioners need to be able to obtain an overview and examine details of anatomical structures, such as abnormal arterial branches or tumor growth. However, complete control over navigation in 3D space can cause users to lose their sense of direction and find it difficult to find their way.

[0030] Surgeons and most oncologists work under immense time pressure, and learning how to navigate and interact in a 3D environment using unfamiliar input systems requires significant time and effort, something surgeons rarely do. For example, in certain workflows, surgeons may have to make decisions based on 2D medical images, so using a 3D environment would require switching back and forth between two different input systems and working methods.

[0031] Existing 3D medical visualization systems are disconnected from their 2D versions. For example, in the context of evaluating complex pancreatic tumor cases, surgeons often focus on routine scans and frequently fail to use and examine 3D visualizations, despite their potential benefits.

[0032] Some embodiments relate to 3D visualization of medical image data. An overlay can be generated based on a specific 2D image slice from multiple 2D image slices and segmented anatomical entities therein. In the overlay, the segmented anatomical entities are interactive to be selected or deselected. A 3D visualization is shown where one or more selected anatomical entities are visually distinguished.

[0033] This invention allows medical practitioners to use 3D navigation by enhancing 2D image interfaces they are already familiar with (e.g., traditional DICOM viewers). This enhancement makes relevant elements on the scan results interactive. For example, a surgeon can operate a 2D viewer as they would normally, but they can also choose to interact with it by selecting elements such as organs and vessels, thereby navigating the corresponding 3D visualization. This selection will trigger a dynamic 3D view of the scan. For example, a surgeon can control a camera that rotates in 3D around the selected element via keyboard input. Keyboard input can be used in combination with indicator input; for example, a mouse modifier such as holding down the CTRL key while using mouse interaction.

[0034] Furthermore, the embodiments have the advantage that 3D visualizations can be selected without taking one's eyes off the 2D image currently being studied.

[0035] Figure 1 An example embodiment of a system 100 for medical 3D visualization of medical image data is illustrated schematically. System 100 may include medical devices, such as a medical workstation. For example, system 100 may be used for medical 3D visualization of routine medical scans, such as CT scans.

[0036] System 100 can be configured to generate 3D visualizations from multiple 2D image slices. For example, medical practitioners can use this system to analyze medical images, such as to diagnose patients or plan medical procedures. Generating 3D visualizations can provide medical practitioners with useful insights because it can very accurately approximate the complex anatomy within a patient's body. Analysis of 3D visualizations can help physicians detect abnormalities in anatomical structures, such as tumors or abnormal arterial branches, to better formulate appropriate diagnoses, and / or make decisions about which medical procedures to choose, as well as how to develop the procedures, when to precisely plan them, and how to personalize them for each patient, thus contributing to providing the correct medical treatment.

[0037] Even when a tumor is detected in a medical 2D image, 3D segmentation can help visualize uncertain areas, providing insights into potential tumor size, such as the maximum tumor size, and damaged surrounding blood vessels.

[0038] Of course, in some challenging cases, such as complex cases of pancreatic disease, tumors may grow into veins, and the intertwined structures of tumors and veins can be difficult to distinguish in traditional 2D scans. In these cases, medical professionals can only see a shallow projection from above and must use their imagination to create a 3D image in their minds. 3D visualization can be helpful in assessing such situations. Furthermore, individual differences in anatomical structure can affect surgical planning, but these differences are difficult to understand from 2D images.

[0039] System 100 may include processor system 101, storage device 102, and communication interface 103. Storage device 102 may be, for example, an electronic storage device, a magnetic storage device, etc. The storage device may include a local storage device, such as a local hard disk or electronic storage device. Storage device 102 may include a non-local storage device, such as a cloud storage device. In the latter case, storage device 102 may include a storage interface to a non-local storage device. The storage device may include multiple discrete sub-storage devices, collectively forming storage device 102. The storage device may include volatile writable portions (e.g., RAM), non-volatile writable portions (e.g., flash memory), and non-volatile non-writable portions (e.g., ROM).

[0040] In various embodiments of communication interface 103, a variety of alternatives may be selected for the communication interface. For example, the interface may be a network interface to a local area network or wide area network (e.g., the Internet), a storage interface to an internal or external data storage device, an application programming interface (API), etc.

[0041] Storage device 102 can be a non-transient storage device. For example, storage device 102 can store data when powered on, such as a volatile storage device, such as a random access memory (RAM). For example, storage device 102 can store data when powered on or when not powered, such as a non-volatile storage device (e.g., flash memory).

[0042] System 100 can communicate internally, with other systems, and with other devices, external storage devices, input devices, output devices, and / or one or more sensors via a computer network. The computer network can be the Internet, an intranet, a local area network (LAN), a wireless LAN, etc. System 100 may include connection interfaces that are configured as needed to communicate either within or outside of system 100. For example, the connection interface may include connectors, such as wired connectors (e.g., Ethernet connectors, optical connectors), or wireless connectors (e.g., antennas, such as Wi-Fi, 4G, or 5G antennas).

[0043] Communication interface 103 can be used to receive digital data, such as medical imaging data, including data containing CT, MRI, PET, SPECT, or ultrasound images. Communication interface 103 can be used to send display signals to cause a display to show a 3D visualization generated by system 101. The display showing the 3D visualization can be a holographic display and / or an augmented reality head-mounted display, through which the 3D visualization can be viewed as a hologram. The display showing the 3D visualization can be a VR device, possibly with a mixed reality mode for viewing a medical workstation, or with a virtual medical workstation in VR.

[0044] 3D visualization includes a 3D representation of an anatomical entity, generated from multiple 2D image slices. This 3D representation can be viewed as a stack or connected stack of such or similar 2D image slices in a third dimension. The type of 3D visualization obtained in a 3D display, as a stored image, can be a TIFF (Tagged Image File Format) file, a common format for storing holographic-type visualizations. Representation in a 3D scene includes a 3D model comprising 3D structures, which may include providing surface textures for each structure; and adding lighting using computer graphics techniques to make the 3D scene visible.

[0045] System 100 may have a user interface, which may include well-known elements such as a display or touchscreen. The user interface may be configured to accommodate user interaction, performing selection of anatomical entities as interactive elements on a displayed overlay. This user interaction may be performed via a touchscreen, using a touchscreen pen and / or finger, or via an external pointing device such as a mouse, cursor device, or indicator. User interaction may include selecting specific anatomical entities to be visually distinguished or only shown. It may also include deselecting anatomical entities to make them visually distinguishable or to hide them completely from the generated 3D visualization.

[0046] The execution of system 100 can be implemented in a processor system. System 100 may include functional units for implementing various aspects of the embodiments. The functional units may be part of the processor system. For example, the functional units shown herein may be implemented wholly or partially as computer instructions stored in the system's memory and executable by the processor system.

[0047] A processor system may include one or more processor circuits, such as a microprocessor, CPU, GPU, etc. System 100 may include multiple processors. The processor circuits may be implemented in a distributed manner, for example, as multiple sub-processor circuits. For example, system 100 may use cloud computing.

[0048] Typically, system 100 includes a microprocessor that executes appropriate software stored on the system; for example, the software may have been downloaded and / or stored in a suitable memory, such as volatile memory like RAM or non-volatile memory like flash memory.

[0049] Instead of using software to implement functionality, system 100 can also be implemented wholly or partially using programmable logic, such as a field-programmable gate array (FPGA). The system can also be implemented wholly or partially as a so-called application-specific integrated circuit (ASIC), such as an integrated circuit (IC) customized for a specific purpose. For example, the circuitry can be implemented using CMOS, for instance, using hardware description languages ​​such as Verilog or VHDL. Specifically, system 100 may include circuitry for, for example, graphics processing and / or arithmetic processing.

[0050] In the hybrid embodiment, the functional unit is implemented in part in hardware, such as as a coprocessor, and in part in software, which is stored in the system and run in the system.

[0051] Figure 2 An example embodiment of a system 200 for medical 3D visualization of medical image data is illustrated schematically. For example, system 200 can be used in, for example, reference... Figure 1The system illustrated is implemented on a computer. For example, system 200 can be implemented on a computer.

[0052] In this embodiment, a 2D display is used to view 2D image slices, and another display is used to view 3D visualizations. For example, the latter could be a holographic display, etc. For instance, this embodiment can be used to navigate 3D visualizations derived from 3D imaging scans. Navigation is useful, for example, for assessing and preparing for surgery in complex cancer cases, and can also be used in other situations to help medical practitioners (e.g., surgeons) understand specific anatomical structures and identify key or abnormal landmarks.

[0053] In embodiments, software extensions can be provided to conventional DICOM viewers to enable them to navigate 3D medical data, depending on the embodiment. For example, the extension can be configured to connect the DICOM viewer to an external display (e.g., a holographic display). Other possibilities are also discussed herein, for example, with reference to... Figure 4 .

[0054] System 200 is configured to retrieve medical imaging data 201 from a medical imaging data repository 210. Medical imaging data 201 includes multiple 2D image slices. For example, the multiple 2D image slices may include one or more of the following: CT images, MRI images, PET images, SPECT images, and ultrasound images. The medical imaging device generates a successive series of 2D image slices representing anatomical volumes, such as a patient's brain or heart. For example, this could be a detailed image of a tumor within a human patient's liver or lungs.

[0055] System 200 may include segmentation model 220. Segmentation model 220 may be implemented as software stored in system 200. Segmentation model 220 may also be accessible online, for example, stored on an external server, such as as a cloud service. Segmentation model 220 is configured to segment anatomical entities in multiple 2D image slices.

[0056] Segmentation model 220 can use segmentation algorithms to detect anatomical elements, such as organs, tissues, vessels, and tumors, in medical imaging data 201 (e.g., CT scans). For example, segmentation model 220 may include a U-Net neural network. U-Net, as a convolutional neural network architecture, has achieved good results in medical image segmentation tasks. This means that segmentation model 220 can effectively delineate complex structures and provide detailed insights into the anatomical structures present in CT scans.

[0057] In addition to or in place of U-Net, segmentation model 220 can also use other algorithms and / or neural network architectures.

[0058] Segmentation of medical 2D images can also be achieved using techniques such as active contour models, which adaptively adjust curves on the image to identify and separate desired structures. Furthermore, random forests (a machine learning algorithm) can also be used to segment complex medical images. Another related technique in the field of medical imaging is image registration, where images are mapped to templates. Specifically, this process may involve aligning and transforming different sets of images so that they spatially correspond to a template or reference image.

[0059] After segmenting multiple 2D images in medical imaging data 201, 3D models of organs and / or other anatomical structures can be constructed, for example in 3D visualization tool 240 or in dedicated 3D structural units. Since each 2D image slice represents a thin portion of an anatomical volume in the body, when combined sequentially, they together describe the 3D volume.

[0060] For example, within this 3D volume, each pixel in the 2D image can be converted into a voxel. Voxels can be labeled with segmentation tags and / or tissue types, which can be obtained from 2D segmentation, such as bone, organ tissue, air, specific organ, blood vessel, etc. Various surface rendering techniques can be used to convert segmented voxels into 3D objects, such as polygonal meshes, typically triangles. For example, the moving cube algorithm can be used to generate a surface representation from the voxel data. For example, Paul Burke's polygonization algorithm, described at https: / / paulbourke.net / geometry / polygonise / (included by reference in this document), can be used. Further processing can also be performed, such as smoothing out imperfections.

[0061] Alternatively, 3D segmentation can be obtained directly from multiple 2D image slices. For example, a neural network (such as U-net) can receive multiple 2D image slices and generate 3D segments. 2D segments of a specific image slice can then be obtained through 3D segmentation.

[0062] System 200 is configured to display image slices to a user of system 200 on display 261. Display 261 may be a 2D display. For example, system 200 may be configured to select a specific image from a plurality of 2D image slices. For example, system 200 may be configured to receive a selection of a specific image from the plurality of 2D image slices from the user and display it. The user can change the selection of the specific image to be displayed by interacting with system 200 (e.g., via a keyboard, etc.). For example, displaying a specific 2D image may require the use of a DICOM viewer.

[0063] System 200 may include overlay generator 230 configured to generate overlay 202 for a specific 2D image slice among a plurality of 2D image slices in medical imaging data 201.

[0064] Overlay 202 illustrates segmented anatomical entities within a specific 2D image slice. By displaying overlay 202 (typically overlaid on the specific 2D image that generated overlay 202), the user can see the various anatomical entities identified by the segmentation algorithm. Overlay 202 and the specific image are displayed on monitor 261.

[0065] In overlay 202, there are several ways to visually distinguish segmented anatomical entities; for example, one or more of the following: color coding, texture mapping, opacity, outline or border style, brightness or contrast, and highlighting. Specifically, visual distinction can indicate whether a segment and its corresponding anatomical entity are selected; and / or whether the segment is hovered over.

[0066] In embodiments, visual distinctions can be obtained from selected and deselected anatomical entities according to predetermined rules. For example, when a segment is selected, adjacent segments that are not selected themselves may still be displayed in 3D, but may be displayed in a different style, such as semi-transparent. For example, the rules may display deselected segments based on resectability criteria, such as examining the relationship between critical structures (e.g., blood vessels) and, for example, the segmented tumor.

[0067] Interestingly, the segmented anatomical entities in overlay 202 are interactive, allowing selection or deselection within the overlay 202. System 200 includes a user interface 250 for interacting with visually distinguishable segmented anatomical entities in overlay 202. User interface 250 is configured to receive selections of one or more anatomical entities in overlay 202 displayed on display screen 261 when a user interacts with the displayed segmented anatomical entities. For example, system 200 (e.g., user interface 250) can receive input signals from a pointing device (e.g., a mouse, cursor, or indicator) from the user within the displayed overlay 202, such as clicking or hovering over a segmented anatomical entity shown in overlay 202.

[0068] Please note that some indicator devices do not support hovering over sections. For example, mice support hovering, and eye-tracking also supports it. On the other hand, touchscreens support selection (such as clicking) but generally do not support hovering.

[0069] The 2D viewer may then receive events, such as those from the operating system, including mouse events (e.g., mouse press events) and the coordinates of the events. An implemented program that can then be integrated into system 200 can determine that the coordinates lie within the interactive area, specifically within a portion of overlay 202.

[0070] For example, system 200 (e.g., an image viewer) may receive events, for example, from the operating system of system 200. These events may include indicator events, such as mouse events. Indicator events may include click or hover events, and the coordinates of the event. For example, this could be a mouse press event. The program might determine that the coordinates are within an interactive area, specifically within a fragment overlaid in 202.

[0071] Mouse clicks or hover operations on a portion of an image slice can be detected using an image or an event listener that shows the display area overlaid 202. The event listener listens for events in the image, such as mouse actions like "click" or "hover." Once an event is detected, its X and Y coordinates can be captured; for example, the coordinates of the mouse pointer at the time the event occurs. Using the captured coordinates, system 200 can check whether the point is located within a segment of the image. For example, the coordinates can be compared to a mask representation of the segment.

[0072] In this embodiment, receiving selected user interaction includes receiving user eye-tracking signals within the displayed overlay 202 obtained from the camera. For example, a user watching a clip can be detected. This can be considered a hover signal. The user can also emit a click event signal, for example, by pressing a key on the device. The user eye-tracking signals can be obtained from an external camera or from an internal camera embedded in the 2D display.

[0073] One advantage of System 200 is that the interaction is located in the same position as Overlay 202, and typically, the specific image is also visible. This means that users can examine the segments shown in Overlay 202, and / or medical image data in specific images, and select or deselect segments within Overlay 202 without switching focus from Overlay 202 to another user interface. This is advantageous because analyzing medical images involves carefully following the representation of anatomical structures in the images, where even a brief shift of focus to another input interface risks losing the ongoing analytical chain. This can be avoided by placing the interactive segments in the same position as the parts and / or the image itself. Users can select or deselect segments while continuing to view Overlay 202 and / or the specific image associated with it.

[0074] In a sense, related elements, such as organs, tissues, tumors, and blood vessels, become interactive. For example, these elements in overlay 202 can respond to hover and are selectable, for example, by mouse click. In addition to a mouse, any other pointing interface that allows local interaction can be used, such as a touchscreen, stylus, and / or external pointing devices, such as a mouse, cursor device, or indicator.

[0075] When anatomical entities are selected or deselected, this can be reflected in 3D visualization, as will be discussed further in this paper. There can be a distinction between hovering and clicking a segment. For example, hovering over a segment can be considered a temporary selection, lasting for the same duration as the mouse hovering over that section. When a selection event such as clicking occurs, the segment can be selected until later deselection. These two events can also be visually distinguishable, for example, a temporary selection during hovering can be displayed with different visual styles, such as different colors.

[0076] The user's segment selection will affect the 3D visualization. System 200 can generate some pre-selections based on the anatomical entities segmented from the plurality of 2D image slices. For example, the pre-selections can be empty, allowing the user to add segments to the selection. Alternatively, the initial selection can be complete, i.e., selecting all anatomical entities or segments. The user can then change the initial selection by deselecting segments.

[0077] In this embodiment, the preselection is a non-empty and strict subset of the segmented anatomical entities; that is, not all segmented anatomical entities are selected in the preselection, nor are all segmented anatomical entities deselected. For example, system 200 may generate a preselection to help the user of system 200 quickly find the most suitable selection for the current task, such as analyzing images, making diagnoses, or planning workflows. The preselection rules may take into account that if an entity is selected, other parts will also be visible by default. For example, a tumor may be close to a vein, such as the portal vein, so if the tumor is selected, the vein, as a key structure, may also be shown if the tumor is very close to it (e.g., close to the portal vein segment). On the other hand, if a key structure is selected, the tumor may be highlighted relative to that structure even if the tumor itself is not selected.

[0078] An optional anomaly detection algorithm in system 200 can be used to identify abnormal regions in segmented medical images. For example, the algorithm can learn normal features of an image or regions therein and flag any deviations from normal values ​​exceeding a threshold. This anomaly detection algorithm can be trained using unsupervised learning. The anomaly detection algorithm can also be trained to detect specific anatomical abnormalities in anatomical entities, such as one or more of the following: lesions, abnormal arterial branches, tumors, and skeletal abnormalities. For example, a pre-selection may include identified anatomical abnormalities. Skeletal abnormalities may include fractures, skeletal deformities, etc.

[0079] Pre-selection of anatomical entities and / or corresponding segments can be achieved using rule-based systems, such as expert systems, which analyze data using a predefined set of rules, including, for example, segment labels and / or detected anomalies. For instance, a rule-based system might have rules for marking regions larger than a certain size, with irregular boundaries, or with pixel intensities exceeding typical ranges. These rules can be developed by medical experts.

[0080] Pre-selection of anatomical entities can also or alternatively utilize a machine learning model trained to label segments to determine whether pre-selection is necessary. For example, supervised classification can be used to train this model. The model can be trained on a set of 2D images or multiple medical imaging datasets (e.g., on a set of multiple images) and on corresponding selections made by the user of system 200. In this way, the model can learn what choices the user of system 200 typically makes and learn to select segments in the same manner.

[0081] System 200 is configured to allow users to modify pre-selections, thereby obtaining choices for creating 3D visualizations. Using pre-selections allows users to move more quickly from initially viewing a new set of images to obtaining useful 3D visualizations.

[0082] User interactions with the overlay 202 may be reflected in both the overlay 202 and the 3D visualization. For example, if a user hovers the mouse over a segment, the hovered segment may be highlighted in the overlay 202. The corresponding 3D objects in the 3D visualization can also be adjusted; for example, the opacity may be changed, such as increased. If a user selects one or more of these segments, for example, by clicking them, the portion on the overlay 202 may visually represent this; the corresponding 3D element may be highlighted. The visual representation of hovered or selected segments may differ in the overlay 202 and / or the 3D visualization. For example, the selected segment may be filled with color on a 2D image slice. These interactions enable users (e.g., surgeons) to navigate in a 3D environment, examine anatomical landmarks in detail, and gain a comprehensive understanding of patient-specific anatomy.

[0083] System 200 includes a 3D visualization tool 240 configured to generate a 3D visualization 241 from multiple 2D image slices. The 3D visualization 241 visually distinguishes one or more selected anatomical entities.

[0084] The 3D visualization 241 can be displayed on the display 260. The display 260 can be separate from the display 261. For example, the display 260 can be a stand-alone projection device, such as a holographic display and / or an augmented reality (AR) head-mounted display. The display 260 is preferably capable of 3D projection, such as a stereoscopic 3D display, in which different images are shown to the user's two eyes.

[0085] Display 260 may be a 2D display, but is used to display 3D visualization 241. For example, it may use 2.5D or pseudo-3D. For example, techniques such as parallax scrolling, shadows, and occlusion may be used to indicate the depth of 3D visualization 241. In fact, display 260 and display 261 may be the same display, for example, using different areas of the display for 2D and 3D visualization.

[0086] Preferably, the 2D image viewer (e.g., a DICOM viewer) is synchronized with the 3D visualization viewer in real time so that the 3D will react immediately when interacting with one or more elements on the 2D image.

[0087] The selection of anatomical entities can be made using multiple specific images. For example, a first specific 2D image can be displayed to the user. The user can select anatomical entities in the first specific 2D image displayed to him / her, and can also modify the selection. Then, a second specific 2D image may be displayed to the user. The user can then select anatomical entities in the second specific 2D image displayed to him / her, and can also modify the selection. This allows for the selection / deselection of anatomical entities that are not visible in a particular 2D image.

[0088] The 3D visualization tool 240 can visually distinguish the selection of anatomical entities in 3D visualization 241 in a variety of ways.

[0089] For example, 3D visualization tool 240 may include selected anatomical entities and / or exclude deselected anatomical entities in 3D visualization 241. In other words, 3D visualization may only show the selected anatomical entities and not the other entities. In an embodiment, deselected but hovered anatomical entities may be included in 3D visualization 241, possibly with a different visual style.

[0090] For example, 3D visualization tool 240 may include deselected anatomical entities in 3D visualization 241, but using a different visual style. For instance, selected and deselected anatomical entities may differ in one or more of the following aspects: color coding, texture mapping, opacity, outline or border style, brightness or contrast, saturation and / or desaturation, and highlighting. Furthermore, deselected anatomical entities hovering over them may be displayed with a different visual style.

[0091] In one embodiment, overlaid 2D image slices are shown, which are then used to control the appearance of the corresponding 3D visualization. However, system 200 can also be configured to allow modification of selections in the displayed 3D visualization. For example, a user can select or deselect 3D objects displayed in the displayed 3D visualization. The user's selection can then be modified accordingly. Thus, overlay 202 can be modified to show segments that have been selected or deselected. Thus, 3D visualization 241 can be modified to show segments that have now been selected or deselected.

[0092] 3D visualization can utilize camera orientation, for example, including the camera orientation and / or direction and / or zoom level relative to a segmentation of multiple 2D image slices corresponding to a 3D object. System 200 and / or the user can adjust the 3D visualization to allow for a better view of the displayed 3D objects. For example, if a new anatomical entity is selected, the camera position for the 3D visualization can be determined so that the corresponding 3D object organ is centered. The camera position can provide the user with a specific viewpoint for examining a particular 3D object (e.g., the last selected object). For example, the camera orientation can be selected to avoid overlap of objects within a region of interest or object.

[0093] 3D visualization241 may have navigation control functions, such as changing the camera orientation and zoom level. For example, in display mode, the camera rotates around the overlapping area to display the involved organs, blood vessels, and cancerous tissue with reduced opacity, allowing for clearer observation of their interactions. Surgeons can use the magnification function to examine relevant points in detail.

[0094] For example, system 200 may allow users to save entity selections and / or camera orientations. These can be saved during planning and retrieved later, such as during preparation or surgery.

[0095] Figure 3An example embodiment of a medical 3D visualization system 300 for medical image data, including pre-computed 3D objects, is illustrated schematically. System 300 may include system 200. In this embodiment, segmentation model 220 forwards segmentation results to a pre-computer 310 for 3D objects, processing anatomical entities in multiple 2D image slices. These 3D objects may be data structures configured to represent objects in three dimensions. The objects can then be rendered, for example, by voxels representing data values ​​on a regular grid, a common practice in medical data visualization; other options may include polygonal representations on a grid, in which case vertex coordinates may also include information about the object, or 3D objects can be constructed via parametric analysis, triangulation, and intersecting 3D cubic splines, as in photogrammetry. The pre-computer 310 then pre-computes various 3D objects 321, 322, 323 corresponding to the anatomical entities in the multiple 2D image slices. Specific anatomical entities can be identified in the multiple 2D image slices using artificial intelligence algorithms or similar methods. This algorithm can detect anatomical entities in CT scans, such as organs, tissues, blood vessels, and tumors, and / or create corresponding 3D visualizations.

[0096] Generating the corresponding 3D visualization may include pre-computed 3D objects corresponding to the selected anatomical entity and / or visually distinguish the 3D objects within the 3D visualization. Then, based on the various 3D objects 321, 322, 333, various 3D visualizations 331, 332, 333 for the 3D objects of the anatomical entity in multiple 2D image slices can be created, for example, using 3D visualization tool 240. The 3D visualizations 331, 332, 333 are then merged into a single 3D visualization 340.

[0097] Generating visual representations of 3D objects separately and then combining them in a single view is a highly efficient approach. This combination may take into account occlusion, as well as shadows or other lighting effects. However, it's also possible to pre-compute 3D objects without generating separate visual representations. For example, pre-computed 3D objects corresponding to selected entities can be included to form a 3D model, which is then rendered as a whole. Other options exist as well.

[0098] Pre-computed 3D objects corresponding to anatomical entities can be stored, for example, in a medical image library. This storage may be temporary, such as short-term storage, for example, during the entire process of generating a corresponding session for 3D visualization, or as long as a 2D image slice is displayed, after which the 3D object can be removed from storage. The storage of pre-computed 3D objects corresponding to anatomical entities can also be non-temporary, in which case they can be stored in shared storage, such as cloud storage, and can be shared, for example, within a hospital's medical imaging department, and / or shared throughout the hospital, for example, for research and / or training purposes.

[0099] Figure 4 An example embodiment of device 400 is schematically illustrated for medical 3D visualization of medical image data. A 2D display 410 displays an overlay generated from a specific 2D image slice from a plurality of 2D image slices and their segmented anatomical entities; within the overlay, segmented anatomical entities can be interactively selected or deselected. The 2D display 410 may include a so-called DICOM (Digital Imaging and Communications in Medicine) viewer. The 2D display may also include a touchscreen. A keyboard 430 is available for user input, such as navigation within the displayed overlay. A pointing device 420 (possibly located outside the keyboard 430) is available for user input, such as selecting one or more anatomical entities within the displayed overlay, the selection being achieved through user interaction with the displayed segmented anatomical entities using the pointing device 420. The pointing device 420 may include a mouse, cursor device, active stylus, passive stylus, and / or indicator. The 2D display 410 may also include a touchscreen. A 3D display 440 can display the generated 3D visualization. The 3D display 440 may be a holographic display and / or an augmented reality head-mounted display.

[0100] Figure 5 An example embodiment of 3D visualization 500 of medical image data is schematically illustrated. In this embodiment, the 3D representation 500 of the visualizeable anatomical entity 530 can display anatomical structures of potential interest, such as abnormal arterial branches and / or tumor growth. Figure 5 In the illustrated embodiment, the location 510 of a specific 2D image slice among multiple 2D image slices is displayed and shown on the 3D model 500. Location 510 can be displayed using a 2D plane within the 3D representation 500; this plane can be limited to its intersection with the displayed anatomical entity 530 in the 3D presentation, thus forming a segment 520. This can be shown simply as an intersection line, for example, as a dashed line or other shading, and / or color-coded. For ease of orientation, the plane can also be displayed outside the anatomical entity 530 displayed in the 3D presentation 500. It can be displayed as a bounded, finite plane or as an infinite plane extending throughout the entire 3D representation 500.

[0101] The 3D representation 500 can be displayed on a 3D display in a specific orientation. A default orientation could be to display the patient's anatomical entity in the position the patient would be in when placed on the operating table. The position 520 of a segment 520 of an anatomical entity corresponding to a specific 2D image slice, which is part of the position 510 of that 2D image slice, may also be visible in the 3D representation 500. Position 520 then indicates the exact location of that segment within the anatomical entity.

[0102] Figure 6 An example embodiment of 3D visualization, overlay, and selection is schematically illustrated. According to the embodiment, a medical 3D visualization system 600 for medical image data may include a 2D display and a 3D display. The system 600 may include an overlay 610, which may be displayed on the 2D display. The displayed overlay 610 may be generated from specific 2D image slices and segmented anatomical entities therein. The overlay 610 may include interactive segments 612, 613, 614, which may correspond to the segmented anatomical entities. The overlay 610 may receive user interaction. It may receive user interaction in the form of input received during interaction with a cursor 611. The cursor 611 may send input it may receive from a pointing device (e.g., a mouse, cursor device, active stylus, passive stylus, and / or indicator).

[0103] In user interaction, cursor 611 can select or deselect interactive segments 612, 613, and 614. User interaction may then include receiving user input signals. User input signals may include clicks and / or hovers. When an anatomical entity is clicked and / or hovered over in the displayed overlay 610, it may be visually distinguished in response to the user interaction via, for example, by means of color change and / or highlighting. User interaction can be interpreted as selecting and / or deselecting one or more anatomical entities in the displayed overlay. In response to the user interaction, the visual representation of the displayed 3D visualization 620 may change accordingly based on the selected and / or deselected anatomical entity through interaction between the 2D model displayed in the 2D display overlay 610 (e.g., the selected anatomical entity) and / or the 3D model displayed in the 3D visualization 620 on the 3D display. Then, the 3D display can show the generated 3D visualization 620, wherein the selected one or more anatomical entities can be displayed by visually distinguishing one or more anatomical entities that have been selected and / or deselected, for example, including only the selected one or more anatomical entities, excluding the deselected anatomical entities, and / or in a manner that visually distinguishes them from the deselected anatomical entities and / or anatomical entities that have neither been selected nor deselected.

[0104] Visually distinguishing one anatomical entity from another may include differentiating opacities, color coding, shadow lines, texture mapping, outline or border styles, brightness or contrast, saturation and / or desaturation and / or highlighting. Using the same methods, selected and / or deselected anatomical entities can be further visually distinguished within overlay 610.

[0105] In this configuration, two of the three interactive segments 611-613 (612 and 613) display only borders without fill, while segment 614 displays both borders and fill color. In both cases, the segments can be transparent to reveal the corresponding 2D image displayed below.

[0106] User interaction generating user input can also include responding to user input by replacing a specific 2D image slice among a plurality of 2D image slices currently displayed with its overlay 610 with another 2D image slice among the plurality of 2D image slices. The new overlay can be displayed along with the new slice. This allows the user to navigate through the 2D slices. This input can also come from a pointing device, such as a mouse, cursor device, active stylus, passive stylus, and / or indicator. If the user replaces a specific 2D image slice, the overlay will also change to the overlay corresponding to the new 2D image slice.

[0107] System 600 can also receive pre-selections of anatomical entities segmented from multiple 2D image slices. This pre-selection result can be obtained through an algorithm (e.g., an artificial intelligence algorithm) and displayed by system 600 on display 610. The user can interact with this pre-selection result. In response to user interaction, the user can modify this pre-selection result to obtain a final selection for one or more anatomical entities. The final selection can then be displayed in 3D visualization 620.

[0108] System 600 can also receive selections of one or more anatomical entities in a displayed 3D visualization. These selections may come from user input or from the result of an algorithm (such as an artificial intelligence algorithm). System 600 can then display the modified selections in a 3D visualization 620 and / or a 2D overlay 610.

[0109] Figure 7 An example embodiment of 3D visualization and its overlay is illustrated schematically. A system 700 for medical 3D visualization of medical image data may include an overlay 710, selected segments 712, 713 corresponding to anatomical entities on the overlay 710, a displayed 3D visualization 720, and corresponding selected elements 722, 723 in the 3D model displayed in the 3D visualization 720.

[0110] When the user selects user interaction segments 712 and 713, they can be visually distinguished. Correspondingly, elements 722 and 723 corresponding to the selected segments can be visually distinguished in the 3D visualization 720. In the 3D visualization 720, the camera position can be adjusted based on the user input (e.g., selecting one or more segments 712, 713) to center the selected elements 722 and 723 and point them with the correct orientation. Furthermore, the selected elements 722 and 723 can be correctly scaled so that they fit properly into the 3D visualization, and / or the user can investigate and analyze them appropriately.

[0111] Figure 8 An example embodiment of a 3D visualization system 800 is illustrated schematically. In system 800, generating 3D visualizations 830, 840 may include defining or selecting 3D cameras 801, 802 for selected anatomical entities. For example, 3D cameras 801, 802 may be selected, and the 3D visualization system 800 may be connected to the 3D cameras 801, 802. 3D visualization may also include selecting camera orientations for the 3D cameras 801, 802 relative to 3D models of one or more anatomical entities, including camera position and / or camera orientation. The orientation of the 3D model may be fixed in a default orientation, preferably the default orientation of the 3D model aligns with the orientation of the anatomical entity simulated by the 3D model during surgical procedures.

[0112] The 3D visualization position (including position and orientation) of 3D cameras 801 and 802 can be adjusted in response to user input, allowing viewing around the selected anatomical entity. For example... Figure 8 The 3D cameras 801 and 802, represented by black dots, can be controlled via user input, such as through a user interface (e.g., a touchscreen) or via keyboard input. In other words, the 3D visualization system 800 is configured to provide control commands to move the 3D cameras 801 and 802 in response to modifications of selections obtained from user interaction, thereby placing the selected anatomical entity at the 3D center. Orbital spheres 810 and 820 can represent the degrees of freedom for the movement of the 3D cameras 801 and 802 around the 3D model. The movement of the 3D cameras 801 and 802 can be controlled along the orbital spheres 810 and 820: the position and orientation of the 3D cameras 801 and 802 can be controlled along the orbital spheres 810 and 820, which represent orbits around the 3D model. Because the 3D model is in a fixed orientation, the 3D cameras 801 and 802 can be smoothly controlled to move to various positions and orientations on the orbital spheres 810 and 820 around the 3D model. Figure 8A 3D model of the selected anatomical entity is displayed, and orbital spheres 810 and 820 are drawn around it. Note that orbital spheres 810 and 820 may not typically be displayed to the user. The user can then control 3D cameras 801 and 802 via input, causing the 3D cameras 801 and 802 to rotate and scale around the selected anatomical entity in the 3D model, thereby producing 3D visualization effects 830 and 840 corresponding to the control of 3D cameras 801 and 802.

[0113] In system 800, the movement of 3D cameras 801 and 802 can also be controlled in response to modifications in the selection of anatomical entities obtained through user interaction, so as to center the selected anatomical entity in 3D. Since user interaction may include selection, for example by clicking on a specific part corresponding to an anatomical entity, such selection can bring the specific anatomical entity to a specific viewpoint in the resulting 3D view of 3D visualization 830 and 840 of 3D cameras 801 and 802, for example by optimizing orientation to make the specific anatomical entity more centered in the view, and / or by means of zooming in so that the user can thoroughly examine it.

[0114] In embodiments, 3D visualizations 830, 840 may include selected anatomical entities displayed with reduced opacity, and the display of 3D visualizations 830, 840 may include animations, for example, from 3D cameras 801, 802, rotating around the selected anatomical entities on orbital spheres 810, 820. The animations may be displayed in response to user input. The 3D cameras may then rotate around overlapping areas of the selected anatomical entities, showing specific anatomical entities, such as involved organs, vessels, and cancerous tissue, in the animation, and reducing their opacity so that the user can observe the interactions between the anatomical entities more thoroughly. Based on user input, the zoom capabilities of the 3D cameras 801, 802 can be used to observe specific anatomical entities more closely.

[0115] In the embodiments, in the 3D visualization 830, 840, in response to user input, when navigating using 3D cameras 801, 802, it is possible to switch between different types of models: for example, it is possible to switch from a 3D model showing a specific anatomical entity based on the selection to a 3D model showing different anatomical entity selections.

[0116] Figure 9An example embodiment of a medical 3D visualization method for medical image data is illustrated schematically. Step 901 may include retrieving medical imaging data from a medical imaging data repository, the medical imaging data comprising multiple 2D image slices. Step 902 may include segmenting anatomical entities in the multiple 2D image slices using a segmentation model. Step 903 may include generating and displaying an overlay from specific 2D image slices of the multiple 2D image slices and their segmented anatomical entities, wherein, in the overlay, segmented anatomical entities can be interactively selected or deselected. Step 904 may include receiving a selection of one or more anatomical entities in the displayed overlay from user interaction with the displayed segmented anatomical entities. Step 905 may include generating a 3D visualization from the multiple 2D image slices, the 3D visualization visually distinguishing the selected one or more anatomical entities. Step 906 may include displaying the 3D visualization.

[0117] It is possible to perform this method in many different ways, as will be apparent to those skilled in the art. For example, the steps can be performed in the order shown, but the order of the steps can be changed or some steps can be performed in parallel. Furthermore, other method steps can be inserted between the steps. The inserted steps may represent an improvement to the method as described herein, or may be unrelated to the method. For example, some steps may be performed at least partially in parallel. Moreover, a given step may not be fully completed before the next step begins. Embodiments of this method can be performed using software comprising instructions for causing a processor system to perform method 900. The software may only include steps taken by a specific sub-entity of the system. The software can be stored in a suitable storage medium, such as a hard disk, memory, optical disk, etc. The software can be transmitted as a signal via a wired, wireless, or data network (e.g., the Internet). The software is available for download and / or remote use on a server.

[0118] It should be understood that the currently disclosed object also extends to computer programs, particularly computer programs on or within a carrier, suitable for putting the currently disclosed object into practice. The program may be source code, object code, code between source code and object code (e.g., in a partially compiled form), or any other form suitable for implementing embodiments of the method. Embodiments relating to the computer program product include computer-executable instructions corresponding to each processing step of at least one of the illustrated methods. These instructions may be subdivided into subroutines and / or stored in one or more files that may be statically or dynamically linked. Another embodiment relating to the computer program product includes computer-executable instructions corresponding to each device, unit, and / or portion of at least one of the illustrated systems and / or products.

[0119] Figure 10aA computer-readable medium 1000 having a writable portion 1010 and a computer-readable medium 1001 also having a writable portion are shown. The computer-readable medium 1000 is shown in the form of an optically readable medium. The computer-readable medium 1001 is shown in the form of an electronic memory, in this example, a memory card. The computer-readable media 1000 and 1001 can store data 1020, wherein the data can indicate instructions that, when run by a processor system, cause the processor system to perform an embodiment of the method according to an embodiment. The computer program 1020 can be embodied on the computer-readable medium 1000 as a physical marker or by magnetization of the computer-readable medium 1000. However, any other suitable embodiments are conceivable. Furthermore, it should be understood that although the computer-readable medium 1000 is shown herein as an optical disc, the computer-readable medium 1000 can be any suitable computer-readable medium, such as a hard disk, solid-state storage, flash memory, etc., and may be non-recordable or recordable. The computer program 1020 includes instructions required for causing a processor system to execute an embodiment of the method for medical 3D visualization of medical image data.

[0120] Figure 10b A schematic representation of a processor system 1140 according to an embodiment for a 3D visualization system is shown. The processor system includes one or more integrated circuits 1110. The architecture of the one or more integrated circuits 1110 is as follows: Figure 10b The circuit 1110 is schematically illustrated. Circuit 1110 includes a processing unit 1120, such as a CPU, for running computer program components to perform methods according to embodiments and / or implement modules or units thereof. Circuit 1110 includes a memory 1122 for storing programming code, data, etc. A portion of the memory 1122 may be read-only. Circuit 1110 may include a communication element 1126, such as an antenna, a connector, or both. Circuit 1110 may include an application-specific integrated circuit 1124 for performing some or all of the processing defined in the method. Processor 1120, memory 1122, application-specific IC 1124, and communication element 1126 may be interconnected via interconnect 1130 (e.g., a bus). Processor system 1110 may be arranged for contact and / or contactless communication using antennas and / or connectors, respectively.

[0121] Although system 1140 is shown as including one of each described component, in various embodiments there may be multiple components. For example, processing unit 1120 may include multiple microprocessors configured to independently execute the methods described herein, or configured to execute steps or subroutines of the methods described herein, such that the multiple processors cooperate to achieve the functionality described herein. Furthermore, in the case of implementing system 1140 in a cloud computing system, the various hardware components may belong to different physical systems. For example, processing unit 1120 may include a first processor in a first server and a second processor in a second server.

[0122] It should be noted that the above embodiments are illustrative and not limiting of the subject matter disclosed herein, and those skilled in the art will be able to devise many alternative embodiments.

[0123] In the claims, any reference numerals placed in parentheses shall not constitute a limitation on the claims. In the claims, the verb "comprising" and its variations do not exclude the presence of elements or steps other than those stated in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. Expressions such as "at least one" preceding a list of elements indicate a selection from all or any subset of the elements in the list. For example, the expression "at least one of A, B, and C" should be understood to include only A, only B, only C, both A and B, both A and C, both B and C, or all of A, B, and C. The subject matter currently disclosed can be implemented by hardware comprising several discrete elements and by means of a suitably programmed computer. In device-type claims that enumerate several parts, several of these parts can be implemented by the same item of hardware. Although specific measures are recited in mutually distinct dependent claims, this does not indicate that combinations of these measures cannot be advantageously used.

[0124] In the claims, the reference numerals enclosed in parentheses refer to reference numerals in the drawings of exemplary embodiments or formulas of embodiments, thus increasing the comprehensibility of the claims. These reference numerals should not be construed as limiting the claims.

[0125] The computer system also includes main memory and static memory, wherein the memories in the computer system communicate with each other and with the processor via a bus. Either or both of the main memory and / or static memory can be considered representative examples of controller memory and store instructions for implementing some or all aspects of the methods and processes described herein. The memories described herein are tangible storage media for storing data and executable software instructions, and are non-transient during the time the software instructions are stored. Main memory and static memory are articles of manufacture and / or machine parts. Main memory and static memory are computer-readable media from which a computer (or, for example, a processor) can read data and executable software instructions. Both main memory and static memory can be implemented in one or more ways: random access memory (RAM), read-only memory (ROM), flash memory, electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, magnetic tape, optical disc read-only memory (CD-ROM), digital versatile optical disc (DVD), floppy disks, Blu-ray discs, or any other form of storage medium known in the art. The memory can be volatile or non-volatile, secure and / or encrypted, insecure and / or unencrypted.

[0126] “Memory” is an example of a computer-readable storage medium. Computer memory is any memory that a processor can directly access. Examples of computer memory include RAM, registers, and register files. The reference to “computer memory” or “memory” should be interpreted as potentially including multiple memories. Memory can be, for example, multiple memories within the same computer system. Memory can also be multiple memories distributed across multiple computer systems or computing devices. Memory can store various software applications, including computer-executable instructions that, when run on a processor, implement the methods and systems defined herein. Other forms of memory, such as storage devices and mass storage devices, may also be included and accessed via a bus by the processor (or multiple processors). Storage devices and mass storage devices can each be included in any or all of the methods and systems discussed herein.

[0127] The computer system may also include a communication interface through which it can connect to a network and receive data for performing the methods and systems described herein, as well as transmit information to other devices. The computer system also includes a video display unit as an output device through which information can be output, such as a liquid crystal display (LCD), an organic light-emitting diode (OLED), a flat panel display, a solid-state display, or a cathode ray tube (CRT). Additionally, the computer system includes input devices, such as a keyboard / virtual keyboard or a touch input screen or voice input with voice recognition, and cursor control devices, such as a mouse or a touch input screen or touchpad. The computer system may also optionally include a disk drive unit, signal generating devices (e.g., speakers or remote controls), and / or network interface devices.

[0128] The processes and demonstrations presented herein are not inherently related to any particular computer or other device. Various general-purpose systems may also be used in conjunction with the procedures taught herein, or it may prove convenient to construct more specialized devices to perform one or more method steps. Structures for various such systems are discussed in the following description. Furthermore, any programming language sufficient to implement the techniques and implementations of this disclosure may be used. Moreover, the language used in this specification has been chosen primarily for readability and instructional purposes and may not have been chosen to depict or limit the subject matter of this disclosure. Therefore, this disclosure is intended to illustrate, and not limit, the scope of the concepts discussed herein.

[0129] According to various embodiments of this disclosure, the methods described herein can be implemented using a hardware computer system that executes software programs. Furthermore, in exemplary non-limiting embodiments, implementations may include distributed processing, component / object distributed processing, and parallel processing. Virtual computer system processing can implement one or more methods or functions as described herein, and the processors described herein can be used to support virtual processing environments.

[0130] The illustrations of the embodiments described herein are intended to provide a general understanding of the structure of various embodiments. These illustrations are not intended to fully depict all elements and features of the disclosure described herein. Many other embodiments will likely be apparent to those skilled in the art after reviewing this disclosure. Other embodiments can be utilized and derived from this disclosure, allowing structural and logical substitutions and changes to be made without departing from the scope of this disclosure. Furthermore, these illustrations are representative only and may not be drawn to scale. Some scales in the illustrations may be enlarged, while others may be minimized. Therefore, this disclosure and the accompanying drawings should be considered illustrative rather than restrictive.

[0131] While specific embodiments have been illustrated and described herein, it should be understood that any subsequent arrangements designed to achieve the same or similar purposes may replace the specific embodiments shown. This disclosure is intended to cover all subsequent adaptations or variations of the various embodiments. Combinations of the above embodiments, as well as other embodiments not specifically described herein, will be apparent to those skilled in the art.

[0132] The foregoing description of the disclosed embodiments is provided to enable any person skilled in the art to practice the concepts described in this disclosure. Therefore, the subject matter disclosed above should be considered illustrative rather than restrictive, and the appended claims are intended to cover all such modifications, enhancements, and other embodiments falling within the scope of this disclosure. Accordingly, to the fullest extent permitted by law, the scope of this disclosure will be determined by the broadest permissible interpretation of the following claims and their equivalents, and should not be limited to or restricted by the foregoing detailed description.

Claims

1. A computer-implemented method for medical 3D visualization of medical image data, the method comprising: Retrieve the medical imaging data from a medical imaging data repository, the medical imaging data comprising multiple 2D image slices; The anatomical entities in the plurality of 2D image slices are segmented using a segmentation model; An overlay is generated based on a specific 2D image slice from the plurality of 2D image slices and segmented anatomical entities within the specific 2D image slice, and the overlay is displayed, wherein the segmented anatomical entities in the overlay are interactive to be selected or deselected. Receive selections for one or more anatomical entities in the displayed overlay, the selections being derived from user interaction with the displayed segmented anatomical entities; The 3D visualization is generated based on the plurality of 2D image slices, the 3D visualization visually distinguishing the selected one or more anatomical entities; and The 3D visualization is displayed.

2. The method according to claim 1, in, The user interaction for receiving the selection includes: receiving a user input signal within a displayed overlay from a pointing device, such as a mouse, cursor, or indicator; and / or The user interaction for receiving the selected interaction includes receiving user eye-tracking signals from a display overlay obtained from a camera.

3. The method according to any one of the preceding claims, wherein, In the overlay, the segmented anatomical entities are visually distinguished by one or more of the following methods: color coding, texture mapping, opacity, outline or border style, brightness or contrast, and highlighting.

4. The method according to any one of the preceding claims, wherein, In the 3D visualization, the selected anatomical entities are visually distinguished in the following ways: The 3D visualization includes selected anatomical entities and / or excludes deselected anatomical entities; or One or more of the following: color encoding, texture mapping, opacity, outline or border style, brightness or contrast, saturation and / or desaturation, and highlighting.

5. The method according to any one of the preceding claims, comprising: Pre-compute 3D objects for anatomical entities in the plurality of 2D image slices, wherein generating the 3D visualization includes: including the pre-computed 3D objects corresponding to the selected anatomical entities in the 3D visualization, and / or visually distinguishing the 3D objects.

6. The method according to any one of the preceding claims, comprising: A preselection is generated based on the segmented anatomical entities in the plurality of 2D image slices, and user interaction is enabled to modify the preselection to obtain the selection.

7. The method of claim 6, comprising: Identify anatomical abnormalities in the anatomical entity, including lesions, abnormal arterial branches, tumors, and skeletal abnormalities, wherein the preselection includes the identified anatomical abnormalities.

8. The method according to any one of the preceding claims, wherein, The method further includes the following: One or more selected anatomical entities in the displayed 3D visualization are interactive. Receive selections for one or more anatomical entities in the displayed 3D visualization, and modify the selections of the anatomical entities obtained based on the user interaction accordingly; The modified selection is displayed in the 3D visualization and / or 2D overlay.

9. The method according to any one of the preceding claims, wherein, Generating the 3D visualization includes selecting a camera orientation relative to a 3D model of the one or more anatomical entities, the camera orientation including camera position and / or camera direction, and the method further includes: The camera orientation is adjusted in response to user input for the 3D visualization, thereby allowing the selected anatomical entity to be viewed; and / or In response to user input, a particular 2D image slice in the plurality of 2D image slices in the overlay is replaced with another 2D image slice in the plurality of 2D image slices.

10. The method according to any one of the preceding claims, wherein, The 3D visualization is displayed on a separate projection device, such as a holographic display and / or an augmented reality (AR) head-mounted device.

11. The method according to any one of the preceding claims, wherein, Generating the 3D visualization includes selecting a 3D camera relative to the selected anatomical entity, the method comprising: Provide control instructions for moving the 3D camera in response to a modification of the selection obtained based on the user interaction, thereby centering the selected anatomical entity in 3D.

12. The method according to any one of the preceding claims, wherein, In 3D visualization, the position of a specific 2D image slice among the plurality of 2D image slices is visible.

13. The method according to any one of the preceding claims, wherein, The 3D visualization includes a selected anatomical entity with reduced opacity, and the 3D visualization includes an animation of rotation around the selected anatomical entity.

14. A system comprising: One or more processors; And one or more storage devices for storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the operations described in any one of claims 1-13.

15. A transient or non-transient computer-readable medium comprising data representing instructions that, when executed by a processor system, cause the processor system to perform one or more steps of the method according to any one of claims 1-13.

Images