Method for displaying a 3D model of a patient

Abstract

The present invention relates to a computer-implemented method for displaying a 3D model of a patient's body in a 3D scene, where each voxel of the displayed 3D model is associated with a respective pixel of at least one corresponding cross-sectional image of the patient's body, the 3D scene further comprising a cursor and at least one viewing window, the method comprising the steps of: calculating a current position of the cursor in a scene coordinate system of the 3D scene based on a current position of a user point in the user coordinate system; and, for each active viewing window of the at least one viewing window, displaying a display image based on a cross-sectional image that includes a pixel associated with the voxel that is closest to the calculated current position of the cursor and has a corresponding image cross-sectional plane that coincides with the window cross-sectional plane of the active viewing window.

Description

[Technical field]

[0001] The present invention relates to a computer-implemented method for displaying a 3D model including a three-dimensional representation of at least a portion of a patient's body in a three-dimensional scene.

[0002] The present invention also relates to visualization devices and methods of receiving a patient for diagnosis, treatment and / or surgery.

[0003] The present invention relates to the field of medical imaging. [Background technology]

[0004] It is commonly known to use cross-sectional images of a patient, such as CT scan ("computed tomography scan") images, PET ("positron emission tomography") images or MRI ("magnetic resonance imaging") images, to identify lesions (e.g. tumors). Such cross-sectional images may further be used to locate lesions as a preliminary step to surgery to remove them.

[0005] However, such an approach is not entirely satisfactory.

[0006] Indeed, in complex cases with multiple lesions, the use of general radiological two-dimensional visualization based on the above-mentioned cross-sectional images does not allow many surgeons to fully understand the actual volume and exact localization of the lesion, which may lead to excessive tissue removal when performing resection of said lesion, which is unacceptable from the point of view of the patient's physical integrity.

[0007] It is an object of the present invention to provide a method for visualizing at least a portion of a patient's body that provides better readability to a medical provider, thereby improving the ability to precisely locate lesions and minimizing unnecessary tissue removal. Summary of the Invention

[0008] To this end, the present invention relates to a method of the above-mentioned kind, comprising the steps of: each voxel of the displayed 3D model is associated with a respective pixel of at least one corresponding cross-sectional image of at least a portion of the patient's body; Each cross-sectional image is associated with a corresponding image cross-sectional plane. The 3D scene has a corresponding scene coordinate system associated with it; the three-dimensional scene further includes a cursor and at least one viewing window; Each viewing window in each viewing plane has a given equation in the scene coordinate system, Each viewing window is associated with a corresponding window cross-sectional plane; The method comprises: - calculating a current position of the cursor in the scene coordinate system based on the current position of a given user point in a given user coordinate system; for each active viewing window of at least one viewing window, contains pixels associated with an active voxel, which is the voxel having the closest position to the calculated current position of the cursor, and Has a corresponding image cross-sectional plane that matches the window cross-sectional plane of the active viewing window displaying a display image in an active viewing window based on the cross-sectional image; - upon receiving an object display command, displaying the 3D object at the current position of the cursor in the three-dimensional scene; The method includes:

[0009] Indeed, such a method allows interactive navigation within the 3D model and provides in real time a cross-sectional image related to the currently selected voxel, said cross-sectional image providing guidance to the user as to where the selected voxel of the 3D model is actually located within the cross-sectional image that the user is typically familiar with. Thus, a complete volumetric understanding of the lesion location within the patient's body and its 3D geometry in relation to the surrounding tissues can be achieved by the user.

[0010] Furthermore, the present invention allows the user to simultaneously include within his field of view both a global three-dimensional representation of the patient's anatomy and at least one local standard two-dimensional image associated with the desired voxel, thus allowing the surgeon to visualize the standard cross-sectional images with which he is accustomed to work in order to plan the operation, while at the same time referencing a three-dimensional model that more closely resembles the real scene he will encounter during the operation.

[0011] Furthermore, the present invention provides the user with the opportunity to accurately place 3D objects (such as landmarks or predefined volume objects) within the three-dimensional scene, which allows for a better assessment of the geometry and / or position of the patient's internal anatomical structures, due to the fact that the present invention allows the user to accurately determine the position of the area near the cursor in real time.

[0012] The method according to the invention also provides a tool that allows efficient collaboration between medical professionals with different specializations (i.e. between surgeons and radiologists). Indeed, on the one hand, the volumetric visualization of the 3D model in a three-dimensional scene provides a visual result that is close to the one that the surgeon has in the operating room during the operation. On the other hand, the cross-sectional images displayed interactively and synchronously in the background are standard radiological images (i.e. CT scans or MRI slice images) that are mainly used by medical professionals and surgeons for diagnosis and planning of the treatment. This allows a better mutual understanding between said medical professionals, which is advantageous for the patient undergoing the operation.

[0013] Furthermore, the generation of the 3D model does not require predefining tissue boundaries, thereby reducing computational complexity and time.

[0014] Furthermore, when dealing with lesions with poorly defined borders, as is often the case in breast MRI, the method according to the invention allows an unbiased analysis of the border regions.

[0015] Thus, when the claimed method is used for surgical planning, it may allow for more preservation of tissue during surgical resection of malignant lesions, thereby increasing the probability of performing oncoplastic surgery with less disfigurement and without compromising patient safety.

[0016] According to other advantageous aspects of the invention, the method comprises one or more of the following features, taken alone or in any technically possible combination:

[0017] The method further comprises the step of, for each displayed 3D object and for each active viewing window, displaying a corresponding 2D object in the active viewing window, wherein each second point of the 2D object corresponds to a first point of the 3D object, and wherein at least one second point of the 2D object is located in a cross-sectional image corresponding to the display image displayed in the active viewing window at a pixel position associated with a voxel that is closest to the first point of the 3D object that corresponds to said second point of the 2D object.

[0018] The method includes receiving an object movement command including a selected 2D object, an initial position, and a final position, - moving the position of the selected 2D object in the active viewing window based on the initial position and the final position to obtain a registered 2D object; - updating, based on the registered 2D object position, the position of the corresponding 3D object such that for at least one first point of the 3D object, the voxel closest to said first point is the voxel associated with the pixel on the cross-sectional image corresponding to the display image on the active viewing window that is closest to a second point on the 2D object corresponding to said first point; Further includes:

[0019] The method includes, for a given pair of consecutive 3D objects, where each pair includes a first 3D object and a second 3D object associated with a first reception of a first display command and a second reception of a second display command, respectively, and where the first reception and the second reception are consecutive, displaying in the three-dimensional scene a segment connecting the two 3D objects of the consecutive 3D objects of the pair, and the method preferably includes displaying a cumulative length of the segment.

[0020] The object display instructions include an initial and final position of the cursor in the scene coordinate system, and a shape, size and / or orientation of the 3D object depending on the initial and final position of the cursor.

[0021] The method further includes displaying data representative of the geometry of the at least one 3D object.

[0022] The method further includes, upon receiving the transformation command, applying a predetermined transformation to each voxel of the 3D model located within a predetermined range relative to a current position of the cursor in the three-dimensional scene.

[0023] The transformation is - reducing the opacity of each voxel of the 3D model located at a distance greater than a first predetermined radius from the current position of the cursor in the three-dimensional scene; and / or modifying the color value of each voxel of the 3D model located at a distance less than a second predetermined radius from the current position of the cursor in the 3D scene to form an enhanced voxel; Includes.

[0024] The method further includes, for each enhanced voxel, for each active viewing window, and for each display image corresponding to a cross-sectional image that includes a pixel associated with the enhanced voxel, modifying a color of the pixel associated with the enhanced voxel to form a enhanced pixel.

[0025] The method further includes displaying, for each active viewing window, a line extending between the active voxel and a corresponding pixel of the cross-sectional image displayed in the active viewing window.

[0026] The method comprises: - determining a current observation vector representing the direction in which a user is looking at the 3D model in a three-dimensional scene; - at least one active display window, · The determined current observation vector and The normal vector of each viewing plane oriented relative to the 3D model, determining a viewing window corresponding to a positive value of the scalar product of Further includes:

[0027] Each cross-sectional image is associated with an image orientation, each viewing window is associated with a window orientation, and a viewing image corresponding to each cross-sectional image is - a cross-sectional image if the orientation of that image is the same as the orientation of the window, or - A mirror image of the cross-sectional image if the image orientation is different from the window orientation It is.

[0028] The invention further relates to a computer program product comprising instructions which, when executed by a computer, cause the computer to carry out the method defined above.

[0029] The invention also relates to a non-transitory computer-readable storage medium having stored thereon computer-executable instructions which, when executed by a computer, cause the computer to perform the steps of the method defined above.

[0030] The present invention relates to a visualization device comprising a user interface, a display device and a processing device, The user interface is configured to obtain a current position of a given user point in a given user coordinate system; the display device is configured to display a 3D model in the three dimensional scene, the 3D model including a three dimensional representation of at least a portion of the patient's body, each voxel of the 3D model being associated with a respective pixel of at least one corresponding cross sectional image of the at least a portion of the patient's body, each cross sectional image being associated with a corresponding image cross sectional plane, the three dimensional scene having a corresponding scene coordinate system associated therewith, the three dimensional scene further including a cursor and at least one viewing window, each viewing window in a respective viewing plane having a predetermined equation in the scene coordinate system, each viewing window being associated with a corresponding window cross sectional plane; The processing device includes: - Calculate the current position of the cursor in the scene coordinate system based on the current position of the user point in the user coordinate system; for each active viewing window of at least one viewing window, contains pixels associated with an active voxel, which is the voxel having the closest position to the calculated current position of the cursor, and Has a corresponding image cross-sectional plane that matches the window cross-sectional plane of the active viewing window Controlling the display device to display a display image in an active viewing window based on the cross-sectional image; and upon receiving an object display command from the user interface, controlling the display device to display the 3D object at the current position of the cursor in the three-dimensional scene; The present invention also relates to a visualization device configured as described above.

[0031] According to other advantageous aspects of the invention, the visualization device comprises one or more of the following characteristics, taken alone or in any technically possible combination:

[0032] The display device comprises at least a portion of a virtual reality headset, and the user interface preferably comprises a handheld motion tracking sensor.

[0033] The at least one image cross-sectional plane is one of a sagittal plane, a coronal plane, or a transverse plane.

[0034] The present invention relates to a method of accepting a patient for diagnosis and / or treatment and / or surgery, comprising the steps of: - generating a 3D model of at least a part of the patient's body, each voxel of the 3D model being associated with a respective pixel of at least one corresponding cross-sectional image of the at least a part of the patient's body, each cross-sectional image being associated with a corresponding image cross-sectional plane; - displaying the 3D model to a user in a three-dimensional scene having a corresponding scene coordinate system associated therewith, the three-dimensional scene further comprising a cursor and at least one viewing window, each viewing window in a respective viewing plane having a predefined equation in the scene coordinate system, each viewing window being associated with a corresponding window cross-sectional plane; - calculating the current position of the cursor in the scene coordinate system based on the current position of a predefined user point associated with the user in a predefined user coordinate system; for each active viewing window of at least one viewing window, contains pixels associated with an active voxel, which is the voxel having the closest position to the calculated current position of the cursor, and Has a corresponding image cross-sectional plane that matches the window cross-sectional plane of the active viewing window displaying a display image in an active viewing window based on the cross-sectional image; - upon receiving an object display command, displaying in the three-dimensional scene the 3D object located at the current position of the cursor; providing diagnostic and / or therapeutic and / or surgical strategy recommendations based on at least the results of navigating the 3D model in the three-dimensional scene by the user and / or the 3D object; The present invention also relates to a method comprising the steps of:

[0035] According to another advantageous aspect of the invention, the user is a medical professional, preferably a surgeon or a radiologist.

[0036] The invention will be better understood with reference to the accompanying drawings. [Brief description of the drawings]

[0037] [Figure 1] 1 is a schematic diagram of a visualization device according to the present invention; [Diagram 2] FIG. 2 is a schematic diagram of a three-dimensional scene displayed by the visualization device of FIG. 1; [Diagram 3] 3 is a schematic illustration of the three-dimensional scene of FIG. 2, with a viewing direction different from that of FIG. 2; [Figure 4] Similar to FIG. 2, but with 3D objects also displayed in the 3D scene. [Diagram 5] FIG. 2 is a schematic diagram of a processing unit of the visualization device of FIG. 1. [Figure 6] 1 is a flow chart of a method for admitting a patient according to the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0038] This specification illustrates the principles of the disclosure, and it will thus be appreciated that those skilled in the art may devise various arrangements which, although not explicitly described and shown in this specification, embody the principles of the disclosure and are included within its scope.

[0039] All examples and conditional language set forth herein are intended for educational purposes to aid the reader in understanding the principles of the present disclosure and concepts contributed by the inventors to advance the art, and should not be construed as being limited to such specifically set forth examples and conditions.

[0040] Moreover, all statements herein reciting principles, aspects, and embodiments of the disclosure, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, such equivalents are intended to include both currently known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.

[0041] Thus, for example, it will be appreciated by those skilled in the art that the block diagrams provided herein may represent conceptual views of illustrative circuitry embodying the principles of the present disclosure. Similarly, it will be appreciated that any flow charts, flowcharts, and the like represent various processes that may be substantially embodied in a computer-readable medium and thus executed by a computer or processor, whether or not such a computer or processor is explicitly shown.

[0042] The functions of the various elements shown in the figures may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared.

[0043] It will be appreciated that the elements shown in the figures may be implemented in various forms of hardware, software, or a combination thereof. Preferably, these elements are implemented in a combination of hardware and software on one or more appropriately programmed general-purpose devices, which may include a processor, memory, and input / output interfaces.

[0044] A visualization device 2 according to the invention is shown in FIG.

[0045] The structure and operation of the visualization device 2 will be described with reference to FIGS.

[0046] The visualization device 2 is configured to display to a user 4, such as a medical professional (e.g., a surgeon or radiologist), a 3D model 5 forming a three-dimensional representation of at least a portion of a patient's body. The visualization device 2 is further configured to provide the user with the ability to interact with the 3D model 5.

[0047] "3D model" means a 3D volume rendering of a 3D medical image (or in other words a voxel-based three-dimensional representation of a medical image). A medical image consists of voxels. The 3D volume rendering can be achieved by various techniques: surface segmentation and rendering, ray casting, ray tracing, path tracing, etc. The 3D model is rendered in a 3D scene using a coordinate system. Thus, "each voxel of the displayed 3D model" should be understood as "each voxel of the three-dimensional scene in which the 3D model is being rendered".

[0048] To generate a correspondence between the 3D model and the original data, voxels of the original 3D medical image are related to voxels of the three-dimensional scene. In one embodiment, the 3D model is obtained by surface rendering of at least one object file corresponding to a particular anatomical structure. The object file is registered to the original medical image data in the scene. In another embodiment, techniques involving modeling of light diffusion (ray casting, ray tracing, path tracing, etc.) are used for volume rendering that shows a direct relationship between the rendering in the three-dimensional scene and the representation of the medical image data.

[0049] The at least part of the patient may for example include at least a part of one or several organs of the patient.

[0050] While the visualization device 2 described herein is versatile and provides several functions that can be implemented in alternative or any cumulative manner, other implementations within the scope of this disclosure include devices having only a portion of this functionality.

[0051] The visualization device 2 is advantageously an apparatus or physical part of an apparatus designed, configured and / or adapted to perform the mentioned functions and bring about the mentioned effects or results. In other implementations, the visualization device 2 is embodied as a set of apparatus or physical parts of an apparatus, whether grouped on the same machine or on different, possibly remote machines. The visualization device 2 may have functionality distributed, for example in a cloud infrastructure and available to users as a cloud-based service, or may have remote functionality accessible via an API ("Application Programming Interface").

[0052] The visualization device 2 comprises a user interface 6 for interacting with the 3D model 5, a display device 8 for displaying the 3D model 5, and a processing device 10 for processing data received from the user interface 6 and controlling the display device 8.

[0053] User Interface 6 The user interface 6 is arranged to obtain the current position of a given user point A in a given user coordinate system 12. The given user coordinate system 12 is fixed with respect to the user's real environment.

[0054] The user coordinate system 12 is, for example, fixed relative to the user's environment.

[0055] The user interface 6 may comprise a handheld motion tracking sensor, allowing the user 4 to interact with the 3D model 5 (and / or the three-dimensional scene 14 in which the 3D model 5 is represented) using hand gestures. In this case, the user point A may be a point of the motion tracking sensor.

[0056] User interface 6 may also include other devices, such as a touch screen, a game controller, a mouse and / or a keyboard, which further enable user 4 to interact with 3D model 5 (and / or the three-dimensional scene in which 3D model 5 is displayed).

[0057] User interface 6 may further comprise buttons and / or switches configured to enable user 4 to interact with 3D model 5 (and / or the three-dimensional scene in which 3D model 5 is displayed). Such buttons and / or switches may be included in the handheld motion tracking sensor or may be included in a separate device of user interface 6, such as the touch screen, game controller, mouse and / or keyboard mentioned above.

[0058] Preferably, the user interface 6 is configured to enable the user 4 to input (e.g. via said buttons and / or switches) specific commands, such as object display commands, object movement commands or transformation commands. Such commands advantageously enable the user 4 to add and / or manipulate 3D objects in the same three-dimensional scene as the 3D model 5, as will be explained below.

[0059] The user interface 6 may also be configured to allow the user to manipulate the 3D model 5 (e.g., rotate and / or zoom in or out) and / or change the direction in which the user 4 views the 3D model 5 displayed by the display device 8.

[0060] The user interface 6 may also be configured to enable a user to manipulate (e.g. move and / or rotate) the three-dimensional scene, and more specifically, 2D objects displayed in a viewing window of the three-dimensional scene (described below).

[0061] Alternatively or additionally, the user interface 6 comprises virtual buttons displayed in the three-dimensional scene to enable the user 4 to interact with the 3D model 5, the three-dimensional scene, the 3D objects and / or the 2D objects.

[0062] Display device 8 As mentioned above, the display device 8 is configured to display the 3D model 5 .

[0063] Advantageously, the display device 8 is at least part of a virtual reality headset that allows a stereoscopic visualization of the 3D model 5. This is particularly advantageous in the field of medicine, since the virtual reality visualization of the 3D model 5 allows the user 4 to have a good understanding of the actual volume and an accurate localization of potential lesions. In the case of oncology, for example, this increases the probability of performing oncoplastic surgery with less disfigurement and without compromising the safety of the patient.

[0064] The virtual reality headset may include one or more sensors, such as accelerometers and / or gyroscopes, where each sensor is part of the user interface 6 and is configured to output a signal representative of the direction in which the user 4 is looking at the 3D model 5 in a three-dimensional scene 14, described below.

[0065] Alternatively or additionally, the display device 8 comprises a screen 9, as shown in FIG.

[0066] More specifically, the 3D model 5 forms a three-dimensional representation of at least a portion of a patient's body in said three-dimensional scene 14. This three-dimensional scene (hereinafter "3D scene") has a scene coordinate system 16 associated therewith.

[0067] Furthermore, each voxel of the 3D model 5 is associated with a respective pixel of at least one corresponding cross-sectional image of at least a portion of the patient's body.

[0068] Each cross-sectional image is associated with a corresponding image cross-sectional plane and preferably illustrates a slice 17 of the portion in the corresponding image cross-sectional plane.

[0069] Such image cross-sectional plane may be one of a sagittal, coronal or transverse plane, more particularly a sagittal, coronal or transverse plane of the patient.

[0070] By way of example, each cross-sectional image may have been previously acquired using magnetic resonance imaging (MRI), such as T1-weighted contrast MRI, CT scan imaging, or PET imaging, or any acquisition technique that may be used for pre-operative preparation and planning.

[0071] Additionally, each cross-sectional image is preferably stored in a database 18 at a medical facility, for example a hospital's medical image management system.

[0072] Preferably, each cross-sectional image is stored in database 18 in standard DICOM ("Digital Imaging and Communications in Medicine") file format. This is advantageous because a DICOM file is associated with a scale at which the imaged anatomical structure is represented. Thus, if the scale of 3D scene 14 matches the scale of the DICOM file, accurate measurements of geometric features (distances, surfaces and / or volumes) on 3D model 5 are available.

[0073] Advantageously, each cross-sectional image is further associated with an image orientation.

[0074] By "image orientation of a cross-sectional image" is meant in the context of the present invention the direction in which the patient is viewed in said cross-sectional image.

[0075] For example, for a cross section along the sagittal plane of the patient, the image orientation can be "left lateral view" or "right lateral view."

[0076] As another example, for a cross-section along the coronal plane of a patient, the image orientation can be "front view" or "back view."

[0077] As a further example, in the case of a cross-section along the transverse plane of a patient, the image orientation may be "top view" or "bottom view."

[0078] Additionally, the 3D scene 14 displayed by the display device 8 further includes a cursor 20 and at least one viewing window 22. For example, the 3D scene 14 includes six viewing windows 22.

[0079] Each viewing window 22 is in a respective viewing plane, which has a predefined equation in the scene coordinate system 16. For example, at least two viewing planes are orthogonal, thereby mimicking the orthogonality between cross-sectional planes (sagittal, coronal and transverse) commonly used in medical imaging, thus improving readability of the 3D scene 14. As visible in Figures 2 and 3, the viewing planes surround the 3D model 5, thereby allowing the 3D model 5 and each viewing window 22 to be visualized simultaneously in three dimensions.

[0080] More specifically, each viewing plane is preferably orthogonal to the four other viewing planes, such that there are six viewing windows 22, one on each face of the parallelepiped.

[0081] Further, each viewing window 22 is associated with a corresponding window cross-sectional plane. Such window cross-sectional plane may be one of a sagittal plane, a coronal plane, or a transverse plane.

[0082] Furthermore, at any given time, there may be zero, one or more active viewing windows 22 out of the total number of viewing windows 22, preferably three active viewing windows 22.

[0083] Each active viewing window is defined as a viewing window that displays an image, and such image is hereinafter referred to as a “displayed image.” Conversely, viewing windows 22 that are not the active viewing window at a given time cannot display an image, and therefore cannot be visible in the 3D scene 14.

[0084] The determination of the active viewing window is described below.

[0085] Advantageously, each viewing window 22 is associated with a respective window orientation. Furthermore, two viewing windows 22 in parallel viewing planes are associated with opposing window orientations such as "left side view" and "right side view", "front view" and "rear view", or "top view" and "bottom view". The advantages of such features are discussed below.

[0086] As shown in FIG. 4, the display device 8 is further configured to display at least one 3D object 24 in the 3D scene 14 .

[0087] Each 3D object may include any number of points. For example, the 3D object is a landmark that includes at least one point. Or the 3D object is a segment that includes at least two points, each corresponding to one end of the segment. As another example, the 3D object is a surface that includes at least three non-coplanar points. Or the 3D object has a three-dimensional shape and includes at least four non-coplanar points.

[0088] The 3D object may also be a 2D or 3D segmentation mask (such as a mesh) for identifying and / or highlighting anatomical structures of a given nature. In this case, the visualization device 2 may be provided with a library in which segmentation masks are stored. The 3D object may also represent a medical device. In the latter case, the 3D object may be generated from a 3D object file (e.g. provided in .stl or .obj file format) representing said physical medical device.

[0089] Processing device 10 According to the present invention, the term "processing device" should not be construed as being limited to hardware capable of executing software, but generally refers to a processing device, which may be, for example, a microprocessor, an integrated circuit, or a programmable logic device (PLD). The processing device may also encompass one or more graphics processing units (GPUs) or tensor processing units (TSUs), whether utilized for computer graphics and image processing or other functions. Furthermore, instructions and / or data capable of implementing the relevant functions and / or obtained functions may be stored in any processor-readable medium, such as, for example, an integrated circuit, a hard disk, an optical disk such as a CD (compact disk), a DVD (digital versatile disk), a RAM (random access memory), or a ROM (read only memory). The instructions may be stored in particular in hardware, software, firmware, or any combination thereof.

[0090] The processing unit 10 is connected to a user interface 6 and a display device 8, respectively.

[0091] The processing device 10 may, for example, be compatible with a workstation, a laptop, a tablet, a smartphone, a programmable logic device (eg, FPGA) for on-board computing, or a head mounted display (HMD), such as a virtual reality headset.

[0092] As shown in FIG. 4, the processing unit 10 comprises the following elements, connected together by an address and data bus 95, which also carries a clock signal: · A microprocessor 91 (or CPU), A graphics card 92 equipped with several Graphics Processing Units (or GPUs) 920 and Graphic Random Access Memory (GRAM) 921 (GPUs are very suitable for image processing due to their highly parallel structure), 96 non-volatile ROM type memories, RAM97, Power 98, and High Frequency Unit 99 The device may include:

[0093] Alternatively, the power supply 98 is external to the processing device 10 .

[0094] For example, a user interface 6 is connected to at least some of the above modules, for example via a bus 95 .

[0095] The display device 8 is connected to the graphics card 92, for example via a suitable interface. For example, a cable can be used for tethered transmission or an RF unit 99 can be used for wireless transmission.

[0096] Each of the memories 97 and 921 comprises registers, which in each of said memories can specify low-capacity (some binary data) memory areas as well as large-capacity memory areas (allowing to store an entire program or to display all or part of data representing calculated data). Also, the registers depicted for the RAM 97 and the GRAM 921 can be arranged and organized in any manner. They do not necessarily correspond to adjacent memory locations, but can be otherwise distributed (this particularly includes the situation where one register contains several smaller registers).

[0097] When switched on, the microprocessor 91 loads and executes instructions of a program 970 contained in the RAM 97 to enable operation of the visualization device 2 as described in this disclosure.

[0098] As will be appreciated by those skilled in the art, the presence of a graphics card 92 is not required and may be replaced by the entire CPU processing and / or other implementations.

[0099] The processing unit 10 is configured to calculate the current position of a cursor 20 in the scene coordinate system 16 based on the current position of a user point A in the user coordinate system 12 .

[0100] Further, for each active viewing window 22, the processing unit 10 is configured to control the display device 8 to display a corresponding display image in said active viewing window. Such a display image may be Contains pixel 19 associated with the current active voxel, and having a corresponding first cross-sectional plane that coincides with the window cross-sectional plane of the active viewing window; It is based on cross-sectional images.

[0101] Such an active voxel is defined as the voxel having a position in the 3D scene 14 closest to the calculated current position of the cursor 20 .

[0102] Furthermore, in the context of the present invention, the expression "the first cross-sectional plane coincides with the window cross-sectional plane of the active viewing window" means that the first cross-sectional plane and the window cross-sectional plane are the same.

[0103] The processing device 10 is further configured to detect an object display command input by the user 4 via the user interface 6. Upon receiving the object display command, the processing device 10 is configured to control the display device 8 to display in the 3D scene 14 the 3D object at the current position of the cursor 20.

[0104] Advantageously, the object display instructions further comprise an initial and final position of a cursor 20 in the scene coordinate system 16. In this case, the processing device 10 is configured to determine the shape, size and / or orientation of the 3D object 24 depending on the initial and final position of the cursor 20.

[0105] The initial and final positions in the scene coordinate system 16 may be determined by the processing unit 10 based on the motion of the user point A in the user coordinate system 12 .

[0106] This is advantageous, for example, if the 3D object is one of several 3D segmentation masks stored in a library, each mask corresponding to an organ or part of an organ, or if the 3D object represents a medical device to be implanted in a patient. In this case, the user 4 can adjust the features of the mask (such as its shape, size and / or orientation) or the model of the medical device via the user interface 6.

[0107] More specifically, the processing device 10 is configured to control the display device 8 to display the 3D object such that the position of a predefined point of the 3D object coincides with the current position of the cursor 20. Such predefined point may be the centroid of the 3D object, an edge of the 3D object, a point on an outer surface of the 3D object, etc.

[0108] Optionally, the processing device 10 is further configured to determine for each displayed 3D object 24 at least one corresponding 2D object 26. In this case, the processing device 10 is configured to control the display device 8 to display each determined 2D object 26 in a corresponding active viewing window.

[0109] Preferably, for a given 3D object 24 and for a given active viewing window, the 2D objects correspond to slices of the 3D object in a corresponding image cross-sectional plane. In other words, for a given 3D object 24 and for a given active viewing window 22, each 2D object 26 corresponds to an intersection of the 3D object 24 with a plane defined by voxels associated with pixels of the cross-sectional image on which the display image currently displayed in said active viewing window 22 is based.

[0110] Preferably, each second point of the 2D object corresponds to a first point of the 3D object. In this case, for each displayed 3D object 24 and for each active viewing window 22, the processing unit 10 is configured to control the display device 8 such that at least one second point of the 2D object 26 is located in the cross-sectional image corresponding to the display image currently displayed in said active viewing window at the position of a pixel associated with a voxel that is closest to the first point of the 3D object 24 that corresponds to said second point of the 2D object 26.

[0111] Advantageously, the processing unit 10 is further configured to control the display device 8 to display, for each active viewing window 20, a line extending between the active voxel and a corresponding pixel 19 of the cross-sectional image displayed in said active viewing window 22. The user 4 can thus easily locate the active voxel in the 3D model 5 using the position of each corresponding pixel 19 in the image currently displayed in the viewing window.

[0112] Preferably, the processing device 10 is configured to detect an object movement command input by the user 4 via the user interface 6. Such an object movement command includes a selected 2D object, an initial position and a final position. The initial and final positions may be, for example, the initial and final positions of a user point A in the user coordinate system 12, or the initial and final positions of a cursor 20 in the scene coordinate system 16.

[0113] Upon receiving an object movement command, the processing device 10 is configured to move the position of the selected 2D object in the corresponding active viewing window based on the initial position and the final position to obtain a registered 2D object.

[0114] Furthermore, the processing device 10 is configured to update the position of the 3D object 24 in the 3D scene 14 based on the position of the registered 2D object 26. More specifically, the processing device 10 is configured to update the position of the 3D object 24 such that, for at least one first point of the 3D object 24, the closest voxel to said first point is the voxel associated with a pixel on the cross-sectional image corresponding to the display image currently displayed in the active viewing window that is closest to a second point in the 2D object corresponding to said first point.

[0115] Thus, by moving each 2D object in its corresponding active viewing window, the user 4 can interact with and move the corresponding 3D object 24. This is particularly advantageous when the active viewing windows are displayed on the screen 9 by displaying 2D images in a 2D interface, thus providing the user 4 with a working environment more suitable for such interaction. Such a feature further enables the user 4 to navigate the 3D model 5 using stereoscopic visualization and to quickly place 3D objects in the 3D scene 14 and adjust them on the 2D interface to which the user is accustomed.

[0116] The processing device 10 is also preferably configured to detect transformation command input by the user 4 via the user interface 6 .

[0117] In this case, the processing device 10 is further configured, upon receiving the transformation command, to apply the predetermined transformation to at least a portion of the voxels of the 3D model, more particularly, the processing device 10 is further configured to apply the predetermined transformation to each voxel located within a predetermined range relative to a current position of the cursor 20 in the 3D scene 14.

[0118] For example, the processing device 10 is configured to control the display device 8 to reduce the opacity of each voxel of the 3D model 5 that is located at a distance greater than a first predetermined radius from the current position of the cursor 20 in the 3D scene 14. This is advantageous, since such a feature allows the user 4 to efficiently navigate the 3D model 5 and to focus on an area of ​​interest that is located close to the cursor 20 by reducing visual interference that may be caused by other voxels located in front of and / or behind said area of ​​interest.

[0119] As another example, the processing unit 10 is configured to control the display device 8 to modify a color value of each voxel of the 3D model 5 located in the 3D scene 14 at a distance less than a second predetermined radius from the current position of the cursor 20. As a result, each voxel having a modified color value forms an enhanced voxel.

[0120] In the latter case, the transformation instructions are preferably part of said object display instructions.

[0121] In this case, each highlighted voxel may form a landmark and define a 3D object co-located with said highlighted voxel.

[0122] Alternatively, each cluster of enhanced voxels may define a 3D object forming a surface or volume. For example, the 3D object is a mesh that connects the enhanced voxels of the cluster. As another example, the 3D object is a volume object that includes points having positions that interpolate the positions of at least some of the enhanced voxels of the cluster.

[0123] For example, a cluster of highlighted voxels is defined as highlighted voxels that are within a predefined distance from each other. As another example, a cluster of highlighted voxels is defined as voxels that are highlighted between two predefined events, such as a button press and subsequent press on the user interface 6.

[0124] Preferably, the processing unit 10 is configured to control the display device 8 to, for each highlighted voxel, for each active viewing window and for each display image corresponding to a cross-sectional image that includes a pixel associated with the highlighted voxel, modify the color of said pixel associated with the highlighted voxel to form a highlighted pixel. The user 4 can thus see the result of the applied transformation and adjust the position of the cursor 20 accordingly.

[0125] Preferably, the processing device 10 is configured to control the display device 8 to display, for at least two consecutive 3D objects, a segment connecting said two consecutive 3D objects in the 3D scene 14. This is particularly the case when each 3D object is a 3D landmark, i.e. a point.

[0126] By "two successive 3D objects" in the context of the present invention is meant two 3D objects respectively associated with the receipt of two successive display instructions, in other words two successive 3D objects which have been successively added by the user 4 in the displayed 3D scene 14.

[0127] Such displayed segments are advantageous because they enable user 4 to better understand the geometric relationships (such as relative position and / or alignment) between anatomical structures shown in the 3D model.

[0128] Advantageously, the processing device 10 is further configured to calculate data representative of the geometry of the at least one 3D object, in which case the processing device 10 is further configured to control the display device 8 to display the calculated data.

[0129] For example, the data corresponds to an accumulated distance in a 3D scene between at least two consecutive 3D objects, such as at least two consecutive 3D landmarks, or in other words, the data is equal to an accumulated length of segments connecting the at least two consecutive 3D objects.

[0130] According to another example, the calculated data includes the volume of 3D objects (such as spheres, ellipsoids and cubes) drawn by the user in the 3D scene 14. Alternatively or additionally, the calculated data includes the cumulative volume of highlighted voxels defined as the result of multiplying the volume of a voxel by the total number of highlighted voxels.

[0131] Advantageously, to determine each active viewing window 22, the processing unit 10 is configured to first determine a current observation vector. Such observation vector represents the direction in which the user 4 is looking at the 3D model 5 in the 3D scene 14 and may be obtained by the interaction of the user 4 with the 3D scene 14 via the user interface 6.

[0132] The processing device 10 further includes a display window. On the one hand, the determined current observation vector, On the other hand, the normal vector of each viewing plane, which is oriented relative to the 3D model 5, and determining that the viewing window is an active viewing window if the scalar product of

[0133] Preferably, the processing device 10 determines that the display image corresponding to each cross-sectional image is: the cross-sectional image itself, if the orientation of that image is the same as the orientation of the window, or If the image orientation is different from the window orientation, the mirror image of the cross-sectional image is used. The display device 8 is controlled so that

[0134] In the latter case, the mirror image of a given cross-sectional image of size (N;M) is an image in which each pixel (x;y) has the same value as pixel (N+1-x;y) of said original cross-sectional image, where N and M are integers greater than 0. Thus, N is the number of pixels of the cross-sectional image along its width. As a result, the displayed image appearing in the active viewing window appears to the user 4 as a slice of the 3D model as viewed from the user's perspective (i.e. "from the left", "from the right", "from the front", "from the back", "from above" or "from below"). This improves comprehension.

[0135] Thus, for a given voxel, two viewing windows in parallel viewing planes may display images that are mirror images of each other with the same active voxel, as can be seen from Figures 2 and 3. In this case, the images shown on active displays 22L (Figure 2) and 22R (Figure 3) are mirror images of each other.

[0136] More specifically, the example of Fig. 3 corresponds to the same 3D model 5 in the same 3D scene 14 as Fig. 2. The scene of Fig. 3 is obtained from the scene of Fig. 2 by a counterclockwise rotation about a vertical axis Z of about 45°.

[0137] operation Next, the operation of the visualization device 2 will be described with reference to FIG.

[0138] During an optional acquisition step 30, a patient is admitted to a medical facility and cross-sectional images of the patient are acquired. Each cross-sectional image is associated with a corresponding image cross-sectional plane.

[0139] Then, during a modeling step 32, a 3D model 5 of at least a part of the patient's body is generated, preferably based on the acquired cross-sectional images. Each voxel of the 3D model is associated with a respective pixel of at least one corresponding acquired cross-sectional image.

[0140] The visualization device 2 is then used to display the 3D model to the user 4 in the 3D scene 14 during a navigation step 34. As previously mentioned, the 3D scene 14 has a corresponding scene coordinate system 16 associated therewith. The 3D scene 14 further includes a cursor 20 and the at least one viewing window 22 described above.

[0141] The user 4 is preferably a medical professional such as a surgeon or radiologist.

[0142] During a navigation step 34, the processing unit 10 calculates the current position of the cursor 22 in the scene coordinate system 16 based on the current position of a given user point A in the given user coordinate system 12. The current position of the user point A is obtained via the user interface 6.

[0143] Preferably, during navigation step 34, processing device 10 determines a current observation vector that represents the direction in which the user is viewing the 3D model in three-dimensional scene 14. Processing device 10 then selects each active viewing window 22 based on the determined current observation vector.

[0144] Further, during the navigation step 34, the processing device 10 Contains the pixel associated with the current active voxel, and Has a corresponding image cross-sectional plane that matches the window cross-sectional plane of the active viewing window The display device 8 is controlled to display a display image in each active display window 22 based on the cross-sectional image.

[0145] As stated above, the currently active voxel is the voxel having a position closest to the calculated current position of the cursor 20 .

[0146] Further, during the navigation step 34, upon receiving an object display command, the processing device 10 controls the display device 8 to display in the three-dimensional scene the 3D object at the current position of the cursor. The shape, size and orientation of the 3D object may be selected by the user via the user interface.

[0147] Then, during an evaluation step 36, a diagnosis and / or treatment recommendation and / or surgical strategy recommendation is provided based at least on the result of the navigation of the 3D model and / or the 3D object in the 3D scene 14 by the user 4. Such a diagnosis and / or treatment recommendation and / or surgical strategy recommendation may be determined by the user 4 himself / herself.

Claims

1. 1. A computer-implemented method for displaying a 3D model (5) comprising a three-dimensional representation of at least a part of a patient's body in a three-dimensional scene (14), comprising: each voxel of the displayed 3D model (5) is associated with a respective pixel of at least one corresponding cross-sectional image of said at least part of the patient's body; Each cross-sectional image is associated with a corresponding image cross-sectional plane; said three-dimensional scene (14) having a corresponding scene coordinate system (16) associated therewith; The three-dimensional scene (14) further includes a cursor (20) and at least one viewing window (22); Each viewing window (22) in each viewing plane has a predetermined equation in said scene coordinate system (16); Each viewing window (22) is associated with a corresponding window cross-sectional plane; The method comprises: - calculating the current position of said cursor (20) in said scene coordinate system (16) based on the current position of a given user point (A) in a given user coordinate system; for each active display window (22) of said at least one display window (22), - including a pixel (19) associated with an active voxel, which is the voxel having the closest position to the calculated current position of the cursor (20); and having the corresponding image cross-sectional plane coincident with a window cross-sectional plane of the active viewing window (22); displaying a display image in the active display window (22) based on the cross-sectional image; - upon receiving an object display command, displaying in the three-dimensional scene (14) a 3D object (24) at the current position of the cursor (20); The method includes:

2. For each displayed 3D object (24), and For each active display window (22): displaying a corresponding 2D object (26) in said active viewing window (22). Further comprising: each second point of the 2D object (26) corresponds to a first point of the 3D object (24); at least one second point of the 2D object (26) is located on a cross-sectional image corresponding to the display image displayed in the active viewing window (22) at a pixel position associated with a voxel that is closest to the first point of the 3D object (24) corresponding to the second point of the 2D object (26); The method of claim 1.

3. Upon receiving an object movement command including a selected 2D object (26), an initial position and a final position, - moving the position of the selected 2D object (26) in the active viewing window (22) based on the initial position and the final position to obtain a registered 2D object (26); - updating the position of the corresponding 3D object (24) based on the position of the registered 2D object (26) such that for at least one first point of the 3D object (24), the closest voxel to the first point is the voxel associated with the pixel on the cross-sectional image corresponding to the display image on the active viewing window (22) that is closest to the second point in the 2D object (26) corresponding to the first point; The method of claim 2 , further comprising:

4. 4. The method according to claim 1, further comprising, for a given pair of consecutive 3D objects (24), each pair of consecutive 3D objects (24) comprising a first 3D object and a second 3D object (24) associated respectively with a first reception of a first display command and a second reception of a second display command, and wherein the first reception and the second reception are consecutive, displaying in the three-dimensional scene (14) a segment connecting two 3D objects (24) of the pair of consecutive 3D objects (24), preferably comprising a cumulative length of the segment.

5. The method according to any one of claims 1 to 4, wherein the object display instructions include an initial and a final position of the cursor (20) in the scene coordinate system (16), and a shape, size and / or orientation of the 3D object (24) depending on the initial and final position of the cursor (20).

6. The method of any one of claims 1 to 5, further comprising the step of displaying data representative of the geometry of at least one 3D object (24).

7. The method according to any one of claims 1 to 6, further comprising, upon receiving a transformation command, applying a predefined transformation to each voxel of the 3D model (5) that is located within a predefined range relative to the current position of the cursor (20) in the three-dimensional scene (14).

8. The above modification is - reducing the opacity of each voxel of the 3D model (5) that is located at a distance greater than a first predetermined radius from the current position of the cursor (20) in the three-dimensional scene (14), and / or - modifying the color value of each voxel of the 3D model (5) located at a distance smaller than a second predetermined radius from the current position of the cursor (20) in the three-dimensional scene (14) to form highlighted voxels. The method of claim 7, comprising:

9. For each enhanced voxel, For each active viewing window (22), and For each display image corresponding to a cross-sectional image that includes a pixel associated with the enhanced voxel, modifying the color of the pixel associated with the highlighted voxel to form a highlighted pixel. The method of claim 8 further comprising:

10. The method of any one of claims 1 to 9, further comprising the step of displaying, for each active viewing window (22), a line extending between the active voxel and the corresponding pixel (19) of the cross-sectional image displayed in the active viewing window (22).

11. - determining a current observation vector representing the direction in which the user is looking at the 3D model (5) in the three-dimensional scene (14); said at least one active display window (22), The determined current observation vector; the normal vector of each of the display planes oriented opposite the 3D model (5); determining a viewing window (22) corresponding to positive values ​​of the scalar product of The method of any one of claims 1 to 10, further comprising:

12. Each cross-sectional image is associated with an image orientation, each viewing window (22) is associated with a window orientation, and a viewing image corresponding to each cross-sectional image is the cross-sectional image if the orientation of the image is the same as the orientation of the window, or - a mirror image of said cross-sectional image if the orientation of said image differs from the orientation of said window The method according to any one of claims 1 to 11, wherein

13. A computer program product comprising instructions which, when said program is executed by a computer, cause said computer to carry out the method according to any one of claims 1 to 12.

14. A visualization device comprising a user interface, a display device and a processing device, The user interface is configured to obtain a current position of a given user point in a given user coordinate system (12); the display device is configured to display a 3D model (5) comprising a three-dimensional representation of at least a portion of a patient's body in a three-dimensional scene (14), each voxel of the 3D model (5) being associated with a respective pixel of at least one corresponding cross-sectional image of the at least a portion of the patient's body, each cross-sectional image being associated with a corresponding image cross-sectional plane, the three-dimensional scene (14) having a corresponding scene coordinate system (16) associated therewith, the three-dimensional scene (14) further comprising a cursor (20) and at least one viewing window (22), each viewing window (22) in a respective viewing plane having a predefined equation in the scene coordinate system (16), each viewing window (22) being associated with a corresponding window cross-sectional plane; The processing device includes: - calculating the current position of the cursor (20) in the scene coordinate system (16) based on the current position of a user point in the user coordinate system (12); for each active display window (22) of said at least one display window (22), - including a pixel (19) associated with an active voxel, which is the voxel having a position closest to the calculated current position of the cursor (20); and having the corresponding image cross-sectional plane coincident with a window cross-sectional plane of the active viewing window (22); controlling the display device to display a display image in the active viewing window (22) based on the cross-sectional image; and - upon receiving an object display command from the user interface, controlling the display device to display in the three-dimensional scene (14) a 3D object (24) at the current position of the cursor (20); A visualization device configured as follows.

15. 15. The visualization device of claim 14, wherein the display device comprises at least part of a virtual reality headset and the user interface preferably comprises a handheld motion tracking sensor.

Images