Method for displaying a 3D model of a patient

Abstract

A method for displaying a 3D model of a patient. The invention relates to a computer-implemented method for displaying a 3D model of a patient's body in a 3D scene, each voxel of the displayed 3D model being associated with a respective pixel of at least one corresponding tomographic image of the patient's body, the 3D scene further comprising a cursor and at least one display window, the method comprises:-calculating a current position of a cursor in a scene coordinate system of the 3D scene based on a current position of a user point in a user coordinate system; and-for each active display window of the at least one display window, displaying a display image based on a tomographic image comprising pixels associated with voxels having a position closest to the calculated current position of the cursor; and has a corresponding image tomographic plane that matches a window tomographic plane of the active display window. The abstract drawing is figure 4.

Description

Technical Field

[0001] The invention relates to a computer-implemented method for displaying a 3D model comprising a three-dimensional representation of at least one part of a patient in a three-dimensional scene.

[0002] The invention also relates to a visualization device and a method of receiving a patient for diagnosis, treatment and / or surgery.

[0003] The present invention relates to the field of medical imaging. Background Art

[0004] It is generally known to use a tomographic image of a patient, such as a CT scan (Computed Tomography scan) image, a PET (Positron Emission Tomography) image or an MRI (Magnetic Resonance Imaging) image, to identify a lesion (e.g., a tumor). Such a tomographic image can also be used to locate the lesion as a preliminary step to surgically ablate it.

[0005] However, this approach is not entirely satisfactory.

[0006] In fact, for many surgeons, in complex cases with multiple lesions, the actual volume and precise location of the lesions cannot be well understood using the two-dimensional visualization based on conventional radiology using the above-mentioned tomographic images. This may lead to excessive tissue removal when ablating the lesions, which is unacceptable from the perspective of the patient's physical integrity.

[0007] An object of the present invention is to provide a method for visualizing at least one portion of a patient's body that provides improved readability to a healthcare provider, thereby improving the ability to accurately locate lesions and minimizing unnecessary tissue removal. Summary of the invention

[0008] For this purpose, the invention is a method of the type described above, wherein each voxel of the displayed 3D model is associated with a corresponding pixel of at least one corresponding tomographic image of at least one part of the patient's body,

[0009] Each tomographic image is associated with a corresponding image tomographic plane.

[0010] The three-dimensional scene comes with a corresponding scene coordinate system.

[0011] The three-dimensional scene also includes a cursor and at least one display window.

[0012] Each display window is located in a corresponding display plane, which has a predetermined equation in the scene coordinate system.

[0013] Each display window is associated with a corresponding window slice plane.

[0014] The method comprises:

[0015] - calculating the current position of the cursor in the scene coordinate system based on the current position of the predetermined user point in the predetermined user coordinate system;

[0016] - for each active display window of the at least one display window, displaying on the active display window a display image based on a tomographic image, the tomographic image:

[0017] • including pixels associated with an active voxel, the active voxel being the voxel having a position closest to the calculated current position of the cursor; and

[0018] ● having a corresponding image slice plane that matches the window slice plane of the active display window; and

[0019] - after receiving the object display instruction, displaying the 3D object at the current position of the cursor in the three-dimensional scene.

[0020] In fact, this method allows interactive navigation in the 3D model and provides in real time the relevant slice images of the currently selected voxel. The displayed slice images provide the user with guidance on the actual position of the voxels of the selected 3D model in the slice images to which he is usually accustomed. Thus, the user can achieve a complete volumetric understanding of the lesion location in the patient's body and the 3D geometry in relation to the surrounding tissue.

[0021] Furthermore, the invention allows the user to include in his field of view both a global three-dimensional representation of the patient's anatomy and at least one local standard two-dimensional image associated with the desired voxels. Thus, the surgeon will be able to visualize the standard tomographic images he uses to plan the operation while being able to refer to a three-dimensional model, which is closer to the reality he will face during the operation.

[0022] Furthermore, the present invention provides the user with the opportunity to precisely place 3D objects (such as landmarks or predetermined volumes) in a three-dimensional scene, which can better assess the geometry and / or position of anatomical structures in the patient's body. This is because the present invention allows the user to accurately determine the position of the area near the cursor in real time.

[0023] In addition, the method according to the invention provides a tool that allows efficient collaboration between medical professionals (i.e. surgeons and radiologists) with different fields of expertise. In fact, on the one hand, the stereoscopic visualization of the 3D model in the three-dimensional scene provides a visual effect close to that of a surgeon operating in a surgical operating room. On the other hand, the tomographic images displayed on the background in an interactive and synchronous manner are standard radiological images (i.e. CT scans or MRI slice images), which are mainly used by health care personnel and surgeons for diagnosis and treatment planning. This promotes a better mutual understanding between the medical professionals, which is beneficial to patients undergoing surgery.

[0024] Furthermore, the generation of 3D models does not require pre-definition of tissue boundaries, thus reducing computational complexity and computational time.

[0025] Furthermore, when dealing with poorly defined lesions, which is often the case in breast MRI, the method according to the invention allows analyzing the border regions in an unbiased manner.

[0026] Therefore, using the method in surgical planning can save more tissue during surgical resection of malignant lesions. This increases the chances of performing oncoplastic surgery with less disfiguring effects and without compromising patient safety.

[0027] According to other advantageous aspects of the invention, the method comprises one or more of the following features, which are achieved individually or in any technically possible combination:

[0028] The method further comprises, for each displayed 3D object, and for each active display window, displaying a corresponding 2D object on the active display window, each second point of the 2D object corresponding to a first point of the 3D object, and at least one second point of the 2D object being located at a position of a pixel associated with a voxel closest to a first point of the 3D object corresponding to the second point of the 2D object on a tomographic image corresponding to a display image displayed in the active display window;

[0029] The method further comprises, after receiving the object movement instruction (including the selected 2D object, the initial position and the final position):

[0030] - Based on the initial position and the final position, the position of the selected 2D object in the active display window is moved to obtain a registered 2D object;

[0031] -Based on the position of the registered 2D object, update the position of the corresponding 3D object so that for at least one first point of the 3D object, the voxel closest to the first point is the voxel associated with the pixel of the second point of the 2D object that is closest to the first point on the tomographic image corresponding to the display image on the active display window.

[0032] The method further comprises, for a given pair of consecutive 3D objects, each pair of consecutive 3D objects comprising a first 3D object and a second 3D object respectively associated with a first reception of a first display instruction and a second reception of a second display instruction, the first reception and the second reception being consecutive, displaying in the three-dimensional scene a line segment connecting two 3D objects of the pair of consecutive 3D objects, the method preferably comprising displaying a cumulative length of the line segment;

[0033] The object display instruction includes an initial position and a final position of the cursor in a scene coordinate system, and a shape, size and / or orientation of a 3D object depending on the initial position and the final position of the cursor;

[0034] The method further comprises displaying data representing the geometry of at least one 3D object;

[0035] The method further includes, upon receiving the transformation instruction, applying a predetermined transformation to each voxel of the 3D model within a predetermined range relative to the current position of the cursor in the three-dimensional scene;

[0036] Transformations include:

[0037] - reducing the opacity of each voxel of the 3D model in the three-dimensional scene whose distance from the current position of the cursor is greater than a first predetermined radius; and / or

[0038] - modifying the color value of each voxel of the 3D model whose distance from the current position of the cursor in the three-dimensional scene is less than a second predetermined radius to form a highlighted voxel;

[0039] The method further comprises, for each highlighted voxel, for each active display window, and for each displayed image corresponding to the tomographic image including the pixel associated with the highlighted voxel, modifying the color of the pixel associated with the highlighted voxel to form a highlighted pixel;

[0040] The method further comprises, for each active display window, displaying a line extending between the active voxel and a corresponding pixel of the tomographic image displayed on the active display window;

[0041] The method further comprises:

[0042] - determining a current viewing vector representing the direction in which the user is viewing the 3D model in the three-dimensional scene; and

[0043] - Determine as at least one active display window a display window corresponding to a positive value of the scalar product between the following vectors:

[0044] ● the determined current observation vector; and

[0045] ● The normal vector of the corresponding display plane oriented opposite to the 3D model;

[0046] Each tomographic image is associated with an image orientation, each display window is associated with a window orientation, and the display image corresponding to each tomographic image is:

[0047] - a tomographic image, when the image orientation is the same as the window orientation; or

[0048] - Mirror image of the slice image, in case the image orientation is different from the window orientation.

[0049] The invention also relates to a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method as described above.

[0050] The present invention also relates to a non-transitory computer-readable storage medium, in which computer-executable instructions are stored. When executed by a computer, the instructions cause the computer to perform the steps of the method as described above.

[0051] The present invention also relates to a visualization device comprising a user interface, a display unit and a processing unit, wherein:

[0052] The user interface is configured to obtain a current position of a predetermined user point in a predetermined user coordinate system;

[0053] The display unit is configured to display a 3D model including a three-dimensional representation of at least one part of the patient in a three-dimensional scene, each voxel of the 3D model is associated with a corresponding pixel of at least one corresponding tomographic image of the at least one part of the patient, each tomographic image is associated with a corresponding image tomographic plane, the three-dimensional scene is accompanied by a corresponding scene coordinate system, the three-dimensional scene also includes a cursor and at least one display window, each display window is located in a corresponding display plane, the display plane has a predetermined equation in the scene coordinate system, and each display window is associated with a corresponding window tomographic plane; and

[0054] The processing unit is configured as:

[0055] - Based on the current position of the user point in the user coordinate system, calculate the current position of the cursor in the scene coordinate system;

[0056] - for each active display window of the at least one display window, controlling the display windows to display a display image based on the tomographic image on the active display window, the tomographic image:

[0057] • including pixels associated with an active voxel, the active voxel being the voxel having a position closest to the calculated current position of the cursor; and

[0058] ● having a corresponding image slice plane that matches the window slice plane of the active display window; and

[0059] - after receiving an object display instruction from the user interface, controlling the display unit to display the 3D object at the current position of the cursor in the three-dimensional scene.

[0060] According to other advantageous aspects of the invention, the visualization device comprises one or more of the following features, which are achieved individually or in any technically possible combination:

[0061] The display unit comprises at least a portion of a virtual reality helmet, and the user interface preferably comprises a handheld motion tracking sensor;

[0062] At least one image slice plane is one of: a sagittal plane, a coronal plane, or a transverse plane.

[0063] The present invention also relates to a method of receiving a patient for diagnosis, treatment and / or surgery, the method comprising:

[0064] - generating a 3D model of at least one portion of the patient, each voxel of the 3D model being associated with a corresponding pixel of at least one corresponding tomographic image of the at least one portion of the patient, each tomographic image being associated with a corresponding image tomographic plane;

[0065] - displaying the 3D model to a user in a three-dimensional scene with a corresponding scene coordinate system, the three-dimensional scene also comprising a cursor and at least one display window, each display window being located in a corresponding display plane, the display plane having a predetermined equation in the scene coordinate system, each display window being associated with a corresponding window slice plane;

[0066] - calculating the current position of the cursor in the scene coordinate system based on the current position of a predetermined user point bound to the user in the predetermined user coordinate system; and

[0067] - for each active display window of the at least one display window, displaying on the active display window a display image based on a tomographic image, the tomographic image:

[0068] • including pixels associated with an active voxel, the active voxel being the voxel having a position closest to the calculated current position of the cursor; and

[0069] ● having a corresponding image slice plane that matches the window slice plane of the active display window;

[0070] - after receiving the object display instruction, displaying the 3D object at the current position of the cursor in the three-dimensional scene; and

[0071] - Providing diagnosis and / or treatment and / or surgical strategy suggestions based at least on the result of the user's navigation of the 3D model and / or the 3D object in the three-dimensional scene.

[0072] According to a further advantageous aspect of the invention, the user is a medical professional, preferably a surgeon or a radiologist. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] The present invention will be better understood through the accompanying drawings, in which:

[0074] Figure 1 is a schematic diagram of a visualization device according to the present invention;

[0075] Figure 2 Is Figure 1 A schematic diagram of a three-dimensional scene displayed by a visualization device; and

[0076] Figure 3 yes Figure 2 Schematic diagram of a three-dimensional scene with the observation direction and Figure 2 The observation direction is different;

[0077] Figure 4 and Figure 2 Similarly, 3D objects in the three-dimensional scene are further displayed;

[0078] Figure 5 yes Figure 1 A schematic diagram of a processing unit of a visualization device; and

[0079] Figure 6 is a flow chart of a method of receiving a patient according to the present invention. DETAILED DESCRIPTION

[0080] This specification illustrates the principles of the present disclosure. It should therefore be appreciated that those skilled in the art will be able to design various devices that, although not explicitly described or shown herein, embody the principles of the present disclosure and are included within the scope of the present disclosure.

[0081] All examples and conditional language described herein are intended for educational purposes to help the reader understand the principles of the present disclosure and the concepts contributed by the inventors to further the art, and should be understood as not being limited to these specifically recited examples and conditions.

[0082] In addition, all statements describing the principles, aspects and embodiments of the present disclosure and their specific examples herein are intended to include their structural and functional equivalents. In addition, it is contemplated that such equivalents include currently known equivalents as well as equivalents developed in the future (i.e., any element that performs the same function, regardless of the structure).

[0083] Thus, for example, those skilled in the art will appreciate that the block diagrams presented herein may represent conceptual views of illustrative circuits embodying the principles of the present disclosure. Similarly, it will be appreciated that any flow charts, flow diagrams, and the like represent various processes that may be substantially represented in a computer-readable medium and thus executed by a computer or processor, whether or not such computer or processor is explicitly shown.

[0084] The functions of the various elements shown in the figures may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, a single shared processor, or a plurality of individual processors, some of which may be shared.

[0085] It should be understood that the elements shown in the figure can be implemented with various forms of hardware, software or a combination thereof. Preferably, these elements are implemented with a combination of hardware and software on one or more appropriately programmed general-purpose devices, which may include a processor, a memory and an input / output interface.

[0086] The visualization device 2 according to the present invention is Figure 1 Shown in.

[0087] Will refer to Figure 1 and Figure 2 The structure and operation of the visualization device 2 will be described.

[0088] The visualization device 2 is configured to display a 3D model 5 forming a three-dimensional representation of at least one part of a patient's body to a user 4, such as a medical professional (eg a surgeon or a radiologist). The visualization device 2 is also configured to provide the user with the ability to interact with the 3D model 5.

[0089] By "3D model" is meant a 3D volume rendering of a 3D medical image (or in other words, a voxel-based three-dimensional representation of a medical image). The medical image consists of voxels. 3D volume rendering can be achieved by a variety of techniques: surface segmentation and rendering, ray casting, ray tracing, path tracing, etc. The 3D model is rendered in a 3D scene with a coordinate system. The term "each voxel of the displayed 3D model" should therefore be understood as "each voxel of the three-dimensional scene in which the 3D model is rendered".

[0090] The voxels of the original 3D medical image are associated with the voxels of the three-dimensional scene, thereby establishing a correspondence between the 3D model and the original data. In an embodiment, the 3D model is obtained by surface rendering of at least one object file corresponding to a specific anatomical structure. The object file is registered with the original medical image data in the scene. In another embodiment, volume rendering uses techniques involving light diffusion modeling (ray casting, ray tracing, path tracing, etc.), which means that there is a direct relationship between the rendering and the representation of the medical image data in the three-dimensional scene.

[0091] The at least one part of the patient mentioned above includes, for example, at least one part of one or more organs of the patient.

[0092] Although the presently described visualization device 2 is generic and provides several functions that can be performed alternatively or in any cumulative manner, other implementations within the scope of the present disclosure include devices having only some of the described functions.

[0093] The visualization device 2 is advantageously a device or a physical part of a device designed, configured and / or adapted to perform the mentioned functions and produce the mentioned effects or results. In alternative embodiments, the visualization device 2 is embodied as a set of devices or physical parts of devices, whether grouped in the same machine or in different, possibly remote machines. The visualization device 2 may, for example, have functionality distributed on a cloud infrastructure and available as a cloud-based service to users, or have remote functionality accessible via an API (application programming interface).

[0094] The visualization device 2 comprises a user interface 6 for interacting with the 3D model 5 , a display unit 8 for displaying the 3D model 5 , and a processing unit 10 for processing data received from the user interface 6 and for controlling the display unit 8 .

[0095] User Interface 6

[0096] The user interface 6 is configured to acquire the current position of a predetermined user point A in a predetermined user coordinate system 12. The predetermined user coordinate system 12 is fixed with respect to the user's real environment.

[0097] The user coordinate system 12 is fixed, for example, relative to the user's real environment.

[0098] The user interface 6 may include a handheld motion tracking sensor, allowing the user 4 to interact with the 3D model 5 (and / or the three-dimensional scene 14 in which the 3D model 5 is represented) using gestures. In this case, the user point A may be the point of the motion tracking sensor.

[0099] The user interface 6 may also include other devices, such as a touch screen, a game controller, a mouse and / or a keyboard, to further allow the user 4 to interact with the 3D model 5 (and / or the three-dimensional scene in which the 3D model 5 is displayed).

[0100] The user interface 6 may also include buttons and / or switches configured to allow the user 4 to interact with the 3D model 5 (and / or the three-dimensional scene in which the 3D model 5 is displayed). Such buttons and / or switches may be included in the handheld motion tracking sensor, or may be included in a separate device of the user interface 6, such as the touch screen, game controller, mouse and / or keyboard mentioned above.

[0101] Preferably, the user interface 6 is configured to allow the user 4 to input specific instructions (e.g., via the buttons and / or switches), such as object display instructions, object movement instructions, or transformation instructions. The instructions advantageously allow the user 4 to add and / or manipulate 3D images in the same scene as the 3D model 5, as described below.

[0102] The user interface 6 may also be configured to allow the user to manipulate (eg, rotate and / or zoom in or out) the 3D model 5 , and / or change the direction in which the user 4 views the 3D model 5 displayed by the display unit 8 .

[0103] The user interface 6 may also be configured to allow a user to manipulate (eg, move and / or rotate) 2D objects displayed in the three-dimensional scene, and more specifically 2D objects displayed in a display window (described below) of the three-dimensional scene.

[0104] Alternatively, or additionally, the user interface 6 comprises virtual buttons displayed in the three-dimensional scene to allow the user 4 to interact with the 3D model 5 , the three-dimensional scene, one or more 3D objects and / or one or more 2D objects.

[0105] Display unit 8

[0106] As mentioned before, the display unit 8 is configured to display the 3D model 5 .

[0107] Advantageously, the display unit 8 is at least part of a virtual reality helmet, allowing stereoscopic visualization of the 3D model 5. This is particularly advantageous in the medical field, since virtual reality visualization of the 3D model 5 allows the user 4 to have a good understanding of the actual volume and precise location of the underlying lesion. For example, in the case of a tumor, this increases the chances of performing oncoplastic surgery with less disfiguring effects and without compromising patient safety.

[0108] The virtual reality helmet may include one or more sensors, such as one or more accelerometers and / or one or more gyroscopes. In this case, each sensor is part of the user interface 6 and is configured to output a signal representing the direction in which the user is viewing the 3D model 5 in the three-dimensional scene 14 described below.

[0109] Alternatively, or additionally, the display unit 8 comprises Figure 1 Screen 9 shown.

[0110] More precisely, in the above-mentioned three-dimensional scene 14, the 3D model 5 forms a three-dimensional representation of at least one part of the patient's body. The three-dimensional scene (hereinafter also referred to as "3D scene") is accompanied by a scene coordinate system 16.

[0111] Furthermore, each voxel of the 3D model 5 is associated with a respective pixel of at least one corresponding tomographic image of at least one part of the patient.

[0112] Each tomographic image is associated with a corresponding image slice plane, and preferably, a slice 17 of the portion is shown in the corresponding image slice plane.

[0113] The image slice plane may be one of the sagittal, coronal or transverse planes, and more specifically the sagittal, coronal or transverse plane of the patient.

[0114] For example, each tomographic image is previously acquired using magnetic resonance imaging (MRI), such as T1-weighted enhanced MRI, CT scan imaging, or PET imaging, or any acquisition technique that can be used for preoperative preparation and planning.

[0115] Furthermore, each tomographic image is preferably stored in a database 18 (eg, a picture archiving and communication system) such as a medical institution (eg, a hospital).

[0116] Preferably, each slice image is stored in the database 18 in a standard DICOM (Digital Imaging and Communications in Medicine) file format. This is advantageous because a DICOM file is associated with a scale representing the imaged anatomical structure. Therefore, when the scale of the 3D scene 14 matches the scale of the DICOM file, geometric features (distances, surface areas and / or volumes) associated with the 3D model 5 can be accurately measured.

[0117] Advantageously, each slice image is also associated with an image orientation.

[0118] The so-called “image orientation of a tomographic image” in the context of the present invention means the direction along which the patient is observed on the tomographic image.

[0119] For example, where the cross section is along the patient's sagittal plane, the image orientation may be "left view" or "right view."

[0120] As another example, where the cross section is along the patient's coronal plane, the image orientation may be "front view" or "posterior view."

[0121] Also for example, where the cross section is along a transverse plane of the patient, the image orientation may be a "top view" or a "bottom view."

[0122] In addition, the 3D scene 14 displayed by the display unit 8 further includes a cursor 20 and at least one display window 22. For example, the 3D scene 14 includes six display windows 22.

[0123] Each display window 22 is located in a respective display plane, and the display plane has a predetermined equation in the scene coordinate system 16. For example, at least two display planes are orthogonal, thereby simulating the orthogonality between commonly used tomographic planes (sagittal plane, coronal plane, and transverse plane) in medical imaging, thereby improving the readability of the 3D scene 14. Figure 2 and 3 As shown, the display plane surrounds the 3D model 5, thereby allowing the 3D model 5 and each display window 22 to be simultaneously visualized in three dimensions.

[0124] More specifically, each display plane is preferably orthogonal to the other four display planes, so that each of the six display windows 22 is located on a corresponding face of the parallelepiped.

[0125] In addition, each display window 22 is associated with a corresponding window slice plane, which can be a sagittal plane, a coronal plane or a transverse plane.

[0126] Furthermore, there may be zero, one, or more, and preferably three, active display windows 22 out of the total number of display windows 22 at any given time.

[0127] Each active display window is defined as a display window that displays an image, which is hereinafter referred to as a “display image.” Conversely, at a given time, a display window 22 that is not an active display window may not display an image and therefore may not be visible in the 3D scene 14 .

[0128] The determination of the active display window will be described below.

[0129] Advantageously, each display window 22 is associated with a respective window orientation. In addition, two display windows 22 located in parallel display planes are associated with opposite window orientations, such as: "left view" and "right view", "front view" and "rear view", or "top view" and "bottom view". The advantages of this feature will be discussed below.

[0130] The display unit 8 is further configured to display at least one 3D object 24 in the 3D scene 14, such as Figure 4 shown.

[0131] Each 3D object may include any number of points. For example, the 3D object is a sign including at least one point. Alternatively, the 3D object is a line segment including at least two points, each point corresponding to an end of the line segment. For another example, the 3D object is a surface including at least three non-coplanar points. Alternatively, the 3D object has a three-dimensional shape and includes at least four non-coplanar points.

[0132] The 3D object may also be a 2D or 3D segmentation mask (e.g., a mesh) for identifying and / or highlighting anatomical structures having predetermined properties. In this case, the visualization device 2 may be provided with a library of segmentation masks stored thereon. The 3D object may also represent a medical device. In the latter case, the 3D object may be generated from a 3D object file (e.g., provided in .stl or .obj file format) representing the physical medical device.

[0133] Processing unit 10

[0134] According to the present invention, the expression "processing unit" should not be interpreted as being limited to hardware capable of executing software, and refers to a processing device in a general manner, which can include, for example, a microprocessor, an integrated circuit or a programmable logic device (PLD). The processing unit may also include one or more graphics processing units (GPUs) or tensor processing units (TSUs), whether for computer graphics and image processing or other functions. In addition, instructions and / or data capable of performing related and / or resulting functions may be stored on any processor-readable medium, such as an integrated circuit, a hard disk, a CD (compact disk), an optical disk (such as a DVD (digital versatile disk), a RAM (random access memory) or a ROM (read-only memory). Instructions may be stored in particular in hardware, software, firmware or any combination thereof.

[0135] The processing unit 10 is connected to each of the user interface 6 and the display unit 8 .

[0136] The processing unit 10 corresponds to, for example, a workstation, a laptop computer, a tablet computer, a smart phone, a programmable logic device (such as an FPGA) for onboard computing, or a head-mounted display (HMD) such as a virtual reality helmet.

[0137] like Figure 4 As shown, the processing unit 10 may include the following elements, which are connected to each other via an address and data bus 95, which also transmits a clock signal:

[0138] - microprocessor 91 (or CPU);

[0139] - Graphics card 92, which includes several graphics processing units (or GPUs) 920 and graphics random access memory (GRAM) 921; GPUs are well suited for image processing due to their highly parallel structure;

[0140] - ROM type non-volatile memory 96;

[0141] - RAM 97;

[0142] - a power supply 98; and

[0143] - Radio frequency unit 99.

[0144] Alternatively, power supply 98 is external to processing unit 10 .

[0145] The user interface 6 is connected to at least part of the above-mentioned modules, for example, via a bus 95 .

[0146] The display unit 8 is connected to the graphics card 92 via a suitable interface, for example. For example, a cable may be used for wired transmission, or an RF unit 99 may be used for wireless transmission.

[0147] Each of the memories 97 and 921 includes registers (which can be specified in each memory), a low-capacity memory area (some binary data), and a large-capacity memory area (capable of storing an entire program or all or part of a data representation of calculation or display data). Also, the registers representing the RAM 97 and the GRAM 921 can be arranged and constructed in any manner. Each of these registers does not necessarily correspond to adjacent storage locations, and can be distributed in other ways (this is particularly applicable to the case where one register contains several smaller registers).

[0148] When turned on, microprocessor 91 loads and executes instructions of program 970 contained in RAM 97 to allow visualization device 2 to be operated in the manner described in the present disclosure.

[0149] As the skilled person will appreciate, the graphics card 92 is not mandatory and may be replaced with full CPU processing and / or other implementations.

[0150] The processing unit 10 is configured to calculate the current position of the cursor 20 in the scene coordinate system 16 based on the current position of the user point A in the user coordinate system 12 .

[0151] Furthermore, for each active display window 22, the processing unit 10 is configured to control the display unit 8 to display a corresponding display image on the active display window. The display image is based on a tomographic image, wherein:

[0152] - includes the pixel 19 associated with the currently active voxel; and

[0153] - having a corresponding first slice plane that matches the window slice plane of the active display window.

[0154] Such an active voxel is defined as the voxel having a position in the 3D scene 14 that is closest to the calculated current position of the cursor 20 .

[0155] Furthermore, in the context of the present invention, the expression "the first slice plane matches the window slice plane of the active display window" means that the first slice plane is identical to the window slice plane.

[0156] Furthermore, the processing unit 10 is configured to detect an object display instruction input by the user 4 through the user interface 6. Upon receiving the object display instruction, the processing unit 10 is configured to control the display unit 8 to display the 3D object at the current position of the cursor 20 in the 3D scene 14.

[0157] Advantageously, the object display instruction further comprises the initial position and the final position of the cursor 20 in the scene coordinate system 16. In this case, the processing unit 10 is configured to determine the shape, size and / or orientation of the 3D object 24 according to the initial position and the final position of the cursor 20.

[0158] The initial position and the final position in the scene coordinate system 16 may be determined by the processing unit 10 based on the movement of the user point A in the user coordinate system 12 .

[0159] This is advantageous, for example, when the 3D object is one of several 3D segmentation masks stored in a library, each mask corresponding to an organ or a part of an organ, or when the 3D object represents a medical device to be implanted in a patient. In this case, the user 4 can adjust the characteristics of the mask (e.g. shape, size and / or orientation) or the model of the medical device via the user interface 6.

[0160] More specifically, the processing unit 10 is configured to control the display unit 8 to display the 3D object so that the position of a predetermined point of the 3D object coincides with the current position of the cursor 20. The predetermined point may be the center of gravity of the 3D object, an endpoint of the 3D object, a point on the outer surface of the 3D object, etc.

[0161] Optionally, the processing unit 10 is further configured to determine, for each displayed 3D object 24, at least one corresponding 2D object 26. In this case, the processing unit 10 is configured to control the display unit 8 to display each determined 2D object 26 on the corresponding active display window.

[0162] Preferably, for a given 3D object 24, and for a given active display window, the 2D objects correspond to slices of the 3D object on the corresponding image slice plane. In other words, for a given 3D object 24, and for a given active display window 22, each 2D object 26 corresponds to the intersection of the 3D object 24 with a plane defined by voxels associated with pixels of the slice image on which the display image currently displayed in the active display window 22 is based.

[0163] Preferably, each second point of the 2D object corresponds to a first point of the 3D object. In this case, for each displayed 3D object 24 and for each active display window 22, the processing unit 10 is configured to control the display unit 8 so that, on the tomographic image corresponding to the display image currently displayed in the active display window, at least one second point of the 2D object 26 is located at the position of the pixel associated with the voxel of the first point of the 3D object 24 that is closest to the second point of the 2D object 26.

[0164] Advantageously, the processing unit 10 is further configured to control the display unit 8 to display, for each active display window 20, a line extending between the active voxel and the corresponding pixel 19 of the tomographic image displayed on the active display window 22. Thus, the user 4 can simply locate the active voxel in the 3D model 5 using the position of each corresponding pixel 19 in the image currently displayed in the display window.

[0165] Preferably, the processing unit 10 is configured to detect an object movement instruction input by the user 4 through the user interface 6. The object movement instruction includes the selected 2D object, the initial position and the final position. The initial position and the final position may be, for example, the initial position and the final position of the user point A in the user coordinate system 12, or the initial position and the final position of the cursor 20 in the scene coordinate system 16.

[0166] After receiving the object movement instruction, the processing unit 10 is configured to move the position of the selected 2D object in the corresponding active display window based on the initial position and the final position to obtain a registered 2D object.

[0167] Furthermore, the processing unit 10 is configured to update the position of the 3D object 24 in the 3D scene 14 based on the position of the registered 2D object 26. More specifically, the processing unit 10 is configured to update the position of the 3D object 24 so that for at least one first point of the 3D object 24, the voxel closest to the first point is the voxel associated with the pixel of the second point of the 2D object that is closest to the first point on the tomographic image corresponding to the display image currently displayed on the active display window.

[0168] Thus, by moving each 2D object in the corresponding active display window, the user 4 is able to interact with and move the corresponding 3D object 24. This is particularly advantageous when the active display window is also displayed on the screen 9, thereby providing the user 4 with a working environment more suitable for such interaction by displaying 2D images on a 2D interface. In addition, this feature allows the user 4 to use stereoscopic visualization to guide the 3D models 5 and quickly place 3D models in the 3D scene 14 and adjust these 3D models on the 2D interface he is using.

[0169] Preferably, the processing unit 10 is further configured to detect a transformation instruction input by the user 4 through the user interface 6 .

[0170] In this case, the processing unit 10 is further configured to, upon receiving the transformation instruction, apply a predetermined transformation to at least part of the voxels of the 3D model. More specifically, the processing unit 10 is further configured to apply a predetermined transformation to each voxel within a predetermined range relative to the current position of the cursor 20 in the three-dimensional scene 14.

[0171] For example, the processing unit 10 is configured to control the display unit 8 to reduce the opacity of each voxel of the 3D model 5 located at a position greater than a first predetermined radius from the current position of the cursor 20 in the 3D scene 14. This is advantageous because this feature allows the user 4 to focus on the area of ​​interest located near the cursor 20 and effectively navigate the 3D model 5 by reducing visual interference that may be caused by other voxels located in front of or behind the area of ​​interest.

[0172] For another example, the processing unit 10 is configured to control the display unit 8 to modify the color value of each voxel of the 3D model 5 located at a position less than the second predetermined radius from the current position of the cursor 20 in the 3D scene 14. Therefore, each voxel with a modified color value forms a highlighted voxel.

[0173] In the latter case, the transformation instructions are preferably part of the above-mentioned object display instructions.

[0174] In this case, each highlighted voxel is able to define a 3D object forming a landmark and located at the same position as said highlighted voxel.

[0175] Alternatively, each set of highlighted voxels can define a 3D object forming a surface or volume. For example, the 3D object is a grid connecting the highlighted voxels of the set. For another example, the 3D object is a volume comprising points whose positions are inserted into the positions of at least some of the highlighted voxels in the set.

[0176] For example, the set of highlighted voxels is defined as the highlighted voxels within a predetermined distance from each other. For another example, the set of highlighted voxels is defined as the voxels highlighted between two predetermined events (eg, pressing and then re-pressing a button of the user interface 6).

[0177] Preferably, the processing unit 10 is configured to control the display unit 8 so that, for each highlighted voxel, for each active display window, and for each displayed image corresponding to a tomographic image comprising a pixel associated with the highlighted voxel, the color of the pixel associated with the highlighted voxel is modified to form a highlighted pixel. Thus, the user 4 can view the result of the applied transformation and can adjust the position of the cursor 20 accordingly.

[0178] Preferably, the processing unit 10 is configured to control the display unit 8 to display a line segment connecting at least two consecutive 3D objects in the 3D scene 14. This is particularly applicable when each 3D object is a 3D marker (ie, a point).

[0179] In the context of the present invention, the so-called “two consecutive 3D objects” means two 3D objects each associated with the receipt of two consecutive display instructions. In other words, the user 4 adds two consecutive 3D objects to the displayed 3D scene 14 consecutively.

[0180] Such displayed line segments are advantageous because they allow the user 4 to better understand the geometric relationships (eg, relative positions and / or alignments) between anatomical structures displayed on the 3D model.

[0181] Advantageously, the processing unit 10 is further configured to calculate data representative of the geometry of the at least one 3D object. In this case, the processing unit 10 is further configured to control the display unit 8 to display the calculated data.

[0182] For example, the data corresponds to the cumulative distance between at least two consecutive 3D objects (eg, at least two consecutive 3D markers) in the 3D scene. In other words, the data is equal to the cumulative length of the line segment connecting at least two consecutive 3D models.

[0183] According to other examples, the calculated data includes the volume of the 3D object (sphere, ellipsoid, cube, etc.) dragged by the user into the 3D scene 14. Alternatively, or in addition, the calculated data includes the cumulative volume of the highlighted voxels, defined as the product of the voxel volume and the total number of highlighted voxels.

[0184] Advantageously, the processing unit 10 is configured to first determine a current observation vector for each active display window 22. Such an observation vector represents the direction along which the user 4 views the 3D model 5 in the three-dimensional scene 14 and may be generated by the user 4 interacting with the 3D scene 14 through the user interface.

[0185] Furthermore, the processing unit 10 is configured to determine the display window as an active display window when the display window makes the scalar product between the following vectors positive:

[0186] - on the one hand, determining the current observation vector; and

[0187] On the other hand, the normal vector of the respective display plane, which is oriented opposite to the 3D model 5 .

[0188] Preferably, the processing unit 10 is configured to control the display unit 8 so that the displayed image corresponding to each tomographic image is:

[0189] - the tomographic image itself, in the case where the image orientation is the same as the window orientation; or

[0190] - A mirror image of the tomographic image, when the image orientation is different from the window orientation.

[0191] In the latter case, each pixel (x; y) in the mirror image of a given tomographic image of size (N; M) has the same value as a pixel (N+1-x; y) of the original tomographic image, where N and M are both integers greater than 0. Thus, N is the number of pixels of the tomographic image along its width. Thus, the display image presented on the active display window is presented to the user 4 as a slice of the 3D model observed from the user's perspective (i.e. "from the left", "from the right", "from the front", "from the back", "from the top", "from the bottom"). This improves comprehensibility.

[0192] Therefore, for a given voxel, two display windows located in parallel display planes may display mirror images of each other, such as Figure 2 and Figure 3 It can be understood that the active voxels are the same. In this case, in the active display 22L ( Figure 2 ) and 22R( Figure 3 ) are mirror images of each other.

[0193] More precisely, Figure 3 The example corresponds to Figure 2 The same 3D model 5 in the same 3D scene 14. Figure 3 The scene is composed by Figure 2 The scene in is rotated about 45° counterclockwise along the vertical axis Z.

[0194] operate

[0195] Now refer to Figure 5 The operation of the visualization device 2 is described.

[0196] In the optional acquisition step 30, a patient is admitted to a medical institution and tomographic images of the patient are acquired. Each tomographic image is associated with a corresponding image slice plane.

[0197] Then, in a modeling step 32, a 3D model 5 of at least one part of the patient is generated, preferably based on the acquired tomographic images. Each voxel of the 3D model is associated with a corresponding pixel of at least one corresponding acquired tomographic image.

[0198] Then, in a guiding step 34, the 3D model is displayed to the user 4 in a 3D scene 14 using the visualization device 2. As mentioned above, the 3D scene 14 is accompanied by a corresponding scene coordinate system 16. The 3D scene 14 also includes a cursor 20 and the at least one display window 22 mentioned above.

[0199] The user 4 is preferably a medical professional, such as a surgeon or a radiologist.

[0200] In the guidance step 34 , the processing unit 10 calculates the current position of the cursor 22 in the scene coordinate system 16 based on the current position of the predetermined user point A in the predetermined user coordinate system. The current position of the user point A is obtained through the user interface 6 .

[0201] Preferably, in the guiding step 34, the processing unit 10 determines a current viewing vector representing the direction in which the user views the 3D model in the three-dimensional scene 14. Therefore, the processing unit 10 selects each active display window 22 based on the determined current viewing vector.

[0202] Furthermore, in the guiding step 34, the processing unit 10 controls the display unit 8 to display a display image based on the tomographic image on each active display window 22, which:

[0203] ● including the pixel associated with the currently active voxel; and

[0204] ● Having a corresponding image slice plane that matches the window slice plane of the active display window.

[0205] As previously described, the currently active voxel is the voxel having the position closest to the calculated current position of the cursor 20 .

[0206] In addition, in the guiding step 34, after receiving the object display instruction, the processing unit 10 controls the display unit 8 to display the 3D object at the current position of the cursor in the three-dimensional scene. The user can rotate the shape, size and orientation of the 3D object through the user interface.

[0207] Then, in the evaluation step 36, a diagnosis and / or treatment suggestion and / or surgical strategy suggestion is provided based at least on the result of the user 4 guiding the 3D model and / or the 3D object in the 3D scene 14. Such diagnosis and / or treatment suggestion and / or surgical strategy suggestion may be determined by the user 4 at his / her discretion.

Claims

1. A computer-implemented method for displaying a 3D model (5) comprising a three-dimensional representation of at least one part of a patient's body in a three-dimensional scene (14), Each voxel of the displayed 3D model (5) is associated with a corresponding pixel of at least one corresponding tomographic image of the at least one part of the patient's body, Each tomographic image is associated with a corresponding image tomographic plane. The three-dimensional scene (14) is provided with a corresponding scene coordinate system (16), The three-dimensional scene (14) further includes a cursor (20) and at least one display window (22), Each display window (22) is located in a corresponding display plane, the display plane having a predetermined equation in the scene coordinate system (16), Each display window (22) is associated with a corresponding window slice plane. The method comprises: - calculating a current position of a cursor (20) in a scene coordinate system (16) based on a current position of a predetermined user point (A) in a predetermined user coordinate system; - for each active display window (22) of the at least one display window (22), displaying a display image based on a tomographic image on the active display window (22), the tomographic image: • including pixels (19) associated with an active voxel, the active voxel being the voxel having a position closest to the calculated current position of the cursor (20); and ● having a corresponding image slice plane that matches the window slice plane of the active display window (22); and - after receiving the object display instruction, displaying the 3D object (24) at the current position of the cursor (20) in the three-dimensional scene (14).

2. The method according to claim 1, further comprising: For each displayed 3D object (24), and For each active display window (22), displaying a corresponding 2D object (26) on an active display window (22), Each second point of the 2D object (26) corresponds to a first point of the 3D object (24), On a tomographic image corresponding to a display image displayed on an active display window (22), at least one second point of a 2D object (26) is located at a position of a pixel associated with a voxel of a first point of a 3D object (24) that is closest to the second point of the 2D object (26).

3. The method according to claim 2 further comprises, upon receiving an object movement instruction including the selected 2D object (26), the initial position and the final position: - based on the initial position and the final position, moving the position of the selected 2D object (26) in the active display window (22) to obtain a registered 2D object (26); -Based on the position of the registered 2D object (26), updating the position of the corresponding 3D object (24) so ​​that for at least one first point of the 3D object (24), the nearest voxel to the first point is the voxel associated with the pixel of the second point of the 2D object (26) that is closest to the first point on the tomographic image corresponding to the display image on the active display window (22).

4. The method according to any one of claims 1 to 3, further comprising, for a given pair of continuous 3D objects (24), each pair of continuous 3D objects (24) comprising a first 3D object and a second 3D object (24) respectively associated with a first reception of a first display instruction and a second reception of a second display instruction, the first reception and the second reception being continuous, displaying a line segment connecting the two 3D objects (24) of the pair of continuous 3D objects (24) in the three-dimensional scene (14), the method preferably comprising displaying the cumulative length of the line segment.

5. The method according to any one of claims 1 to 4, wherein the object display instruction comprises an initial position and a final position of a cursor (20) in a scene coordinate system (16), and a shape, size and / or orientation of a 3D object (24) depending on the initial position and the final position of the cursor (20).

6. The method according to any one of claims 1 to 5, further comprising displaying data representing the geometry of at least one 3D object (24).

7. The method according to any one of claims 1 to 6 further comprises, after receiving a transformation instruction, applying a predetermined transformation to each voxel of the 3D model (5) within a predetermined range relative to the current position of the cursor (20) in the three-dimensional scene (14).

8. The method of claim 7, wherein the transforming comprises: - reducing the opacity of each voxel of the 3D model (5) in the three-dimensional scene (14) that is at a distance greater than a first predetermined radius from the current position of the cursor (20); and / or - modifying the color value of each voxel of the 3D model (5) whose distance from the current position of the cursor (20) in the three-dimensional scene (14) is less than a second predetermined radius to form a highlighted voxel.

9. The method according to claim 8, further comprising: For each highlighted voxel, For each active display window (22), and For each displayed image corresponding to a tomographic image including pixels associated with a highlighted voxel, The color of the pixel associated with the highlight voxel is modified to form a highlight pixel.

10. The method according to any one of claims 1 to 9, further comprising, for each active display window (22), displaying a line extending between the active voxel and the corresponding pixel (19) of the tomographic image displayed on the active display window (22).

11. The method according to any one of claims 1 to 10, further comprising: - determining a current viewing vector representing the direction in which the user is viewing the 3D model (5) in the three-dimensional scene (14); and - determining as the at least one active display window (22) a display window (22) corresponding to a positive value of the scalar product between the following vectors: The current observation vector determined; and • The normal vector of the corresponding display plane oriented opposite to the 3D model (5).

12. The method according to any one of claims 1 to 11, wherein each tomographic image is associated with an image orientation, each display window (22) is associated with a window orientation, and the display image corresponding to each tomographic image is: - a tomographic image, when the image orientation is the same as the window orientation; or - A mirror image of the tomographic image, when the image orientation is different from the window orientation.

13. A computer program product comprising instructions which, when the program is executed by a computer, cause the computer to perform the method according to any one of claims 1 to 12.

14. A visualization device, comprising a user interface, a display unit and a processing unit, wherein: The user interface is configured to obtain a current position of a predetermined user point in a predetermined user coordinate system (12); The display unit is configured to display a 3D model (5) including a three-dimensional representation of at least one part of the patient's body in a three-dimensional scene (14), each voxel of the 3D model (5) being associated with a corresponding pixel of at least one corresponding tomographic image of the at least one part of the patient's body, each tomographic image being associated with a corresponding image tomographic plane, the three-dimensional scene (14) being accompanied by a corresponding scene coordinate system (16), the three-dimensional scene (14) further comprising a cursor (20) and at least one display window (22), each display window (22) being located in a corresponding display plane, the display plane having a predetermined equation in the scene coordinate system (16), each display window (22) being associated with a corresponding window tomographic plane; and The processing unit is configured as: - based on the current position of the user point in the user coordinate system (12), calculating the current position of the cursor (20) in the scene coordinate system (16); - for each active display window (22) of the at least one display window (22), controlling the display unit to display a display image based on a tomographic image on the active display window (22), wherein the tomographic image: including pixels (19) associated with an active voxel, the active voxel being the voxel having a position closest to the calculated current position of the cursor (20); and having a corresponding image slice plane that matches the window slice plane of the active display window (22); and - after receiving an object display instruction from the user interface, controlling the display unit to display a 3D object (24) at the current position of the cursor (20) in the three-dimensional scene (14).

15. The visualization device of claim 14, wherein the display unit comprises at least a portion of a virtual reality helmet, and the user interface preferably comprises a handheld motion tracking sensor.

Images