Medical image processing method for intraoperative assistance during a surgical procedure

Abstract

The invention relates to a method for processing medical images, said method comprising a step (E2) of generating at least one distorted 3D image of the structure of interest from the preoperative 3D image, said distorted 3D image being capable of representing the structure of interest under the constraint of at least one endovascular tool. The distorted 3D image is generated from an estimation of the deformation fields of the structure of interest under the constraint of the endovascular tool, said estimation being performed using an artificial intelligence module, said distorted 3D image being capable of being fused, during a fusion step, to a 2D image taken during the surgical procedure using an X-ray imaging device, said fusion step being achieved by 3D / 2D registration. The distorted 3D image is obtained from a finite element model of the structure of interest. Figure for the abridged version: Fig. 4

Description

Title of the invention: Medical image processing method for intraoperative assistance during a surgical procedure. Technical field

[0001] The present invention relates to a method for processing medical images for intraoperative assistance in a surgical procedure, a processing device, and an associated computer program product.

[0002] The treatment method finds application in the field of image-guided endovascular interventions. Previous Art

[0003] Endovascular interventions allow for the minimally invasive treatment of vascular diseases. They generally consist of inserting a medical device endovascularly in order to interact with the diseased tissues. Endovascular interventions are notably used to treat aortic aneurysms as well as arterial stenoses and thromboses, via the introduction of various adapted endovascular tools such as a balloon or a stent.

[0004] Unlike conventional surgical procedures that require large incisions in the patient's body to access the tissues of interest, endovascular procedures require only small incisions to insert the instruments into the vascular structure. They offer several advantages, including an increased short-term success rate, as well as a reduction in intraoperative morbidity and length of hospital stay. Despite the increasing use of these procedures, they remain delicate and require safety and reliability improvements. Access to pathological tissues is made difficult by the nature of the procedure. The manipulation and control of the instruments require significant precision to ensure the success of the treatment. Furthermore, monitoring of the surgical procedures can only be performed through intraoperative imaging.

[0005] It is known to use an endovascular tool such as a stiff guidewire for such surgical procedures. Such a stiff guidewire is in the form of a wire with a certain degree of rigidity, serving as a means of navigation within the patient's vascular system for endovascular instruments. Such stiff guidewires are notably discussed in the article "Stiff Guidewires in Endourology: What Is Stiffness," by Kolvatzis et al., Journal of Endourology, Volume 36, Number 11, November 2022. The purpose of the stiff guidewire is, for example, to facilitate the insertion of a stent delivery system.

[0006] The vascular system is composed of a set of arteries and veins forming hollow and elastic tubes, adapted to guide blood in the human body. These arteries and veins constitute soft tissues.

[0007] Thus, the introduction of a rigid guidewire into the patient's vascular system deforms these arteries and veins. The surgeon performing the surgical procedure must have a good understanding of the deformations of the vascular system in order to position the endovascular tool, for example a stent, as precisely as possible in the correct location within the vascular system.

[0008] For this purpose, it has at least one screen in the operating room displaying images representing the complexity of the patient's vascular system during the intervention phase.

[0009] It is known to use two-dimensional images (or 2D images) acquired by fluoroscopy on this screen. Fluoroscopy is a medical imaging technique that allows visualization of anatomical structures in motion and in real time. Since arteries are soft tissues and therefore not visible to X-rays, a radiopaque contrast agent can be administered to the patient to highlight the vascular structure by indicating the path of the arteries.

[0010] Although this technique is effective for visualizing the vascular system during the surgical procedure, the use of the contrast agent is not without side effects for the patient.

[0011] In order to improve the assistance provided during the operative phase, it has been proposed to also use three-dimensional image data (or 3D images), acquired during a preoperative or planning phase, obtained by acquisition techniques such as tomography also called CT for "computerized tomography", magnetic resonance imaging (MRI).

[0012] These 3D images are acquired before surgery for the diagnosis of the disease or to observe a particular pathology such as a type of aneurysm. They allow for preparation of the procedure. These 3D images can also be displayed in the operating room during the procedure.

[0013] These 3D images can, for example, be matched with the 2D images acquired during the intervention phase. This matching is performed using a 3D / 2D registration process that allows the different data to be expressed in the same spatial reference frame.

[0014] Figure 1 illustrates such a registration mechanism known in the prior art. In a first step (SI), three-dimensional image data (referred to as 3D images) of the vascular system are acquired during the preoperative phase by tomography. The vascular system can also be visualized using a contrast agent. The complete 3D data allows for a volumetric representation of the vascular system.

[0015] In a second step S2, the acquired three-dimensional image data are segmented. A geometric model of the vascular system is then obtained by segmentation into a plurality of structures of interest. Such a model can be easily edited and processed. Various algorithms are known for performing segmentation. The model can include centerlines (e.g., three-dimensional centerlines) and surface meshes (e.g., grids).

[0016] In a third step S3, at least one two-dimensional image (referred to as a 2D image) is acquired. The 2D image is acquired, for example, using an image acquisition device capable of rotating around the patient to acquire images, for example, using X-rays and contrast agents previously injected into the patient's body. Such an acquisition device is known as a motorized rotational C-arm and is used in rotational angiography. The 2D image includes the same details of the vascular system. The 2D image primarily shows the endovascular instrument.

[0017] In a fourth step S4, the 3D data set and the 2D image are superimposed, for example combined or crossed. During this superimposition, care is taken to ensure that the 2D image is located in the appropriate spatial and anatomical position of the 3D data set.

[0018] The objective is thus to visualize, simultaneously and on the same image, different types of information such as images of different modalities or previously extracted anatomical models. This provides the practitioner with additional information and a better understanding of the surgical field, enabling them to improve the precision of the intervention and make the surgical procedure safer.

[0019] However, due to the prior insertion of the rigid guidewire into the patient's body, the 3D images acquired before the procedure do not reflect the actual deformations of the vascular system at the time the surgeon uses the endovascular tool. This can lead to a risk of misinterpretation of the images displayed by the surgeon and therefore to errors in guiding the endovascular tool within the patient's vascular system.

[0020] There is therefore a need to further improve the guidance of an endovascular tool by a surgeon during a surgical procedure. Summary of the invention

[0021] The invention relates to a method for processing medical images for intraoperative assistance during a surgical procedure, said method comprising:

[0022] - the reception of at least one preoperative 3D image of a structure of interest of a vascular system at rest of a patient;

[0023] - a step of generating at least one distorted 3D image of the structure of interest from the preoperative 3D image, said deformed 3D image being able to represent the structure of interest under a constraint of at least one endovascular tool used for the surgical procedure;

[0024] - a step of displaying information generated from the deformed 3D image.

[0025] The deformed 3D image is generated from an estimation of the deformation fields of the structure of interest under the constraint of the endovascular tool, said estimation being carried out from an artificial intelligence module, said deformed 3D image being able to be fused with a 2D image of the structure of interest taken during the surgical procedure using an X-ray image acquisition device, said 2D image allowing visualization of a real position of the endovascular tool in the structure of interest at a time t, said fusion being obtained by a 3D / 2D registration.

[0026] The invention thus makes it possible to obtain a predictive and real-time estimate of vascular deformations induced by the introduction of the endovascular tool during surgical procedures. The use of an artificial intelligence module makes it possible to combine display accuracy with calculation times compatible with routine clinical practice (a few seconds).

[0027] In one embodiment, the distorted 3D image includes a plurality of anatomical points of interest of the patient defined on the structure of interest of the vascular system.

[0028] In one embodiment, the artificial intelligence module is a convolutional neural network.

[0029] In one embodiment, the artificial intelligence module is pre-trained from a training set, said training set comprising a plurality of pairs C;, each pair C; comprising an association of a preoperative 3D image with a deformed 3D image, said deformed 3D image being obtained from intraoperative 3D imaging after the introduction of the endovascular tool into the structure of interest during a previous surgical procedure.

[0030] In one embodiment, the deformed 3D image is obtained by CBCT.

[0031] In one embodiment, the artificial intelligence module is previously trained from a training set, said training set comprising a plurality of pairs Ci, each pair C; comprising an association of a pre-operational 3D image with a distorted 3D image, said 3D image deformed being obtained from a finite element modeling of the structure of interest from the image; pre-operational 3D.

[0032] In one embodiment, the artificial intelligence module is pre-trained from a training set, said training set comprising a plurality of pairs Ci, each pair Ci comprising an association of a preoperative 3D image with a deformed 3D image, said deformed 3D image being obtained from a plurality of 2D images of the structure of interest taken during a previous surgical procedure.

[0033] In one embodiment, the estimation of deformation fields of the structure of interest performed by the artificial intelligence module takes into account the rigidity of the endovascular tool.

[0034] In one embodiment, the estimation of deformation fields of the structure of interest carried out by the artificial intelligence module takes into account an elasticity of the structure of interest.

[0035] Another object of the invention relates to a device for implementing a medical image processing method for intraoperative assistance in a surgical procedure according to the invention.

[0036] Another object of the invention relates to a computer program comprising program code instructions for executing the steps of the medical image processing method for intraoperative assistance in a surgical procedure according to the invention. Description of the figures

[0037] Other features and advantages of the invention will become apparent upon reading the detailed description that follows, for an understanding of which reference should be made to the accompanying drawings in which:

[0038] Fig. 1 illustrates a 3D / 2D registration mechanism known according to the prior art;

[0039] Figure 2 illustrates a pre-operational 3D image of a structure of interest of a system vascular at rest in accordance with the prior art;

[0040] Fig. 3 illustrates a distorted 3D image of the structure of interest in Fig. 2 obtained according to the processing method of the invention;

[0041] Fig. 4 illustrates the steps of the processing method for obtaining the distorted 3D image of Fig. 3;

[0042] [Fig.5] illustrates display information generated from the distorted 3D image of [Fig.3];

[0043] Fig. 6 illustrates a mobile trolley comprising a display screen for displaying the information from Fig. 5;

[0044] Fig. 7 illustrates a processing device for implementing the processing method of Fig. 4;

[0045] Fig. 8 illustrates the learning phase of an artificial intelligence module belonging to the processing device of Fig. 7.

[0046] Fig. 2 illustrates a preoperative 3D image I3D(preop) of a structure of interest 1 of a vascular system, said vascular system being segmented into a plurality of structures of interest.

[0047] The structure of interest 1 here comprises a main artery 11, for example the aorta, connected to a first secondary artery 12, for example the right iliac artery, and a second secondary artery 13, for example the left iliac artery. Each artery 11, 12, 13 has, at rest, an external sheath comprising one or more curves.

[0048] The preoperative 3D image I3D(preop) is obtained for example by CT, MRI, CBCT (for "Cone Beam Computed Tomography" in English or "analysis numérique de l'absorption d'un rayon conique" in French) or 3D echo imaging.

[0049] Figure 3 illustrates a distorted 3D image I3D(def) of the structure of interest 1 when endovascular tools such as a first rigid guidewire 2 and a second rigid guidewire 3 are introduced into this structure 1. The first rigid guidewire 2 is adapted to extend into the first secondary artery 12 and the main artery 11. Similarly, the second rigid guidewire 3 is adapted to extend into the second secondary artery 13 and the main artery 11. The presence of the first rigid guidewire 2 and the second rigid guidewire 3 in the structure of interest 1 tends to stiffen the arteries 11, 12, and 13, thereby reducing the curvature of their outer walls. The distorted 3D image I3D(def) thus represents the structure of interest 1 under the stress of the rigid guidewire.

[0050] In another embodiment, the structure of interest is an organ of the patient, such as a collateral artery, for example the renal artery.

[0051] In another embodiment, the deformed 3D image I3D(def) further includes a plurality of anatomical points of interest defined on the structure of interest, for example ostia of a renal artery or a hypogastric artery.

[0052] Fig. 4 illustrates the processing method for obtaining the distorted 3D image of Fig. 3.

[0053] This processing method includes a first step of receiving El at least one pre-operative 3D image I3D(preop) of the structure of interest 1 at rest.

[0054] In a second generation step E2, a distorted 3D image I3D(def) of the vascular system is obtained from the preoperative 3D image I3D(preop). This distorted 3D image is generated from an estimation of the deformation fields of the The vascular system is subjected to the stress of rigid guidewires. To achieve this, the second generation step, E2, includes a first substep, E21, which transmits the preoperative 3D image (I3D(preop)) to an artificial intelligence module. In a second substep, E22, the artificial intelligence module analyzes the transmitted image to obtain specific analysis criteria. For example, one analysis criterion is the identification of the position of structure of interest 1 within the human body. A second criterion is the elasticity of structure of interest 1. This elasticity is deduced, for example, from the presence of calcifications on the artery walls. A high density of calcifications indicates greater artery rigidity. The response of the arteries to the passage of the rigid guidewire thus varies according to their rigidity.Other criteria that cannot be directly deduced from the preoperative 3D image I3D(preop) can also be transmitted to the artificial intelligence module, such as the patient's age, previous pathologies, characteristics of rigid guides, etc. In a third sub-step E23, the artificial intelligence module generates a distorted 3D image.

[0055] In a third step E3, information is displayed. This information is generated from the distorted 3D image.

[0056] In a particular embodiment, the displayed information is a combination of data. Figure 5 illustrates such an example of a data combination. The displayed image thus includes an external delimitation of the main artery 11, the first secondary artery 12, and the second secondary artery 13, which are under the constraint of the first rigid guide 2 and the second rigid guide 3.

[0057] A first part of the information is provided by the deformed 3D image I3D(def).

[0058] The display also includes the position of the first rigid guide 2 and the position of the second rigid guide 3 at a time t. A second part of information is then provided by a 2D image of the vascular system taken during the surgical procedure.

[0059] The displayed image is thus formed from the deformed 3D image I3D(def) and the 2D image of the vascular system which are matched by a 3D / 2D registration.

[0060] It will be noted that the resulting displayed image of the deformed 3D image I3D(def) and the 2D image of the vascular system is, itself, a 2D image.

[0061] Such a 2D image is, for example, displayed on a screen 5 of a mobile cart 6, as illustrated in [Fig.6]. This mobile trolley is suitable for moving around the operating table and for visually providing information to the surgeon.

[0062] Fig. 7 illustrates a treatment device 7 for implementing the treatment process of Fig. 4.

[0063] This treatment device 7 comprises:

[0064] - an input / output module 71;

[0065] - a memory 72;

[0066] - a 73 processor;

[0067] - an artificial intelligence module 74.

[0068] The input / output module 71 is adapted to receive at least one preoperative 3D image I3D(preop) of a patient's resting vascular system 1. This module 71 is also adapted to transmit a distorted 3D image I3D(def) of the vascular system generated from the preoperative 3D image.

[0069] Memory 72 is adapted to store images such as the pre-operative 3D image I3D(preop) and the deformed 3D image I3D(def).

[0070] The processor 73 is adapted to obtain the distorted 3D image I3D(def). To do this, the processor 73 queries the artificial intelligence module 74. In return, the artificial intelligence module 74 provides it with the distorted 3D image I3D(def). The artificial intelligence module 74 thus makes it possible to estimate deformation fields x, y, z through inference by the neural network on the image I3D(preop). These deformation fields are applied directly to the image I3D(preop) to provide the image I3D(def). These deformation fields can also be applied to the organ mesh obtained through prior segmentation and / or to anatomical points of interest in the patient's body.

[0071] Artificial intelligence module 74 is here a convolutional neural network. By "convolutional neural network," we mean a network comprising two types of artificial neurons arranged in layers that successively process information. We thus distinguish between processing neurons that process a limited portion of the image (called the receptive field) through a convolution function, and output pooling neurons (total or partial). Non-linear, spot-point corrective processing can be applied between each layer to improve the relevance of the result. The set of outputs from a processing layer allows the reconstruction of an intermediate image that serves as the basis for the next layer.

[0072] As illustrated in [Fig. 8], the artificial intelligence module 74 is pre-trained from a training set 8. This training set 8 comprises a plurality of pairs Q, each pair Q comprising an association of a pre-operational 3D image with a deformed 3D image. In one embodiment, the deformed 3D image is obtained from a 3D image Intraoperatively, after the introduction of the rigid guidewire 2, 3 into the vascular system. This distorted 3D image is obtained, for example, by CT, MRI, CBCT, or 3D ultrasound imaging. Non-rigid registration can be used between the preoperative 3D image and the distorted 3D image.

[0073] In another embodiment, the deformed 3D image is obtained from a finite element model (FEM for "Finite Element Method") of the vascular system of the preoperative 3D image. This model allows for the analytical representation of the dynamic behavior of the structure of interest. More specifically, the structure of interest is defined using a mesh. This mesh defines a tiling whose tiles are the finite elements. On each of these tiles, it is possible to perform a biomechanical simulation. Strain fields are then calculated by matching.

[0074] In another embodiment, the distorted 3D image is obtained from a plurality of 2D images of the structure of interest taken during the surgical procedure. These 2D images are combined to constitute the 3D image. In this embodiment, angiography (X-ray imaging with injection of a contrast agent) would be used to visualize the arteries and allow for a 3D reconstruction of the vascular structure.

[0075] The input / output module 71 can be adapted to receive a modification instruction. The memory 72 is adapted to store this modification instruction. The processor 73 is adapted to transmit this modification instruction to the artificial intelligence module 74. In return, this module 74 will provide a new distorted 3D image while integrating this modification instruction into the continuous learning of the neural network.

[0076] The treatment device 7 is installed in the mobile trolley 6 for intraoperative assistance during the surgical procedure.

[0077] As can be seen in [Fig.6], the mobile trolley 6 includes means 61 for movement, a storage area 62 for at least one processing unit 63, a fixing column 64, a secondary screen 65.

[0078] The movement means 61 enable the movement of the mobile trolley 6 within the operating room. These means 51 comprise a plurality of swivel casters. These casters are positioned in arms that are centrally connected. At least two of these arms define the storage area 62. This storage area 62 is adapted to accommodate at least one treatment unit 63. The column 64 extends vertically from the arms. It allows the screen 5 and the secondary screen 65 to be attached to the mobile trolley 6. In the embodiment shown in [Fig. 6], the screen 5 is attached to the end of the mounting column 64 and the screen The secondary screen is attached to a side section of column 64 via an articulated arm. Note that the secondary screen 65 is designed to display some of the information from screen 5 as well as additional data.

[0079] The invention also relates to a computer program product comprising program code instructions for executing steps E1 to E3 of the medical image processing method.

[0080] Such a computer program is, for example, stored in the processing unit 63 of the mobile cart 6.

Claims

Demands

1. A method for processing medical images, said method comprising: - receiving (E1) at least one preoperative 3D image (I3D(preop)) of a structure of interest (1) of a patient's resting vascular system; - a step (E2) of generating at least one distorted 3D image (I3D(def)) of the structure of interest (1) from the preoperative 3D image, said distorted 3D image being capable of representing the structure of interest under the constraint of at least one endovascular tool (2, 3); - a step (E3) of displaying information generated from the distorted 3D image;characterized in that the deformed 3D image (I3D(def)) is generated from an estimation of the deformation fields of the structure of interest under the constraint of the endovascular tool (2, 3), said estimation being carried out from an artificial intelligence module (74), said deformed 3D image being able to be fused, during a fusion step, with a 2D image taken during the surgical procedure using an X-ray image acquisition device, said 2D image allowing visualization of a real position of the endovascular tool (2,3) in the vascular system at a time t, said fusion step being obtained by a 3D / 2D registration.;

2. A processing method according to claim 1, wherein the distorted 3D image comprises a plurality of anatomical points of interest of the patient defined on the structure of interest.

3. Processing method according to any one of claims 1 to 2, wherein the artificial intelligence module (74) is a convolutional neural network.

4. A processing method according to any one of claims 1 to 3, wherein the artificial intelligence module (74) is pre-trained from a training set, said training set comprising a plurality of pairs C, each pair Ci comprising an association of a preoperative 3D image with a deformed 3D image, said deformed 3D image being obtained from intraoperative 3D imaging after the introduction of the endovascular tool (2, 3) into the structure of interest during a previous surgical procedure.

5. Processing method according to claim 4, wherein the distorted 3D image is obtained by CBCT.

6. Processing method according to any one of claims 1 to 3, wherein the artificial intelligence module (74) is pre-trained from a training set, said training set comprising a plurality of pairs Ci, each pair Ci comprising an association of a pre-operational 3D image with a deformed 3D image, said deformed 3D image being obtained from a finite element modeling of the structure of interest from the pre-operational 3D image.

7. Processing method according to any one of claims 1 to 3, wherein the artificial intelligence module (74) is pre-trained from a training set, said training set comprising a plurality of pairs C;, each pair Ci comprising an association of a preoperative 3D image with a deformed 3D image, said deformed 3D image being obtained from a plurality of 2D images of the structure of interest taken during a previous surgical procedure.

8. A treatment method according to any one of claims 1 to 7, wherein the estimation of deformation fields of the structure of interest carried out by the artificial intelligence module takes into account a rigidity of the endovascular tool (2, 3).

9. Processing method according to any one of claims 1 to 8, wherein the estimation of deformation fields of the structure of interest carried out by the artificial intelligence module (74) takes into account an elasticity of the structure of interest (1).

10. A processing device comprising means (71, 72, 73, 74) for implementing a medical image processing method according to any one of claims 1 to 9, said method comprising the following steps: - receiving (E1) at least one preoperative 3D image (I3D(preop)) of a structure of interest (1) of a patient's resting vascular system; - a step (E2) of generating at least one distorted 3D image (I3D(def)) of the structure of interest (1) from the 3D image preoperative, said deformed 3D image being able to represent the structure of interest under a constraint of at least one endovascular tool (2, 3); - a step (E3) of displaying information generated from the deformed 3D image; characterized in that the deformed 3D image (I3D(def)) is generated from an estimation of deformation fields of the structure of interest under the constraint of the endovascular tool (2, 3), said estimation being carried out from an artificial intelligence module (74), said deformed 3D image being able to be fused, during a fusion step, with a 2D image taken during the surgical procedure using an X-ray image acquisition device, said 2D image allowing visualization of a real position of the endovascular tool (2,3) in the vascular system at a time t, said fusion step being obtained by a 3D / 2D registration.

11. Product computer program comprising program code instructions for performing the steps of a medical image processing method according to any one of claims 1 to 9, when said program is executed by a processor.

Images