Method for processing medical images for intraoperative assistance in a surgical procedure

Abstract

The invention relates to a method for processing medical images, the method comprising a step (E2) of generating at least one deformed 3D image of the structure of interest from the pre-operative 3D image, the deformed 3D image being able to represent the structure of interest under a constraint of at least one endovascular tool. The deformed 3D image is generated from an estimation of deformation fields of the structure of interest under the constraint of the endovascular tool, the estimation being carried out from an artificial intelligence module, the deformed 3D image being able to be merged, during a merging step, with a 2D image taken during the surgical procedure using an X-ray image acquisition device, the merging step being obtained by 3D / 2D registration. The deformed 3D image is obtained from finite element modelling of the structure of interest.

Description

DESCRIPTION Title of the invention: Method for processing medical images for intraoperative assistance during a surgical procedure

[0001] technical field

[0002] The present invention relates to a method for processing medical images for intraoperative assistance in a surgical procedure, a processing device, and an associated computer program product.

[0003] The treatment process finds application in the field of image-guided endovascular interventions.

[0004] Previous Art

[0005] Endovascular procedures allow for the minimally invasive treatment of vascular diseases. They generally involve the insertion of a medical device through an endovascular route to interact with the diseased tissues. Endovascular procedures are notably used to treat aortic aneurysms, as well as arterial stenoses and thromboses, through the introduction of various adapted endovascular tools such as balloons or stents.

[0006] Unlike traditional surgical procedures that require large incisions to access the tissues of interest, endovascular interventions require only small incisions to insert instruments into the vascular structure. They offer several advantages, including increased short-term success rates, reduced intraoperative morbidity, and shorter hospital stays. Despite the increasing prevalence of these procedures, they remain delicate and require safety and reliability improvements. Access to pathological tissues is challenging due to the nature of the intervention. The manipulation and control of instruments demand significant precision to ensure treatment success. Furthermore, monitoring of the surgical procedure can only be performed through intraoperative imaging.

[0007] It is known to use an endovascular tool such as a rigid guidewire for such surgical procedures. Such a rigid guidewire is a wire with a certain degree of rigidity that serves as a means of Navigation within the patient's vascular system is facilitated by other endovascular tools, such as stents. Rigid guidewires are discussed in detail in the article "Stiff Guidewires in Endourology: What Is Stiffness," by Kolvatzis et al., *Journal of Endourology*, Volume 36, Number 11, November 2022. The rigid guidewire is intended, for example, to facilitate the insertion of a stent delivery system.

[0008] The vascular system is composed of a network of arteries and veins forming hollow, elastic tubes adapted to guide blood throughout the human body. These arteries and veins are made of soft tissue.

[0009] Thus, the introduction of a rigid guidewire into the patient's vascular system deforms these arteries and veins. The surgeon performing the procedure must have a thorough understanding of these vascular deformities in order to position the endovascular tool, such as a stent, as precisely as possible in the correct location within the vascular system.

[0010] For this purpose, it has at least one screen in the operating room displaying images representing the complexity of the patient's vascular system during the intervention phase.

[0011] This screen is known to display two-dimensional (or 2D) images acquired by fluoroscopy. Fluoroscopy is a medical imaging technique that allows visualization of anatomical structures in motion and in real time. Since arteries are soft tissues and therefore not visible on X-rays, a radiopaque contrast agent can be administered to the patient to highlight the vascular structure and trace the path of the arteries.

[0012] Although this technique is effective for visualizing the vascular system during surgery, the use of contrast agents is not without side effects for the patient.

[0013] In order to improve the assistance provided during the operative phase, it has been proposed to also use three-dimensional image data (or 3D images), acquired during a preoperative or planning phase, obtained by acquisition techniques such as tomography also called CT for "computerized tomography", magnetic resonance imaging (MRI).

[0014] These 3D images are acquired before surgery for the diagnosis of the disease or to observe a particular pathology such as a form of aneurysm. They allow for preparation of the procedure. These 3D images can also be displayed in the operating room during the procedure.

[0015] These 3D images can, for example, be matched with the 2D images acquired during the intervention phase. This matching is performed using a 3D / 2D registration process that allows the different data to be expressed in the same spatial reference frame.

[0016] Figure 1 illustrates such a registration mechanism known from prior art. In a first step S1, three-dimensional image data (referred to as 3D images) of the vascular system are acquired during the preoperative phase by computed tomography (CT). The vascular system can also be visualized using a contrast agent. The complete set of 3D data allows for a volumetric representation of the vascular system.

[0017] In a second step, S2, the acquired three-dimensional image data is segmented. A geometric model of the vascular system is then obtained by segmenting it into a plurality of structures of interest. Such a model can be easily edited and processed. Several algorithms are known for performing segmentation. The model can include centerlines (e.g., three-dimensional centerlines) and surface meshes (e.g., grids).

[0018] In a third step, S3, at least one two-dimensional image (called a 2D image) is acquired. The 2D image is acquired, for example, using an image acquisition device capable of rotating around the patient to acquire images, such as X-rays and contrast agents previously injected into the patient's body. Such an acquisition device is known as a motorized rotational C-arm and is used in rotational angiography. The 2D image includes the same details of the vascular system. The 2D image primarily shows the endovascular instrument.

[0019] In a fourth step, S4, the 3D data set and the 2D image are superimposed, for example, combined or crossed. During this superimposition, care is taken to ensure that the 2D image is in the appropriate spatial and anatomical position within the 3D data set.

[0020] The aim is to visualize, simultaneously and on the same image, different types of information such as images from different modalities or previously extracted anatomical models. This provides the practitioner with additional information and a better understanding of the surgical field, allowing them to improve the precision of the procedure and enhance the safety of the surgical technique.

[0021] However, because the rigid guidewire is inserted into the patient's body beforehand, the 3D images acquired prior to the procedure do not accurately reflect the deformations of the vascular system at the time the surgeon uses the endovascular tool. This can lead to misinterpretation of the images displayed by the surgeon and, consequently, errors in guiding the endovascular tool within the patient's vascular system.

[0022] Therefore, there is a need to further improve the guidance of an endovascular tool by a surgeon during a surgical procedure.

[0023] Summary of the invention

[0024] The invention relates to a method for processing medical images for intraoperative assistance during a surgical procedure, said method comprising: - the receipt of at least one preoperative 3D image of a structure of interest of a patient's resting vascular system; - a step of generating at least one distorted 3D image of the structure of interest from the preoperative 3D image, said distorted 3D image being capable of representing the structure of interest under a constraint of at least one endovascular tool used for the surgical procedure; - a step of displaying information generated from the deformed 3D image. The deformed 3D image is generated from an estimation of the deformation fields of the structure of interest under the constraint of the endovascular tool, said estimation being carried out using an artificial intelligence module previously trained on a training dataset, said training dataset comprising a plurality of pairs Ci, each pair Ci comprising an association of a preoperative 3D image with a deformed 3D image, the deformed 3D image being able to be fused with a 2D image of the structure of interest taken during the procedure surgical using an X-ray image acquisition device, said 2D image allowing visualization of a real position of the endovascular tool in the structure of interest at a time t, said fusion being obtained by 3D / 2D registration.

[0025] The invention thus enables predictive, real-time estimation of vascular deformations induced by the introduction of the endovascular tool during surgical procedures. The use of an artificial intelligence module allows for a combination of display accuracy and computation times compatible with routine clinical practice (a few seconds).

[0026] In one embodiment variant, the distorted 3D image includes a plurality of anatomical points of interest of the patient defined on the structure of interest of the vascular system.

[0027] In one embodiment variant, the artificial intelligence module is a convolutional neural network.

[0028] In one embodiment, the deformed 3D image is obtained from intraoperative 3D imaging after the introduction of the endovascular tool into the structure of interest during a previous surgical procedure.

[0029] In one embodiment variant, the deformed 3D image is obtained by CBCT.

[0030] In one embodiment variant, the deformed 3D image is obtained from a finite element modeling of the structure of interest from the pre-operational 3D image.

[0031] In one embodiment, the deformed 3D image is obtained from a plurality of 2D images of the structure of interest taken during a previous surgical procedure.

[0032] In one embodiment variant, the deformed 3D image generation step is capable of handling rigidity of the endovascular tool.

[0033] In one embodiment variant, the deformed 3D image generation step is capable of handling elasticity of the structure of interest.

[0034] In one embodiment, during the generation stage, a plurality of distorted 3D images are generated from different artificial intelligence modules, with a user able to select a 3D image. distorted among the plurality of distorted 3D images for the fusion of said distorted 3D image with the 2D image.

[0035] Another object of the invention relates to a processing device comprising an input / output module, a memory, a processor, an artificial intelligence module, said device being capable of implementing a medical image processing method, the method of the invention, said method comprising a step of receiving at least one preoperative 3D image of a structure of interest of a patient's resting vascular system, a step of generating at least one deformed 3D image of the structure of interest from the preoperative 3D image, said deformed 3D image being capable of representing the structure of interest under a constraint of at least one endovascular tool, a step of displaying information generated from the deformed 3D image.The deformed 3D image is generated from an estimation of the deformation fields of the structure of interest under the constraint of the endovascular tool, said estimation being carried out from the artificial intelligence module, said model being previously trained from a training set, said training set comprising a plurality of pairs Ci, each pair Ci comprising an association of a preoperative 3D imagei with a deformed 3D imagei, the deformed 3D image being able to be fused, during a fusion step, with a 2D image taken during the surgical procedure using an X-ray image acquisition device, said 2D image allowing visualization of a real position of the endovascular tool in the vascular system at a time t, said fusion step being obtained by a 3D / 2D registration.

[0036] Another object of the invention relates to a computer program product comprising program code instructions for executing the steps of the medical image processing method according to the invention when said program is executed by a processor.

[0037] Description of the figures

[0038] Other features and advantages of the invention will become apparent upon reading the detailed description that follows, for which reference should be made to the accompanying drawings in which:

[0039] Figure 1 illustrates a 3D / 2D registration mechanism known from the prior art;

[0040] Figure 2 illustrates a preoperative 3D image of a structure of interest of a vascular system at rest in accordance with the prior art;

[0041] Figure 3 illustrates a distorted 3D image of the structure of interest in Figure 2 obtained according to the processing method of the invention;

[0042] Figure 4 illustrates the steps in the processing procedure to obtain the distorted 3D image in Figure 3;

[0043] Figure 5 illustrates display information generated from the distorted 3D image of Figure 3;

[0044] Figure 6 illustrates a mobile cart with a display screen for displaying the information from Figure 5;

[0045] Figure 7 illustrates a processing device for implementing the processing method of Figure 4;

[0046] Figure 8 illustrates the learning phase of an artificial intelligence module belonging to the processing device in Figure 7.

[0047] Figure 2 illustrates a preoperative 3D image IsD(preop) of a structure of interest 1 of a vascular system, said vascular system being segmented into a plurality of structures of interest.

[0048] The structure of interest 1 here comprises a main artery 11, for example the aorta, connected to a first secondary artery 12, for example the right iliac artery, and a second secondary artery 13, for example the left iliac artery. Each artery 11, 12, 13 has, at rest, an external sheath with one or more curves.

[0049] The preoperative 3D image IsD(preop) is obtained for example by CT, MRI, CBCT (for "Cone Beam Computed Tomography" in English or "analysis numérique de l'absorption d'un rayon conique" in French) or 3D echo imaging.

[0050] Figure 3 illustrates a distorted 3D image IsD(def) of the structure of interest 1 when endovascular tools such as a first rigid guidewire 2 and a second rigid guidewire 3 are introduced into this structure 1. The first rigid guidewire 2 is here adapted to extend into the first secondary artery 12 and into the main artery 11. Similarly, the second rigid guidewire 3 is adapted to extend into the second secondary artery 13 and the main artery 11. The presence of the first rigid guidewire 2 and the second rigid guidewire 3 in the structure of interest 1 tends to stiffen arteries 11, 12, and 13, thereby reducing the curvatures of their outer layers. The deformed 3D image IsD(def) thus represents the structure of interest 1 under the constraint of the rigid guidewire.

[0051] In another embodiment, the structure of interest is a patient organ, such as a collateral artery, for example the renal artery.

[0052] In another embodiment, the deformed 3D image IsD(def) further includes a plurality of anatomical points of interest defined on the structure of interest, for example ostia of a renal artery or a hypogastric artery.

[0053] In another embodiment, the treatment method facilitates monitoring the positioning of a stent along a rigid guidewire in an artery. This also allows the practitioner to avoid stent misplacement, particularly at a collateral vessel, that is, a vein that runs parallel to the course of an artery.

[0054] Figure 4 illustrates the processing method for obtaining the distorted 3D image of figure 3.

[0055] This processing procedure includes a first reception step E1 of at least one preoperative 3D image IsD(preop) of the structure of interest 1 at rest.

[0056] In a second generation step E2, a deformed 3D image IsD(def) of the vascular system is obtained from the preoperative 3D image IsD(preop). This deformed 3D image is generated from an estimation of the deformation fields of the vascular system under the stress of rigid guides. To achieve this, this second generation step E2 includes a first substep E21 of transmitting the preoperative 3D image IsD(preop) to an artificial intelligence module. In a second substep E22, the artificial intelligence module analyzes the transmitted image to obtain certain analysis criteria. For example, a first analysis criterion corresponds to the identification of the position of structure of interest 1 in the human body. A second criterion corresponds to the elasticity of structure of interest 1. Such elasticity is, for example, deduced from the presence of calcifications on the arterial walls.A high density of calcification is an indicator of a higher level of calcification. The arteries exhibit significant rigidity. Their response to the passage of the rigid guidewire therefore varies depending on their rigidity. Other criteria not directly derived from the preoperative 3D image (IsD(preop)) can also be transmitted to the artificial intelligence module, such as the patient's age, prior medical history, and the characteristics of the rigid guidewires. In a third substep (E23), the artificial intelligence module generates a distorted 3D image.

[0057] In a third step E3, information is displayed. This information is generated from the distorted 3D image.

[0058] In one particular embodiment, the displayed information is a combination of data. Figure 5 illustrates such an example of data combination. The displayed image thus includes an external delineation of the main artery 11, the first secondary artery 12, and the second secondary artery 13, which are under the constraint of the first rigid guide 2 and the second rigid guide 3.

[0059] The first part of the information is provided by the distorted 3D image l3 D (def).

[0060] The display also includes the position of the first rigid guide 2 and the position of the second rigid guide 3 at a time t. A second piece of information is then provided by a 2D image of the vascular system taken during the surgical procedure.

[0061] The displayed image is thus formed from the deformed 3D image IsD(def) and the 2D image of the vascular system which are matched by a 3D / 2D registration.

[0062] Note that the resulting displayed image of the deformed 3D image IsD(def) and the 2D image of the vascular system is itself a 2D image.

[0063] Such a 2D image is for example displayed on a screen 5 of a mobile trolley 6, as illustrated in Figure 6. This mobile trolley is adapted to move around the operating table and to visually provide information to the surgeon.

[0064] It should be noted that it is also possible to use different artificial intelligence modules to generate various distorted 3D ISD images. Thus, a first module Artificial intelligence can be specialized in estimating deformations of a structure of interest present on one side of the patient. A second artificial intelligence module can be specialized in estimating deformations of a structure of interest present on a second side of the patient. Finally, a third artificial intelligence module can be specialized in estimating deformations of a structure of interest extending across both the first and second sides of the patient. The various deformed 3D images (ISDs) are determined prior to the operation during step E2.

[0065] In one particular embodiment, it is the practitioner who will choose the type of distorted 3D image that he will want to use for the fusion step from among the distorted 3D images generated by the different artificial intelligence modules.

[0066] Figure 7 illustrates a processing device 7 for implementing the processing method of Figure 4.

[0067] This treatment device 7 includes:

[0068] - an input / output module 71;

[0069] - a memory 72;

[0070] - a 73 processor;

[0071] - an artificial intelligence module 74.

[0072] The input / output module 71 is adapted to receive at least one preoperative 3D image IsD(preop) of a patient's resting vascular system 1. This module 71 is also adapted to transmit a deformed 3D image IsD(def) of the vascular system generated from the preoperative 3D image.

[0073] Memory 72 is suitable for storing images such as the preoperative 3D image IsD(preop) and the deformed 3D image IsD(def).

[0074] Processor 73 is adapted to obtain the deformed 3D image IsD(def). To do this, processor 73 queries the artificial intelligence module 74. In return, the artificial intelligence module 74 provides it with the deformed 3D image IsD(def). The artificial intelligence module 74 then estimates deformation fields x, y, z through neural network inference on the IsD(preop) image. These deformation fields are applied directly to the IsD(preop) image to provide the IsD(def) image. These deformation fields can also be applied to the mesh. of organs obtained through prior segmentation and / or on points of anatomical interest of the patient's body.

[0075] Artificial intelligence module 74 is a convolutional neural network. A convolutional neural network is a network comprising two types of artificial neurons arranged in layers that successively process information. These include processing neurons that handle a limited portion of the image (called the receptive field) using a convolution function, and output pooling neurons (which pool the outputs, either partially or completely). Non-linear, spot-point corrective processing can be applied between each layer to improve the accuracy of the result. The combined outputs of a processing layer are used to reconstruct an intermediate image, which then serves as the basis for the next layer.

[0076] As illustrated in Figure 8, the artificial intelligence module 74 is pre-trained using a training set 8. This training set 8 comprises a plurality of pairs Ci, each pair Ci consisting of an association of a preoperative 3D image with a distorted 3D image. In one embodiment, the distorted 3D image is obtained from intraoperative 3D imaging after the introduction of the rigid guidewire 2, 3 into the vascular system. This distorted 3D image is obtained, for example, by CT, MRI, CBCT, or 3D ultrasound imaging. Non-rigid registration can be used between the preoperative 3D image and the distorted 3D image.

[0077] In another embodiment, the deformed 3D image is obtained from a finite element model (FEM for "Finite Element Method") of the vascular system of the preoperative 3D image. This modeling allows for the analytical representation of the dynamic behavior of the structure of interest. More specifically, the structure of interest is defined using a mesh. This mesh defines a tiling whose tiles are the finite elements. A biomechanical simulation can be performed on each of these tiles. Strain fields are then calculated by matching.

[0078] In another embodiment, the distorted 3D image is obtained from a plurality of 2D images of the structure of interest taken during the surgical procedure. These 2D images are combined to constitute the 3D image. In this embodiment, angiographies (X-ray imaging with injection of contrast agent) are used. contrast) would be used to visualize the arteries and allow a 3D reconstruction of the vascular structure.

[0079] In another embodiment, the artificial intelligence module 74 is trained from data obtained via intraoperative 3D imaging during a previous surgical procedure and / or from data obtained via finite element modeling of the structure of interest from preoperative 3D imaging and / or from data via a plurality of 2D images of the structure of interest taken during a previous surgical procedure.

[0080] Input / output module 71 can be adapted to receive a modification instruction. Memory 72 is adapted to store this modification instruction. Processor 73 is adapted to transmit this modification instruction to the artificial intelligence module 74. In turn, this module 74 will provide a new, distorted 3D image while integrating this modification instruction into the continuous learning of the neural network.

[0081] The treatment device 7 is installed in the mobile trolley 6 for intraoperative assistance during the surgical procedure.

[0082] As can be seen in Figure 6, the mobile trolley 6 includes means 61 for movement, a storage area 62 for at least one processing unit 63, a fixing column 64, a secondary screen 65.

[0083] The movement means 61 enable the mobile trolley 6 to move within the operating room. These means 51 comprise a plurality of swivel casters. These casters are positioned in arms that are centrally connected. At least two of these arms define the storage area 62. This storage area 62 is designed to accommodate at least one processing unit 63. The column 64 extends vertically from the arms. It allows the screen 5 and the secondary screen 65 to be attached to the mobile trolley 6. In the embodiment shown in Figure 6, the screen 5 is fixed to the end of the mounting column 64, and the secondary screen is attached to a lateral portion of the column 64 via an articulated arm. Note that the secondary screen 65 is designed to display some of the information from the screen 5 as well as additional data.

[0084] The invention also relates to a computer program product comprising program code instructions for executing steps E1 to E3 of the medical image processing method.

[0085] Such a computer program is, for example, stored in the processing unit 63 of the mobile cart 6.

Claims

DEMANDS 1. A method for processing medical images, said method comprising: - the reception (E1) of at least one preoperative 3D image (IsD(preop)) of a structure of interest (1) of a patient's resting vascular system; - a step (E2) of generating at least one deformed 3D image (IsD(def)) of the structure of interest (1) from the preoperative 3D image, said deformed 3D image being able to represent the structure of interest under a constraint of at least one endovascular tool (2, 3); - a step (E3) of displaying information generated from the distorted 3D image;characterized in that the deformed 3D image (IsD(def)) is generated from an estimation of the deformation fields of the structure of interest under the constraint of the endovascular tool (2, 3), said estimation being carried out from an artificial intelligence module (74) previously trained from a training set, said training set comprising a plurality of pairs Ci, each pair Ci comprising an association of a preoperative 3D image with a deformed 3D image, the deformed 3D image (IsD(def)) being able to be fused, during a fusion step, with a 2D image taken during the surgical procedure using an X-ray image acquisition device, said 2D image allowing visualization of a real position of the endovascular tool (2,3) in the vascular system at a time t, said fusion step being obtained by a 3D / 2D registration.; 2. Processing method according to claim 1, wherein the distorted 3D image comprises a plurality of anatomical points of interest of the patient defined on the structure of interest.

3. Processing method according to any one of claims 1 to 2, wherein the artificial intelligence module (74) is a convolutional neural network.

4. Processing method according to any one of claims 1 to 3, wherein said distorted 3D image is obtained from an intraoperative 3D image after the introduction of the endovascular tool (2, 3) into the structure of interest during a previous surgical procedure.

5. Processing method according to claim 4, wherein the deformed 3D image is obtained by CBCT.

6. Processing method according to any one of claims 1 to 5, wherein said deformed 3D image is obtained from a finite element modeling of the structure of interest from the pre-operational 3D image.

7. Processing method according to any one of claims 1 to 6, wherein said distorted 3D image is obtained from a plurality of 2D images of the structure of interest taken during a previous surgical procedure.

8. Processing method according to any one of claims 1 to 7, wherein the step (E2) of generating the deformed 3D image (IsD(def)) is capable of treating a stiffness of the endovascular tool (2, 3).

9. Processing method according to any one of claims 1 to 8, wherein the step (E2) of generating the deformed 3D image (IsD(def)) is capable of processing an elasticity of the structure of interest (1).

10. Processing method according to any one of claims 1 to 9, wherein during the generation step (E2), a plurality of distorted 3D images (IsD(def)) are generated from different artificial intelligence modules, a user being able to select a distorted 3D image from the plurality of distorted 3D images for the fusion of said distorted 3D image (IsD(def)) with the 2D image.

11. Processing device comprising an input / output module (71), a memory (72), a processor (73), an artificial intelligence module (74), said device being capable of implementing a medical image processing method according to any one of claims 1 to 10, said method comprising the following steps: - the reception (E1) of at least one preoperative 3D image (IsD(preop)) of a structure of interest (1) of a patient's resting vascular system; - a step (E2) of generating at least one distorted 3D image (IsD(def)) of the structure of interest (1) from the pre-operational 3D image, said distorted 3D image being able to represent the structure of interest under a constraint of at least one endovascular tool (2, 3); - a step (E3) of displaying information generated from the distorted 3D image;characterized in that the deformed 3D image (IsD(def)) is generated from an estimation of the deformation fields of the structure of interest under the constraint of the endovascular tool (2, 3), said estimation being performed using the artificial intelligence module (74), said model (74) being previously trained from a training set, said training set comprising a plurality of pairs Ci, each pair Ci comprising an association of a preoperative 3D image with a deformed 3D image, the deformed 3D image (IsD(def)) being capable of being fused, during a fusion step, with a 2D image taken during the surgical procedure using an X-ray imaging device, said 2D image allowing visualization of a real position of the endovascular tool (2, 3) in the vascular system at time t, said fusion step being obtained by registration 3D / 2D.; 12. Product computer program comprising program code instructions for executing the steps of a medical image processing method according to any one of claims 1 to 10, when said program is executed by a processor.

Images