SYSTEM AND METHOD FOR PROVIDING VIRTUAL 3D MODEL ALIGNMENT THROUGH TRANSPARENT DISPLAY - Patent application

Abstract

The present invention relates to a method for bringing a virtual three-dimensional model, called a 3D model, of a target organ into alignment with at least one image of said target organ in a scene being acquired by an endoscope (12), the method comprising, inter alia, the steps of predicting the position and orientation of the target organ relative to the scene, superimposing on a display device (18) a projection of a semi-transparent 3D model based on the predicted position and orientation onto at least one current image from the endoscope, accepting a command indicating alignment between the semi-transparent projection of the 3D model and an image of the target organ in the current image, and calculating the position and orientation of the target organ in the current image relative to the scene.

Description

[Technical Field]

[0001] The present invention relates to a system and method for the registration of a 3D model on an optical image. In particular, the present invention relates to the registration of a virtual 3D model of an organ obtained from preoperative imaging to an image of the organ obtained by optical imaging, especially stereoscopic or monocular endoscopic imaging. The present invention can be used in particular in laparoscopic imaging and robotics contexts. [Background technology]

[0002] Registration, also called "alignment," is a task that allows achieving a match in terms of position and orientation between a virtual 3D model of an object and an optical image of the same object acquired by a camera. In particular, the goal is to determine the transformation (change and deformation of the coordinate frame) that must be achieved so that the 3D model and the image of the object, which have distinct coordinate systems and distinct states, are aligned. In fact, knowledge of this transformation from registration allows a virtual 3D model from precomputed data to be displayed in an augmented reality context on an image representing the real world acquired by a camera. The goal is then to ensure tracking of the object on the image after registration in order to display the 3D model on it.

[0003] In particular, methods for registration are implemented in the medical and / or surgical field to map a 3D model, called a preoperative model since it is obtained upstream from one or more medical imaging techniques (e.g., fluoroscopy, ultrasound, MRI, CT, etc.), onto a real image obtained by an optical camera, e.g., an endoscope.

[0004] Methods for alignment are generally divided into two main categories: initialization methods, which aim to provide a quick alignment with relatively low accuracy to provide an initial alignment solution, and refinement methods, which allow for improving the earlier alignment, the initial alignment typically obtained by the initialization method. Refinement methods also allow for adaptation to modifications to the shape or position of the object to be tracked.

[0005] There is a complexity involved in the initialization method, i.e., the absence of a detailed concept of the transformations to be performed, since it is this method that must provide the initial alignment transformation. Known initialization methods can be divided into two categories: automatic methods and manual methods.

[0006] Automatic methods, for example, use visual landmarks and descriptors to automatically calculate registration or use automatic matching. These methods have variable results and must be systematically verified or corrected by an operator in medical and / or surgical settings. These automatic methods are also difficult to implement when the 3D model has no texture and only shape due to the availability of only a few visual landmarks, which is common in the case of preoperative 3D models.

[0007] Existing manual methods can be divided into several subcategories: - a method requiring interactive manipulation of the preoperative 3D model, making it possible, by use of a user interface, to apply transformations to the 3D model until it is aligned with the object appearing on the camera image; - A method requiring interactive selection of a match of known areas, curves or points on an image of the object obtained by a camera and on a 3D model.

[0008] These methods have several drawbacks.

[0009] Methods that require manipulation of the model require taking actions on the rotation, position, and potentially scale of the 3D model, which can be very complex even for an expert: some objects have several axes of symmetry that complicate the task, determining a good match between scale and depth can be complicated, objects may only be partially visible, etc.

[0010] Match point methods can be difficult to implement when these match points are difficult to identify, when the colors of the objects seen on the image and the 3D model are different, etc.

[0011] Furthermore, in the medical or surgical context, these methods require operator involvement on an external computing device, which can present hygiene or sterility issues. This involvement also creates a distraction for the operator, who must direct his attention away from the patient and to the external device.

[0012] The inventors sought to provide a method for alignment for initialization, namely one that allows for simplifying operator interaction and limiting the risk of initialization errors common in existing methods. The proposed method for alignment is believed to form a novel subcategory of manual methods. Summary of the Invention [Problem to be solved by the invention]

[0013] The present invention aims to provide a system and method for the registration of a 3D model of a target organ on at least one image of this organ obtained by an endoscopic camera.

[0014] The present invention, in at least one embodiment, aims to provide a system and method for registration that allows for simple initialization of the registration while obtaining robust results.

[0015] The present invention, in at least one embodiment, aims to provide a system and method for registration that can be used to register a 3D model on an image acquired by an endoscope.

[0016] The present invention, in at least one embodiment, aims to provide a system and method for registration that does not require involvement on a computing device other than medical and / or surgical involvement. [Means for solving the problem]

[0017] To this end, the present invention provides - receiving a 3D model of a target organ; - predicting the position and orientation of the target organ with respect to a reference coordinate frame of the scene in which the position and orientation of the endoscope are known based on a reference position and orientation of the endoscope relative to the scene; - simulating a position and orientation of a 3D model based on said predicted position and said predicted orientation of the target organ; - displaying at least one current image from the endoscope on a display device; - superimposing, on the display device based on the predicted position and orientation, a semi-transparent (see-through) projection of the 3D model onto at least one current image from the endoscope, the projection being fixed relative to the endoscope; - receiving a command indicating an alignment between a semi-transparent projection of the 3D model and an image of the target organ on the current image; - calculating the position and orientation of the target organ on the current image relative to the reference coordinate frame; The present invention relates to a method for the registration of a virtual three-dimensional model of a target organ, called a 3D model, with at least one image of the target organ in a scene obtained by an endoscope, comprising:

[0018] The method for alignment according to the present invention facilitates the initialization of alignment by proposing a solution that allows a semi-transparent display of the projection of the virtual 3D model relative to an expected and predetermined reference position and orientation so that the user manipulates the endoscope until the projection of the 3D model is aligned with the target organ. The operator can then indicate that this alignment is complete by interacting with a verification means, e.g., a physical or graphical verification interface. These actions allow the user to concentrate on observing the displayed image and to manage the endoscope while remaining focused on the real-world patient without having to manipulate the virtual 3D model on an external system. The physical verification interface is, for example, a button activated by the operator's hand or a pedal activated by the operator's foot. According to another variant of the present invention, the verification means includes an automatic verification module that allows the automatic determination of the alignment of the projection of the 3D model with the target organ and the indication to the user that the alignment is correct and / or the sending of a command indicating the alignment between the semi-transparent projection of the 3D model and the image of the target organ on the current image.

[0019] The alignment command allows the calculation of the position and orientation of the target organ relative to a reference coordinate frame on the current image from the position of the endoscope relative to the current image when it is accepted.

[0020] The pairing formed by the position and orientation of an object in a 3D model is commonly called the object pose.

[0021] In conventional registration techniques, the operator interacts with the virtual model to make it correspond to the real world, whereas in the present invention, the operator interacts with the real world to make it correspond to the virtual model, a task that can be accomplished very naturally by the operator, i.e., without direct interaction with hardware not used in medical and / or surgical routines.

[0022] The method is also particularly suitable when the target organ currently seen on the image and the 3D model of the target organ have distinctly different states, i.e., when they have different shapes, especially because deformations are applied to the model or to the target organ as currently seen on the image. In contrast to most prior art methods, the present method for registration operates to calculate pose even in the presence of deformations.

[0023] The expected position and orientation are related to prior knowledge of the reference position and orientation of the endoscope relative to the scene and the target organ. The expected position and orientation can be calculated or pre-calculated, i.e., manually or automatically, and can remain fixed during the remainder of the registration method. For example, in the context of laparoscopy, the endoscope is positioned in a relatively identical standard position and orientation for each laparoscopic procedure; therefore, the position of the 3D model can be adjusted based on these standard positions and orientations. Several reference position and orientation pairings can be pre-calculated to accommodate several possible configurations. For example, the endoscope can be positioned at different entry points depending on the type of involvement, the target organ, and the pathology to be potentially treated.

[0024] The reference coordinate frame is, for example, the coordinate frame of the endoscope, which allows to easily define a fixed transformation of the 3D model in order to obtain a fixed projection of the 3D model in the current image. Other coordinate frames can be used if the transformation of this coordinate frame to the endoscope model is known or can be calculated.

[0025] The current image can be a 2D image or a stereoscopic image, depending on the variant of the invention.

[0026] Within a medical or surgical procedure, the steps of accepting a 3D model of a target organ, predicting the position and orientation of the target organ, and simulating the position and orientation of the 3D model can advantageously be performed "pre-operatively", i.e., upstream of the image capture procedure. The steps of displaying at least one current image, superimposing a semi-transparent projection of the 3D model, accepting commands, and calculating the position and orientation of the target organ are performed "intra-operatively" in parallel with the image capture procedure.

[0027] Advantageously and in accordance with the invention, the method comprises a step of defining a canonical coordinate frame of the 3D model of the organ, said canonical coordinate frame being defined by an origin and three axes, and the step of predicting the position and orientation of the target organ makes it possible to define a transformation of the 3D model of the organ between the canonical coordinate frame and a reference coordinate frame.

[0028] According to this aspect of the invention, the transformation from the canonical coordinate frame to the reference coordinate frame allows the position and orientation within the reference coordinate frame of the 3D model that will be displayed in the current image based on its expected position and orientation. This step can be performed upstream, i.e., in a pre-operative step, before the endoscopic capture of the image of the target organ.

[0029] Advantageously and according to the invention, the target organ is the uterus, including in particular the fundus, the anterior uterine wall and the cervix, and the canonical coordinate frame is defined by an origin forming the centre of mass of the distal part of the uterus, a first left-right axis of the uterus, a second axis connecting the centre of mass of the fundus and the centre of mass of the cervix, and a third axis which is the dot product of the first and second axes.

[0030] According to this aspect of the invention, the method for registration is particularly suitable for registering a 3D model of the uterus onto a current image of the uterus as a target organ. The particular shape of the uterus allows for the definition of a canonical coordinate frame suitable for the implementation of the method for registration.

[0031] Advantageously and in accordance with the invention, the step of defining a canonical coordinate frame comprises: - calculating the main axis of the medial part of the fundus of the uterus, called the fundal axis; - calculating the main axis of the medial part of the anterior wall of the uterus, called the anterior wall axis; - calculating the left-right axis of the uterus by the cross product of the fundal axis and the anterior wall axis; - a sub-step of calculating the center of mass of the fundus; - calculating the center of mass of the cervix; - calculating the plane defined by the planes whose points are equidistant from the center of mass of the fundus and the center of mass of the cervix; - determining two parts of the uterus bounded by the calculated plane: a distal part including the fundus and the anterior wall, and a proximal part connected to the cervix; - calculating the center of mass of the distal region; Includes.

[0032] According to another variation of the invention, the target organ may be an organ other than the uterus, and the canonical coordinate frame is determined based on the approximate shape of the target organ.

[0033] For example, the target organ can be the liver (in which case the virtual 3D model is typically obtained from CT imaging), or the kidney (in which case the virtual 3D model is typically obtained by MRI and / or CT imaging), or even other organs.

[0034] When the target organ is the liver or kidney, the canonical coordinate frame can be defined by a center of mass forming the origin, a first axis that is longer and extends along the principal axis, which is a fixed axis along the principal axis of the 3D model obtained, for example, by principal component analysis of the nodes of the mesh, and is positioned parallel to the mid-section of the endoscope, horizontally or vertically depending on the organ of interest, a second axis that is shorter and extends along the principal axis and points towards the endoscope, and a third axis that is the dot product of the first and second axes.

[0035] Advantageously and according to the invention, the method comprises a step of pre-calculating a projection of the 3D model from the 3D model and the expected position and orientation upstream of the step of superimposing this projection onto at least one current image from the endoscope.

[0036] This aspect of the invention allows the projection of a 3D model to obtain a simple two-dimensional image that can be easily combined with the current image to create a translucent effect. The projection can be complete, but can also consist of a silhouette or a complete or partial outline of the target organ.

[0037] Advantageously and according to the invention, the method comprises a step of accepting a 3D model of the scene generated from images captured by the endoscope, called an intraoperative 3D model, and which comprises: - representing the 3D model of the target organ and the intraoperative 3D model in a common coordinate frame based on the position and orientation of the target organ relative to the reference coordinate frame on the current image; - selecting an origin in a common coordinate frame; generating at least one radius extending from an origin in the direction of the optical axis of the endoscope; - calculating the distance between the origin and the intersection of each radius with the 3D model of the virtual organ; - calculating the distance between the origin and the intersection of each radius with the intraoperative 3D model; - calculating a translation amount and a scale factor from each distance between the origin and the intersection of the radius with the 3D model of the virtual organ and each distance between the origin and the intersection of the radius with the intraoperative 3D model; The method includes a step of calculating a translation amount and a scale factor between the 3D model of the target organ and the intraoperative 3D model, including:

[0038] According to this aspect of the present invention, the calculation of translational and scale factors allows for the guaranteed registration of a 3D model of the target organ, referred to as the preoperative 3D model, with the target organ modeled in a model of the scene acquired by the endoscope, referred to as the intraoperative 3D model. These steps are particularly useful when a scale difference exists between the preoperative and intraoperative 3D models. Indeed, calculating the position and orientation of the target organ on the current image allows for the registration of the 3D model on that image, but does not guarantee the registration of the target organ in the preoperative and intraoperative 3D models due to partial knowledge of depth and spacing based on a single image. The steps of calculating translational and scale factors allow for the perfect registration of the preoperative and intraoperative 3D models by mapping the coordinate frames associated with each model, in particular by enabling the use of a metric coordinate frame for each model, thereby enabling the dimensions of each object in each 3D model to be known.

[0039] The use of several radii and associated distances allows the accuracy of the calculation to be improved, for example by using a median or average value of the distance calculations.

[0040] Advantageously and according to the invention, the method comprises the steps of acquiring several images and selecting from these images those that have a higher quality than the others with regard to the alignment of the 3D model.

[0041] According to this aspect of the invention, the acquisition of several images, for example by taking short video extracts, makes it possible to guarantee the acquisition of at least one image of sufficient quality that can be used as the main image for the generation of an intraoperative 3D model and the tracking of the 3D model of the target organ with respect to this intraoperative 3D model.

[0042] The present invention also provides an endoscope configured to capture an image; a display device configured to display an image; and - verification means configured to provide instructions indicating the alignment between the translucent projection of the 3D model and the image of the target organ; a processing unit, 1. A system for registering a virtual three-dimensional model of a target organ with an image of the target organ, comprising: The processing unit - a module for accepting a 3D model of the target organ; a module for predicting the position and orientation of the target organ relative to a reference coordinate frame of the scene, the position and orientation of which are known, based on a reference position and orientation of the endoscope relative to this scene; a module for simulating the position and orientation of a 3D model based on said predicted position and said predicted orientation of a target organ; a module for displaying at least one current image from the endoscope on a display device; - a module for superimposing a translucent projection of the 3D model on at least one current image from the endoscope based on the predicted position and orientation on the display device, the projection being fixed relative to the endoscope; and a module for receiving a command indicating an alignment between a semi-transparent projection of a 3D model and an image of a target organ on a current image; a module for calculating the position and orientation of the target organ relative to a reference coordinate frame on the current image; Includes.

[0043] The system for registration according to the present invention allows for displaying a projection of a 3D model onto at least one current image and accepting a command indicating alignment of the projection of the 3D model with a target organ on the current image. Information relating to the alignment of the projection of the 3D model with the target organ on the current image allows for calculation of the position of the target organ in the current image, and therefore allows for alignment of the 3D model of the target organ with images of the target organ in the current image and in subsequent images captured by the endoscope.

[0044] A module may include, for example, a computing device such as a computer, a group of computing devices, an electronic component, or a group of electronic components, or, for example, a computer program, a group of computer programs, a library of computer programs, or a function of a computer program executed by a computing device such as a computer, a group of computing devices, an electronic component, or a group of electronic components.

[0045] The verification means may preferably be a physical verification interface by the operator and may include, for example, a physical button, a lever, a pedal intended to be actuated by the operator's foot, etc. Pedals allow the operator to interact without using their hands. The verification means may be a graphical verification interface. The verification means may not require any physical contact, for example by enabling sensors to detect the operator's movements or to detect audible commands. Finally, the verification means may be automatic and may in particular include a verification module that allows automatic image processing and automatic detection of alignment between the translucent projection of the 3D model and the image of the target organ on the current image, thereby suggesting to the user that the alignment is perfect or directly sending a command indicating such alignment.

[0046] Advantageously, the system for alignment according to the invention is configured to implement the method for alignment according to the invention.

[0047] Advantageously and according to the invention, the method for alignment according to the invention is adapted to be carried out by the system for alignment according to the invention.

[0048] Advantageously and according to the invention, the processing unit comprises a module for defining a canonical coordinate frame of the 3D model of the organ, said canonical coordinate frame being defined by an origin and three axes, and a module for predicting the position and orientation of the target organ makes it possible to define a transformation of the 3D model of the organ between the canonical coordinate frame and a reference coordinate frame.

[0049] Advantageously and according to the invention, the target organ is the uterus, including in particular the fundus, the anterior uterine wall and the cervix, and the canonical coordinate frame is defined by an origin forming the centre of mass of the distal part of the uterus, a first left-right axis of the uterus, a second axis connecting the centre of mass of the fundus and the centre of mass of the cervix, and a third axis which is the dot product of the first and second axes.

[0050] Advantageously and according to the invention, the module for defining the canonical coordinate frame comprises: - Calculating the main axis of the inner part of the uterine fundus, called the fundal axis; - Calculating the main axis of the medial part of the anterior wall of the uterus, called the anterior wall axis, - Calculate the left-right axis of the uterus by the cross product of the fundal axis and the anterior wall axis; - Calculate the center of mass of the fundus, - Calculate the center of mass of the cervix, - calculating the plane defined by the planes whose points are equidistant from the center of mass of the fundus and the center of mass of the cervix; - determining two parts of the uterus bounded by the calculated plane: a distal part including the fundus and the anterior wall, and a proximal part connected to the cervix; - Calculate the center of mass of the distal segment; It is configured as follows.

[0051] Advantageously and according to the invention, the processing unit comprises a module for pre-calculating the projection of the 3D model from the 3D model and the expected position and orientation.

[0052] Advantageously and according to the invention, the processing unit comprises a module for receiving a 3D model, called an intraoperative 3D model of the scene, generated from the images captured by the endoscope; - Representing the 3D model of the target organ and the intraoperative 3D model in a common coordinate frame based on the position and orientation of the target organ relative to the reference coordinate frame on the current image; -Selecting the origin in a common coordinate frame, generating at least one radius extending from an origin in the direction of the optical axis of the endoscope; Calculating the distance between the origin and the intersection of each radius with the 3D model of the virtual organ; - Calculating the distance between the origin and the intersection of each radius with the intraoperative 3D model; - calculating a translation amount and a scale factor from each distance between the origin and the intersection of the radius with the 3D model of the virtual organ and each distance between the origin and the intersection of the radius with the intraoperative 3D model; a module for calculating a translation amount and a scale factor between the 3D model of the target organ and the intraoperative 3D model configured as described above; Includes.

[0053] Advantageously and according to the invention, the processing unit comprises a module for acquiring several images and a module for selecting from these images those that have a higher quality than the others with regard to alignment with the 3D model.

[0054] The advantages of these variants of the system for alignment according to the invention are similar to the advantages of the above-mentioned variants of the method for alignment according to the invention.

[0055] The invention also relates to a computer program product for the registration of a virtual three-dimensional model, called a 3D model, of a target organ with at least one image of the target organ in a scene obtained by an endoscope, the computer program product being capable of, when it is executed on a computer, - receiving a 3D model of a target organ; - predicting the position and orientation of the target organ relative to a reference coordinate frame of the scene in which the position and orientation of the endoscope are known based on a reference position and orientation of the endoscope relative to the scene; - simulating the position and orientation of a 3D model based on said predicted position and said predicted orientation of the target organ; - displaying at least one current image from the endoscope on a display device; - superimposing a semi-transparent projection of the 3D model on at least one current image from the endoscope based on the predicted position and orientation on the display device, the projection being fixed relative to the endoscope; - receiving a command indicating an alignment between a semi-transparent projection of the 3D model and an image of the target organ on the current image; - calculating the position and orientation of the target organ relative to a reference coordinate frame on the current image; The program code instructions for executing the program are:

[0056] Advantageously and according to the invention, a computer program product for alignment according to the invention comprises program code instructions for carrying out the steps of the method for alignment according to the invention, in particular the steps of the method for alignment of all of the variants of the invention described above, when it is executed on a computer.

[0057] Advantageously and according to the invention, the method for registration according to the invention is adapted to be implemented by a computer program product for registration according to the invention.

[0058] The invention also relates to a system for registration, a method for registration and a computer program product for registration characterized by a combination of all or some of the features mentioned above or below. [Brief explanation of the drawings]

[0059] Other objects, features and advantages of the present invention will become apparent on reading the following description, given in a non-limiting manner only and made with reference to the accompanying drawings, in which:

[0060] [Figure 1] 1 is a schematic diagram of a system 10 for registration according to an embodiment of the present invention integrated into a laparoscopic imaging system in a first configuration. [Figure 2] 1 is a schematic diagram of a system 10 for registration according to an embodiment of the present invention integrated into a laparoscopic imaging system in a second configuration. [Figure 3] FIG. 2 is a schematic diagram of a method for alignment according to an embodiment of the present invention. [Figure 4] 1 is a schematic illustration of a uterus forming a target organ of a method for registration according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0061] In the figures, for purposes of illustration and clarity, strict scale and proportions have not been adhered to.

[0062] Furthermore, identical, similar or analogous elements will be labeled with the same reference symbols in all figures.

[0063] 1 and 2 show a schematic representation of a system 10 for registration according to an embodiment of the present invention integrated into a laparoscopic imaging system. The purpose of the imaging system is to enable the acquisition and output of images taken within a cavity 50 in a patient's body, in this case a cavity within the patient's abdomen (or abdominal cavity 50), in particular during a laparoscopic procedure, e.g., laparoscopic surgery. The laparoscopic surgery may, for example, be aimed at surgery on a target organ 52.

[0064] A laparoscopic imaging system includes, for example, a system for registration 10 according to one embodiment of the present invention that accepts images provided by an endoscopic camera 12 configured to acquire images of a patient's abdominal cavity 50. Endoscopes used within laparoscopic surgery are currently referred to as laparoscopes.

[0065] The system for alignment comprises a number of modules brought together in this case in a processing unit 16 which make it possible to carry out the method according to the invention. The processing unit 16 is for example a processor, for example a processor dedicated to image processing of the method according to the invention, or further a computer or electronic board comprising a general-purpose processor arranged, among other functions, to execute program instructions for the execution of the steps of the method according to the invention.

[0066] The images acquired by the endoscope 12 are displayed on a display device, such as the registration system's operator display screen 18. The acquired images can be augmented, i.e., can include additional information added by the laparoscopic imaging system, which may come from the registration system or other devices.

[0067] To enable close tracking of this additional information on the displayed image, the system for registration 10 is configured to determine the position and orientation of the target organ 52 in the reference coordinate frame of the scene so as to enable display of the additional information based on the position and orientation of the target organ 52. In particular, the aim is to display a 3D model of the target organ on the image of the target organ 52, which requires registration of the 3D model with the image of the target organ 52.

[0068] To do so, the system for alignment 10 implements a method for alignment as shown in FIG.

[0069] The method 100 for alignment comprises: a step 110 of receiving, by the processing unit 16, a 3D model of the target organ, provided for example by an external computing device and obtained in particular by medical imaging of the Magnetic Resonance Imaging (MRI) type; - a step 112 of predicting, by the processing unit 16, the position and orientation of the target organ relative to the reference coordinate frame of this scene, in which the position and orientation of the endoscope are known, based on the predicted position and orientation of the endoscope relative to the scene; - a step 114 of simulating, by the processing unit 16, the position and orientation of a 3D model based on said predicted position and said predicted orientation of the target organ; Includes.

[0070] These steps can be performed upstream of the medical and / or surgical procedure (pre-operative phase).

[0071] The method 100 for registration also comprises the following steps, which are preferably carried out in parallel, in particular during a medical and / or surgical procedure (intraoperative phase): - displaying 116 at least one current image from the endoscope 12 on the display device 18; - superimposing 118 a semi-transparent projection 20 of the 3D model onto at least one current image from the endoscope based on the predicted position and orientation on the display device 18, said projection being fixed relative to the endoscope 12. In Figures 1 and 2 the semi-transparent projection 20 is shown as a dotted line and is fixed relative to the image.

[0072] The method 100 for registration can include a step 128 of pre-calculating a projection of the 3D model from the 3D model and the expected position and orientation, preferably in a pre-operative phase, upstream of the superposition step 118. This phase makes it possible to provide a projection of the 3D model that will be displayed on the display device 18.

[0073] Following this registration step 118, the operator in charge of operating the endoscope can attempt to align the image 22 of the target organ on the current image with the translucent projection 20. Figure 1 shows a first position in which the translucent projection 20 and the image 22 of the target organ are not aligned, and Figure 2 shows a second position in which the translucent projection 20 and the image 22 of the target organ are aligned. When the images are aligned in the operator's opinion, the operator can then activate the verification means, in particular the physical verification interface 24, which may include, for example, a foot-activated pedal for sending a command indicating the alignment between the translucent projection of the 3D model and the image 22 of the target organ on the current image.

[0074] According to another embodiment of the invention, the verification means may not require any physical contact, for example by enabling a sensor to detect the movement of the operator or to detect an audible command. According to another embodiment of the invention, the verification means may be automatic and in particular may comprise a verification module that enables automatic image processing and automatic detection of alignment between the semi-transparent projection of the 3D model and the image of the target organ on the current image, thereby suggesting to the user that the alignment is perfect or directly sending a command indicating such alignment.

[0075] The method 100 for alignment then includes: a step 120 of receiving a command indicating an alignment between a semi-transparent projection 20 of the 3D model and an image 22 of the target organ on the current image; a step 122 of calculating the position and orientation of the target organ relative to the reference coordinate frame on the current image; This calculation is performed from the current image when a command indicating alignment between the projection 20 and the image 22 of the target organ is received;

[0076] The method 100 for registration is preferably performed in a pre-operative phase and also includes a step 124 of defining a canonical coordinate frame of the 3D model of the organ, said canonical coordinate frame being defined by an origin and three axes, and step 112 of predicting the position and orientation of the target organ allows for defining a transformation of the 3D model of the organ between the canonical coordinate frame and the reference coordinate frame.

[0077] As shown in FIG. 4, which shows the uterus 200 in a) transverse plane and b) longitudinal plane, when the target organ is the uterus 200, including in particular the fundus 210, the anterior uterine wall 212, and the cervix 214, the canonical coordinate frame is defined by an origin G that forms the center of mass of the distal portion of the uterus. U , the primary left-right axis U of the uterus, and the center of mass G of the fundus F and the center of mass of the cervix G C and a third axis that is the dot product of the first axis and the second axis.

[0078] In particular, step 124 of defining a canonical coordinate frame for the uterus 200 includes: -fundal axis N F a sub-step of calculating the major axis of the medial portion of the fundus 210, called -Front wall axis N W a sub-step of calculating the major axis of the medial portion of the anterior uterine wall 212, called -fundal axis NF and front wall axis N W a sub-step of calculating the left-right axis of the uterus (not shown) by the cross product of -Center of mass of fundus G F and a sub-step of computing -Cervix center of mass G C and a sub-step of computing -Each point is the center of mass of the fundus G F and the center of mass of the cervix G C and the sub-step of calculating a plane P defined by a plane equidistant from - determining two regions of the uterus bounded by a plane P: a distal region 216a including the fundus and the anterior wall, and a proximal region 216b connected to the cervix; -Center of mass of distal segment G U and a sub-step of computing Includes.

[0079] After these substeps, the origin and axes necessary to define the canonical coordinate frame are known.

[0080] The method 100 for registration also includes a step 126 of accepting a 3D model generated from images captured by the endoscope, referred to as an intraoperative 3D model; and a substep of representing the 3D model of the target organ and the intraoperative 3D model in a common coordinate frame based on the position and orientation of the target organ relative to the reference coordinate frame on the current image. - selecting an origin in a common coordinate frame; generating at least one radius extending from an origin in the direction of the optical axis of the endoscope; - calculating the distance between the origin and the intersection of each radius with the 3D model of the virtual organ; - calculating the distance between the origin and the intersection of each radius with the intraoperative 3D model; - calculating a translation amount and a scale factor from each distance between the origin and the intersection of the radius with the 3D model of the virtual organ and each distance between the origin and the intersection of the radius with the intraoperative 3D model; calculating a translation amount and a scale factor between the 3D model of the target organ and the intraoperative 3D model; may include:

[0081] The endoscope 12 can be configured to acquire several images, and the method 100 for alignment can also include selecting from these images images that have higher quality than other images with respect to alignment of the 3D model.

[0082] The invention is not limited to the described embodiments. The system for registration can be integrated into different types of imaging, in particular other types of medical imaging, especially if the position and orientation of the target organ relative to the endoscope can be predicted. Furthermore, the target organ can be different, i.e. in the context of laparoscopy, the target organ can be another organ that may be operated on laparoscopically in the abdomen, such as the liver or kidneys.

Claims

1. A method for aligning a virtual three-dimensional model called a 3D model of a target organ with at least one image of the target organ in a scene obtained by endoscopy, - The step of receiving the 3D model of the target organ (110), - A step (112) of predicting the position and direction of the target organ relative to a reference coordinate frame of the scene, where the position and direction of the endoscope are known, based on the reference position and direction of the endoscope in the scene, - A step (114) of simulating the position and orientation of the 3D model based on the predicted position and direction of the target organ, - A step (116) of displaying at least one current image from the endoscope on a display device, - Step (118) of superimposing a semi-transparent projection of the 3D model onto at least one current image from the endoscope on the display device based on the predicted position and direction, wherein the projection is fixed to the endoscope, - A step (120) of receiving a command indicating alignment between the translucent projection of the 3D model and the image of the target organ on the current image, - A step (122) of calculating the position and orientation of the target organ on the current image with respect to the reference coordinate frame, A method that includes this.

2. The method includes a step (124) of defining a canonical coordinate frame of the 3D model of the organ, wherein the canonical coordinate frame is defined by an origin and three axes, The step of predicting the position and orientation of the target organ makes it possible to determine the transformation of the 3D model of the organ between the canonical coordinate frame and the reference coordinate frame. The alignment method according to feature 1.

3. The method for alignment according to claim 2, wherein the target organ is, in particular, the uterus (200) including the uterine fundus (210), the anterior uterine wall (21), and the cervix (214), and the canonical coordinate frame is defined by an origin forming the center of mass of the distal portion of the uterus, a first left-right axis of the uterus, a second axis connecting the center of mass of the uterine fundus and the center of mass of the cervix, and a third axis which is the dot product of the first axis and the second axis.

4. The step (124) of defining the canonical coordinate frame is, - A substep to calculate the principal axis of the inner portion of the uterine fundus (210), called the fundal axis (NF), - A substep to calculate the principal axis of the medial portion of the anterior uterine wall (212), called the anterior wall axis (NW), - A substep of calculating the left-right axis of the uterus by the cross product of the uterine fundal axis (NF) and the anterior wall axis (NW), - A substep for calculating the center of mass (GF) of the uterine fundus, - A substep for calculating the center of mass (GC) of the cervix, - A substep of calculating a plane (P) defined by a plane such that each point is equidistant from the center of mass (GF) of the uterine fundus and the center of mass (GC) of the cervix, - A substep of determining two parts of the uterus whose boundaries are defined by the plane (P), namely, a distal part (216a) including the uterine fundus and the anterior wall and a proximal part (216b) connected to the cervix, - A substep for calculating the center of mass (GU) of the distal portion, including, The alignment method according to feature 3.

5. The alignment method according to claim 1, further comprising a step (128) of pre-calculating the projection of the 3D model from the 3D model and the expected position and orientation, upstream of the step (118) of superimposing the projection onto at least one current image from the endoscope.

6. The step (126) includes receiving a 3D model called an intraoperative 3D model of the scene generated from the images captured by the endoscope, - A substep in which the 3D model of the target organ and the intraoperative 3D model of the target organ are represented in a common coordinate frame based on the position and orientation of the target organ on the current image relative to the reference coordinate frame, - A substep of selecting the origin within the aforementioned common coordinate frame, - A substep of generating at least one radius extending from the origin and in the direction of the optical axis of the endoscope, - A substep to calculate the distance between the origin and the intersection points of each radius and the 3D model of the virtual organ, - A substep to calculate the distance between the origin and the intersection points of each radius and the intraoperative 3D model, - A substep of calculating translational amounts and scale coefficients from the distances between the origin and the intersection point of the radius and the 3D model of the virtual organ, and the distances between the origin and the intersection point of the radius and the intraoperative 3D model, The steps include calculating the translation and scaling coefficients between the 3D model of the target organ and the intraoperative 3D model, The alignment method according to feature 1.

7. The method for alignment according to claim 1, comprising the steps of acquiring several images and selecting from the images an image having higher quality than the other images with respect to the alignment of the 3D model.

8. A system for aligning a virtual three-dimensional model of a target organ with an image of the target organ, - An endoscope (12) configured to capture the aforementioned image, - A display device (18) configured to display the aforementioned image, - Verification means (24) configured to provide a command indicating alignment between the semi-transparent projection of the 3D model and the image of the target organ, - Processing unit (16), Includes, The processing unit (16) is - A module for receiving the 3D model of the target organ, - A module for predicting the position and orientation of the target organ relative to a reference coordinate frame of the scene, where the position and orientation of the endoscope are known, based on the reference position and orientation of the endoscope relative to the scene. - A module for simulating the position and orientation of the 3D model based on the predicted position and direction of the target organ, - A module for displaying at least one current image from the endoscope on a display device, - A module for superimposing a semi-transparent projection of the 3D model onto at least one current image from the endoscope, based on the predicted position and orientation on the display device, wherein the projection is fixed to the endoscope, and the superimposing module, - A module for receiving a command indicating the alignment between the translucent projection of the 3D model and the image of the target organ on the current image, - A module for calculating the position and orientation of the target organ on the current image relative to the reference coordinate frame, including, A system for alignment.

9. The alignment system according to claim 8, characterized in that the verification means is a physical verification interface (24) including a pedal intended to be operated by the operator's foot.

10. The processing unit includes a module for defining a canonical coordinate frame of the 3D model of the organ, wherein the canonical coordinate frame is defined by an origin and three axes, and the module for defining the canonical coordinate frame is a module for defining the canonical coordinate frame. The module for predicting the position and orientation of the target organ enables the determination of the transformation of the 3D model of the organ between the canonical coordinate frame and the reference coordinate frame. The alignment system according to feature 8.

11. The alignment system according to claim 8, wherein the target organ is, in particular, the uterus (200) including the uterine fundus (210), the anterior uterine wall (21), and the cervix (214), and the canonical coordinate frame is defined by an origin forming the center of mass of the distal portion of the uterus, a first left-right axis of the uterus, a second axis connecting the center of mass of the uterine fundus and the center of mass of the cervix, and a third axis which is the dot product of the first axis and the second axis.

12. The module for defining the canonical coordinate frame is: -Calculate the principal axis of the inner part of the uterine fundus (210), which is called the uterine fundal axis (NF), -Calculate the main axis of the medial portion of the anterior uterine wall (212), which is called the anterior wall axis (NW), - The left-right axis of the uterus is calculated by the cross product of the uterine fundal axis (NF) and the anterior wall axis (NW), -Calculate the center of mass (GF) of the uterine fundus, -Calculate the center of mass (GC) of the cervix, - Calculate a plane (P) defined by a plane such that each point is equidistant from the center of mass (GF) of the uterine fundus and the center of mass (GC) of the cervix, - Determine the two parts of the uterus whose boundary is defined by the plane (P), namely, the distal part (216a) including the uterine fundus and the anterior wall and the proximal part (216b) connected to the cervix, - Calculate the center of mass (GU) of the distal region. It is configured in such a way. The alignment system according to feature 8.

13. The alignment system according to claim 8, wherein the processing unit includes a module for pre-calculating the projection of the 3D model from the 3D model and the predicted position and direction.

14. The processing unit includes a module for receiving a 3D model of the scene generated from images captured by the endoscope, called an intraoperative 3D model, and further, - Based on the position and orientation of the target organ on the current image relative to the reference coordinate frame, the 3D model of the target organ and the intraoperative 3D model are represented within a common coordinate frame. - Select the origin within the aforementioned common coordinate frame, - Generates at least one radius extending from the origin and in the direction of the optical axis of the endoscope, -Calculate the distance between the origin and the intersection points of each radius and the 3D model of the virtual organ. -Calculate the distance between the origin and the intersection points of each radius and the 3D model during surgery. - Calculate the translation amount and scale coefficient from the distances between the origin and the intersection point of the radius and the 3D model of the virtual organ, and from the distances between the origin and the intersection point of the radius and the intraoperative 3D model. The alignment system according to claim 8, comprising a module configured to calculate the translation amount and scale coefficient between the 3D model of the target organ and the intraoperative 3D model.

15. The alignment system according to claim 8, wherein the processing unit includes a module for acquiring several images and a module for selecting from the images an image having higher quality than other images with respect to the alignment of the 3D model.

16. A computer program product for aligning a virtual three-dimensional model called a 3D model of a target organ with at least one image of the target organ in a scene obtained by endoscopy, When the aforementioned computer program product is executed on a computer, - The step of receiving the 3D model of the target organ (110), - A step (112) of predicting the position and direction of the target organ relative to a reference coordinate frame of the scene, where the position and direction of the endoscope are known, based on the reference position and direction of the endoscope in the scene, - A step (114) of simulating the position and orientation of the 3D model based on the predicted position and direction of the target organ, - A step (116) of displaying at least one current image from the endoscope on a display device, - Step (118) of superimposing a semi-transparent projection of the 3D model onto at least one current image from the endoscope, based on the predicted position and orientation on the display device, wherein the projection is fixed to the endoscope, - A step (120) of receiving a command indicating alignment between the translucent projection of the 3D model and the image of the target organ on the current image, - A step (122) of calculating the position and orientation of the target organ on the current image with respect to the reference coordinate frame, Includes program code instructions for executing, Computer program products.

Images