System and method for registering virtual 3d models by translucent display

Abstract

The invention relates to a method of registering a virtual three-dimensional model of a target organ, referred to as a 3D model, with at least one image of the target organ in a scene obtained by an endoscope (12), especially comprising: a step of predicting the position and orientation of the target organ relative to the scene, a step of superimposing, on a display device (18), a translucent projection of the 3D model on at least one current image from the endoscope based on the predicted position and orientation, a step of receiving a command indicating alignment between the translucent projection of the 3D model and an image of the target organ on the current image, and calculating the position and orientation of the target organ relative to the scene on the current image.

Description

Technical Field

[0001] The present invention relates to a system and method for registering a 3D model on an optical image. In particular, the present invention relates to registering a virtual 3D model of an organ obtained from preoperative imaging with an organ image obtained by optical imaging (in particular, stereoscopic or monocular endoscopic imaging). The present invention can be particularly used in laparoscopic imaging and the robotics field. Background Art

[0002] Registration, also known as "alignment", is an operation that can make a virtual 3D model of an object match an optical image of the same object obtained by a camera in terms of position and orientation. In particular, the aim is to determine the transformation (change of coordinate system and deformation) that must be generated to align the 3D model of the object with the image of the object having different coordinate systems and different states. In fact, through the knowledge of this transformation from registration, a virtual 3D model representing the real world can be displayed on the image obtained by the camera in an augmented reality environment. The aim is to ensure tracking of the object on the image so that the 3D model can be displayed on this image after registration.

[0003] In particular, registration methods are implemented in the medical and / or surgical field to map a 3D model, called a preoperative model, because this model is obtained upstream of one or more medical imaging techniques (e.g., radiological examination, ultrasound, MRI, CT, etc.) and has a real image obtained by an optical camera (e.g., an endoscope).

[0004] Registration methods are generally divided into two main categories: initialization methods, which aim to provide rapid registration with relatively low accuracy, thus providing an initial registration scheme; and refinement methods, which can improve the early registration. Usually, the first registration is obtained through the initialization method. The refinement method also allows for adaptation to modify the shape or position of the object being tracked.

[0005] Initialization methods are complex, that is, they lack an accurate concept of the transformation to be achieved because such methods must provide the first registration transformation. Known initialization methods can be divided into two categories: automatic methods and manual methods.

[0006] Automatic methods use, for example, visual indices and descriptors or automatic matching to automatically calculate the registration. The results of these methods vary, and in a medical and / or surgical environment, they must be systematically verified or corrected by the operator. These automatic methods are also difficult to implement when the 3D model has no texture but only shape because there are few available visual metrics, which is common in preoperative 3D models.

[0007] Existing manual methods can be divided into multiple subcategories:

[0008] - Methods for interactive manipulation of preoperative 3D models, which, due to the use of a user interface, allow transformations to be applied to the 3D model until it is aligned with the objects appearing in the image of the camera;

[0009] - Methods for interactively selecting known regions, curves, or point matches on the object image obtained by the camera and on the 3D model.

[0010] These methods have many drawbacks.

[0011] Methods that require manipulation of the model need to act on the rotation, position, and possibly the scale of the 3D model, which can be very complex, even for experienced individuals: some objects have multiple axes of symmetry, which complicates the manipulation; determining a good match between scale and depth can be complex; the object may be only partially visible, etc.

[0012] Methods using matching points can be difficult to implement if the matching points are difficult to identify, if the color of the object seen in the image is different from the 3D model, etc.

[0013] Moreover, in a medical or surgical environment, these methods require operator intervention on an external computing device, which can pose problems related to hygiene or disinfection. This intervention also distracts the operator, who has to shift their attention from the patient to the external device.

[0014] Therefore, the inventors seek to provide a registration method for initialization that enables simplifying the operator's interaction and limiting the risk of initialization errors common in existing methods. The proposed registration method will form a new subclass of manual methods. Summary of the Invention

[0015] The object of the present invention is to provide a system and method for registering a 3D model of an organ on at least one image of a target organ obtained by an endoscopic camera.

[0016] In at least one embodiment, the object of the present invention is to provide a registration system and method that can simplify the initialization of registration while obtaining robust results.

[0017] In at least one embodiment, the object of the present invention is to provide a registration system and method that can be used to register a 3D model on an image obtained by an endoscope.

[0018] In at least one embodiment, the object of the present invention is to provide a registration system and method that does not require any intervention on a computing device other than medical and / or surgical intervention.

[0019] To this end, the present invention relates to a method for registering a virtual three-dimensional model of a target organ (referred to as a 3D model) with at least one image of the target organ in a scene obtained by an endoscope, comprising:

[0020] - a step of receiving the 3D model of the target organ;

[0021] - a step of predicting the position and orientation of the target organ with respect to a reference coordinate system of the scene based on the reference position and orientation of the endoscope with respect to the scene;

[0022] - a step of simulating the position and orientation of the 3D model based on the predicted position and orientation of the target organ;

[0023] - a step of displaying at least one current image from the endoscope on a display device;

[0024] - a step of superimposing a translucent projection of the 3D model on at least one current image from the endoscope on the display device based on the predicted position and orientation, the projection being fixed with respect to the endoscope;

[0025] - a step of receiving a command indicating alignment between the translucent projection of the 3D model and the image of the target organ on the current image;

[0026] - a step of calculating the position and orientation of the target organ on the current image with respect to the reference coordinate system.

[0027] Thus, the registration method according to the present invention can facilitate the initialization of registration by proposing a solution that can display the projection of the virtual 3D model semi-transparently with respect to the predicted and predefined reference position and orientation, such that the user manipulates the endoscope until the projection of the 3D model is aligned with the target organ. Then, the operator can indicate that the alignment is complete by interacting with a verification device (such as a physical verification interface or a graphical verification interface). These actions allow the user to concentrate on viewing the displayed images and managing the endoscope while maintaining attention on the real world and the patient, without manipulating the virtual 3D model on an external system. The physical verification interface is, for example, a button activated by the operator's hand or a pedal activated by the operator's foot. According to another variant of the present invention, the verification device includes an automatic verification module that can automatically test the alignment between the projection of the 3D model and the target organ and can suggest to the user that the alignment is correct and / or send a command indicating alignment between the translucent projection of the 3D model and the image of the target organ on the current image.

[0028] When a command indicating alignment is received, the command indicating alignment enables the position and orientation of the target organ on the current image with respect to the reference coordinate system to be calculated from the position of the endoscope with respect to the current image.

[0029] The pairing formed by the position and orientation of an object or 3D model is generally referred to as the pose of the object.

[0030] In traditional registration techniques, the operator interacts with the virtual model to make it correspond to the real world; in the present invention, the operator interacts with the real world to make it correspond to the virtual model. The operation can be performed without direct interaction with hardware not used in conventional medical and / or surgical routines, so the operator can operate very naturally.

[0031] The method is also particularly applicable when the target organ seen in the current image and the 3D model of the target organ are in different states, i.e., have different shapes, especially because a deformation is applied to the model or the target organ seen in the current image. Different from most methods in the prior art, this registration method can calculate the pose even in the presence of deformation.

[0032] The predicted position and orientation are related to the prior knowledge of the scene and the reference position and orientation of the endoscope relative to the target organ. Therefore, the predicted position and orientation can be calculated or pre-calculated manually or automatically and remain unchanged in the remaining steps of the registration method. For example, in laparoscopy, the endoscope is arranged in a relatively consistent standard position and orientation for each laparoscopy procedure, so the position of the 3D model can be adjusted according to this standard position and orientation. Multiple pairs of reference positions and orientations can be pre-calculated to adapt to various possible configurations. For example, according to the type of intervention, the target organ, and the pathology that may need to be treated, the endoscope can be arranged at different entry points.

[0033] The reference coordinate system is, for example, the coordinate system of the endoscope, which can easily define the fixed transformation of the 3D model so as to obtain the projection of the fixed 3D model in the current image. Other coordinate systems can be used provided that the transformation from this coordinate system to the endoscope model is known or can be calculated.

[0034] According to a variant of the present invention, the current image can be a 2D image or a stereoscopic image.

[0035] In the scope of medical or surgical operations, the steps of receiving the 3D model of the target organ, predicting the position and orientation of the target organ, and simulating the position and orientation of the 3D model can be advantageously performed "preoperatively", i.e., before the image capture process. The steps of displaying at least one current image, superimposing the semi-transparent projection of the 3D model, receiving commands, and calculating the position and orientation of the target organ are advantageously performed "intraoperatively" in parallel with the image capture process.

[0036] Advantageously, according to the present invention, the method comprises: a step of defining a standard coordinate system of a 3D model of an organ, the standard coordinate system being defined by an origin and three axes; and a step of predicting the position and orientation of the target organ capable of defining the transformation between the 3D model of the organ in the standard coordinate system and the reference coordinate system.

[0037] According to this aspect of the present invention, the transformation from the standard coordinate system to the reference coordinate system enables the position and orientation of the 3D model in the reference coordinate system to be defined based on the predicted position and orientation, and the reference coordinate system will be displayed in the current image. This step can be performed upstream, i.e., in a preoperative step, before the image of the target organ is captured by the endoscope.

[0038] Advantageously, according to the present invention, the target organ is the uterus, particularly including the uterine fundus, the anterior uterine wall, and the cervix; and the standard coordinate system is defined by an origin forming the centroid of the distal part of the uterus, a first left-right axis of the uterus, a second axis connecting the centroid of the uterine fundus and the centroid of the cervix, and a third axis that is the scalar product of the first axis and the second axis.

[0039] According to this aspect of the present invention, the registration method is particularly suitable for registering the 3D model of the uterus onto the current image of the uterus as the target organ. The special shape of the uterus allows for the definition of a standard coordinate system suitable for implementing the registration method.

[0040] Advantageously, according to the present invention, the step of defining the standard coordinate system includes:

[0041] - a sub-step of calculating the principal axis (referred to as the uterine fundus axis) of the medial part of the uterine fundus;

[0042] - a sub-step of calculating the principal axis (referred to as the anterior wall axis) of the medial part of the anterior uterine wall;

[0043] - a sub-step of calculating the left-right axis of the uterus by the vector product of the uterine fundus axis and the anterior wall axis;

[0044] - a sub-step of calculating the centroid of the uterine fundus;

[0045] - a sub-step of calculating the centroid of the cervix;

[0046] - a sub-step of calculating the plane defined by the plane in which each point is equidistant from the centroid of the uterine fundus and the centroid of the cervix;

[0047] - a sub-step of determining the two parts of the uterus bounded by the plane, the distal part including the uterine fundus and the anterior wall, and the proximal part being connected to the cervix;

[0048] - a sub-step of calculating the centroid of the distal part.

[0049] According to other variants of the present invention, the target organ may be an organ other than the uterus, and a standard coordinate system is determined based on the conventional shape of the target organ.

[0050] For example, the target organ may be the liver (wherein, the virtual 3D model is usually obtained by CT imaging) or the kidney (wherein, the virtual 3D model is usually obtained by MRI and / or CT imaging), or even other organs.

[0051] When the target organ is the liver or the kidney, the standard coordinate system may be defined by the centroid forming the origin and the fixed axes along the main axes of the 3D model. For example, these fixed axes are obtained by performing principal component analysis on the mesh nodes, wherein the first axis extends along the main axis, has a longer length and is parallel to the image plane of the endoscope, and is horizontal or vertical according to the organ under discussion; the second axis extends along the main axis, has a shorter length and faces the endoscope; the third axis is the scalar product of the first axis and the second axis.

[0052] Advantageously, according to the present invention, the method includes: before the step of superimposing the projection on at least one current image from the endoscope, a step of pre-computing the projection of the 3D model by the 3D model and the predicted position and orientation.

[0053] According to this aspect of the present invention, the projection of the 3D model enables a simple two-dimensional image to be obtained, and this simple image can be easily combined with the current image to produce a translucent effect. The projection may be complete, but may also consist of the silhouette of the target organ or all or part of the contour.

[0054] Advantageously, according to the present invention, the method includes: a step of receiving a 3D model (referred to as the intraoperative 3D model) generated from an image captured by the endoscope; and the method includes: a step of calculating the translation and scaling factors between the 3D model of the target organ and the intraoperative 3D model, including:

[0055] - a sub-step of expressing the 3D model of the target organ and the intraoperative 3D model in a common coordinate system based on the position and orientation of the target organ relative to the reference coordinate system on the current image;

[0056] - a sub-step of selecting the origin in the common coordinate system;

[0057] - a sub-step of generating at least one radius extending from the origin and extending in the direction of the optical axis of the endoscope;

[0058] - a sub-step of calculating the distances between the origin and the intersection points of each radius and the 3D model of the virtual organ;

[0059] - a sub-step of calculating the distances between the origin and the intersection points of each radius and the intraoperative 3D model;

[0060] - A sub-step of calculating translation and scaling factors based on each distance between the origin and the intersection points of the radius and the 3D model of the virtual organ, and each distance between the origin and the intersection points of the radius and the intraoperative 3D model.

[0061] According to this aspect of the present invention, calculating translation and scaling factors can ensure the registration of the 3D model of the target organ (referred to as the preoperative 3D model) and the target organ modeled in the scene model obtained by the endoscope (referred to as the intraoperative 3D model). In particular, these steps are relevant if there is a scale difference between the preoperative 3D model and the intraoperative 3D model. In fact, calculating the position and orientation of the target organ on the current image enables the registration of the 3D model on the image, but due to the limited knowledge of depth and spacing based on a single image, it is impossible to ensure the registration of the target organ in the preoperative 3D model and the intraoperative 3D model. The steps of calculating translation and scaling factors allow for the complete registration of the preoperative 3D model and the intraoperative 3D model by mapping the coordinate systems associated with each model. In particular, by using a metric coordinate system for each model, the dimensions of each object in each 3D model can be known.

[0062] Using multiple radii and related distances can improve the calculation accuracy, such as using the median or average of the calculated distances.

[0063] Advantageously, according to the present invention, the method includes: acquiring a plurality of images, and a step of registering the 3D model by selecting an image with higher quality than other images from the plurality of images.

[0064] According to this aspect of the present invention, acquiring a plurality of images (e.g., by acquiring a short video clip) can ensure that at least one image with sufficient acquisition quality can be used as a key image for generating the intraoperative 3D model, and tracking the 3D model of the target organ relative to the intraoperative 3D model.

[0065] The present invention also relates to a system for registering a virtual three-dimensional model of a target organ with an image of the target organ, including:

[0066] - An endoscope configured to capture images;

[0067] - A display device configured to display images;

[0068] - A verification device configured to provide a command indicating the alignment between the semi-transparent projection of the 3D model and the image of the target organ;

[0069] - A processing unit,

[0070] The processing unit includes:

[0071] - A module for receiving the 3D model of the target organ;

[0072] - A module for predicting the position and orientation of a reference coordinate system of a scene for predicting the position and orientation of a target organ relative to a known endoscope based on the reference position and orientation of the endoscope relative to the scene;

[0073] - A module for simulating the position and orientation of a 3D model based on the predicted position and predicted orientation of the target organ;

[0074] - A module for displaying at least one current image from the endoscope on a display device;

[0075] - A module for superimposing a translucent projection of the 3D model on at least one current image from the endoscope on the display device based on the predicted position and orientation, the projection being fixed relative to the endoscope;

[0076] - A module for receiving a command indicating alignment between the translucent projection of the 3D model and the image of the target organ on the current image;

[0077] - A module for calculating the position and orientation of the target organ relative to the reference coordinate system on the current image.

[0078] The registration system according to the present invention allows the projection of the 3D model to be displayed on at least one current image and receives a command indicating alignment of the projection of the 3D model with the target organ on the current image. Information related to the alignment of the projection of the 3D model with the image of the target organ on the current image allows the position of the target organ in the current image to be calculated, thereby registering the 3D model of the target organ with the image of the target organ in the current image and future images captured by the endoscope.

[0079] A module can for example consist of a computing device (such as a computer), a set of computing devices, an electronic component or a set of electronic components; or for example consist of a computer program, a set of computer programs, a computer program library or a computer program function executed by a computing device (such as a computer), a set of computing devices, an electronic component or a set of component components.

[0080] The verification means can preferably be a physical verification interface with an operator and can for example include physical buttons, joysticks, pedals activated by the operator's foot, etc. The pedal allows the operator to interact without using their hands. The verification means can also be a graphical verification interface. The verification means can also be without any physical contact, for example by using sensors to detect the operator's movements or to detect voice commands. Finally, the verification means can also be automatic and can in particular include a verification module that allows automatic image processing and automatic detection of the alignment between the translucent projection of the 3D model and the image of the target organ on the current image, thereby prompting the user that the alignment is complete or directly sending a command indicating such alignment.

[0081] Advantageously, the registration system according to the present invention is configured to implement the registration method according to the present invention.

[0082] Advantageously, according to the present invention, the registration method according to the present invention is configured to be implemented by the registration system according to the present invention.

[0083] Advantageously, according to the present invention, the processing unit includes: a module for defining a standard coordinate system of the 3D model of the organ, the standard coordinate system being defined by an origin and three axes; and a module for predicting the position and orientation of the target organ capable of defining the transformation between the 3D model of the organ in the standard coordinate system and the reference coordinate system.

[0084] Advantageously, according to the present invention, the target organ is the uterus, particularly including the uterine fundus, the anterior uterine wall, and the cervix; and the standard coordinate system is defined by an origin forming the centroid of the distal part of the uterus, a first left-right axis of the uterus, a second axis connecting the centroid of the uterine fundus and the centroid of the cervix, and a third axis being the scalar product of the first axis and the second axis.

[0085] Advantageously, according to the present invention, the module for defining the standard coordinate system is configured to:

[0086] - Calculate the principal axis of the medial part of the uterine fundus, called the uterine fundus axis;

[0087] - Calculate the principal axis of the medial part of the anterior uterine wall, called the anterior wall axis;

[0088] - Calculate the left-right axis of the uterus by the vector product of the uterine fundus axis and the anterior wall axis;

[0089] - Calculate the centroid of the uterine fundus;

[0090] - Calculate the centroid of the cervix;

[0091] - Calculate the plane defined by the plane in which each point is equidistant from the centroid of the uterine fundus and the centroid of the cervix;

[0092] - Determine the two parts of the uterus bounded by the plane, the distal part including the uterine fundus and the anterior wall, and the proximal part connected to the cervix;

[0093] - Calculate the centroid of the distal part.

[0094] Advantageously, according to the present invention, the processing unit includes: a module for pre-calculating the projection of the 3D model from the 3D model and the predicted position and orientation.

[0095] Advantageously, according to the present invention, the processing unit includes: a module for receiving a 3D model (referred to as the intraoperative 3D model) of the scene generated from the image captured by the endoscope; and the processing unit includes: a module for calculating the translation and scaling factors between the 3D model of the target organ and the intraoperative 3D model, configured to:

[0096] - Express the 3D model of the target organ and the intraoperative 3D model in a common coordinate system based on the position and orientation of the target organ on the current image relative to the reference coordinate system;

[0097] - Select an origin in the common coordinate system;

[0098] - Generate at least one radius extending from the origin and along the optical axis direction of the endoscope;

[0099] - Calculate the distances between the origin and the intersection points of each radius and the 3D model of the virtual organ;

[0100] - Calculate the distances between the origin and the intersection points of each radius and the intraoperative 3D model;

[0101] - Calculate the translation and scaling factors from the distances between the origin and the intersection points of each radius and the 3D model of the virtual organ, and the distances between the origin and the intersection points of each radius and the intraoperative 3D model.

[0102] Advantageously, according to the present invention, the processing unit includes: a module for acquiring a plurality of images, and a module for registering the 3D model with an image having a quality higher than other images selected from the plurality of images.

[0103] The advantages of these variants of the registration system according to the present invention are similar to the advantages of the variants of the registration method according to the present invention described previously.

[0104] The present invention also relates to a computer program product for registering a virtual three-dimensional model (referred to as a 3D model) of a target organ with at least one image of the target organ obtained by an endoscope in a scene. The computer program product includes program code instructions for performing the following steps when the computer program product is executed on a computer:

[0105] - A step of receiving the 3D model of the target organ;

[0106] - A step of predicting the position and orientation of the target organ relative to the reference coordinate system of the scene based on the reference position and orientation of the endoscope relative to the scene;

[0107] - A step of simulating the position and orientation of the 3D model based on the predicted position and orientation of the target organ;

[0108] - A step of displaying at least one current image from the endoscope on a display device;

[0109] - A step of superimposing a semi-transparent projection of the 3D model on at least one current image from the endoscope on the display device based on the predicted position and orientation, the projection being fixed relative to the endoscope;

[0110] - a step of receiving a command indicating alignment between a translucent projection of a 3D model and an image of a target organ on a current image;

[0111] - a step of calculating the position and orientation of the target organ on the current image relative to a reference coordinate system.

[0112] Advantageously, according to the present invention, a computer program product for registration according to the present invention includes program code instructions for performing the steps of the registration method according to the present invention when the computer program product is executed on a computer, in particular the steps of the registration method in all variants of the present invention described above.

[0113] Advantageously, according to the present invention, the registration method according to the present invention is configured to be implemented by a computer program product for registration according to the present invention.

[0114] The present invention also relates to a registration system, a registration method and a computer program product for registration, characterized by a combination of all or part of the above or below features. Description of the Drawings

[0115] Other objects, features and advantages of the present invention will become apparent by reading the following description given in a non - restrictive manner and referring to the drawings, in which:

[0116] Figure 1 is a schematic diagram of a registration system 10 integrated in a laparoscopic imaging system according to an embodiment of the present invention, the registration system 10 being in a first configuration.

[0117] Figure 2 is a schematic diagram of a registration system 10 integrated in a laparoscopic imaging system according to an embodiment of the present invention, the registration system 10 being in a second configuration.

[0118] Figure 3 is a schematic diagram of a registration method according to an embodiment of the present invention.

[0119] Figure 4 is a schematic diagram of a uterus forming the target organ of a registration method according to an embodiment of the present invention. Detailed Description of the Invention

[0120] In the drawings, for the sake of illustration and clarity, the dimensions and proportions are not strictly adhered to.

[0121] Furthermore, in all the drawings, the same, similar or analogous elements are denoted by the same reference numerals.

[0122] Figure 1 and Figure 2Schematically shows a registration system 10 integrated in a laparoscopic imaging system according to an embodiment of the present invention. The purpose of the imaging system is to acquire and output images taken in a cavity 50 of a patient's body, in this case, the cavity is the patient's abdominal cavity (or peritoneal cavity 50), especially within the scope of a laparoscopic examination process (such as laparoscopic surgery). Laparoscopic surgery can be used, for example, to operate on a target organ 52.

[0123] The laparoscopic imaging system includes a registration system 10 according to an embodiment of the present invention. The registration system 10 receives images provided, for example, by an endoscopic camera 12, which is configured to acquire images of the patient's abdominal cavity 50. The endoscope used in laparoscopic surgery is currently called a laparoscope.

[0124] The registration system includes a plurality of modules such that the method according to the present invention can be implemented. In this case, these modules are centralized in a processing unit 16. The processing unit 16 is, for example, a computer or an electronic board including a processor, which is, for example, a processor dedicated to processing images of the method according to the present invention or even a general - purpose processor, which is configured to execute program instructions for performing the method steps according to the present invention in addition to many functions.

[0125] The images acquired by the endoscope 12 are displayed on a display device (such as a display screen 18) of the registration system for the operator to use. The acquired images can be enhanced, that is, include additional information added by the laparoscopic imaging system, and this additional information can come from the registration system or other devices.

[0126] In order to be able to accurately track the additional information on the displayed images, the registration system 10 is configured to determine the position and orientation of the target organ 52 in a reference coordinate system of the scene, so as to display the additional information based on the position and orientation of the target organ 52. In particular, one purpose is to display a 3D model of the target organ on the image of the target organ 52, which requires registering the 3D model with the image of the target organ 52.

[0127] To this end, the registration system 10 implements a registration method as Figure 3 shown.

[0128] The registration method 100 includes:

[0129] - Step 110: The processing unit 16 receives a 3D model of the target organ, which is provided, for example, by an external computing device and obtained through medical imaging, especially of the Magnetic Resonance Imaging (MRI) type;

[0130] - Step 112: The processing unit 16 predicts the position and orientation of the target organ in the reference coordinate system of the scene with respect to the position and orientation of the known endoscope based on the predicted position and orientation of the endoscope relative to the scene;

[0131] - Step 114: The processing unit 16 simulates the position and orientation of the 3D model based on the predicted position and predicted orientation of the target organ.

[0132] These steps can be performed before a medical and / or surgical procedure (preoperative phase).

[0133] The registration method 100 further includes the following steps, which are preferably implemented in parallel with the medical and / or surgical procedure (intraoperative phase):

[0134] - Step 116: Display at least one current image from the endoscope 12 on the display device 18;

[0135] - Step 118: Superimpose a translucent projection 20 of the 3D model on at least one current image from the endoscope on the display device 18 based on the predicted position and orientation, and this projection is fixed relative to the endoscope 12. In Figure 1 and Figure 2 the translucent projection 20 is shown as a dashed line and is fixed relative to the image.

[0136] Before the superimposing step 118, preferably in the preoperative phase, the registration method 100 may include step 128: pre - calculate the projection of the 3D model from the 3D model and the predicted position and orientation. This phase can prepare the projection of the 3D model to be displayed on the display device 18.

[0137] After this superimposing step 118, the operator in charge of operating the endoscope can attempt to align the image 22 of the target organ on the current image with the translucent projection 20. Figure 1 shows a first position where the translucent projection 20 and the image 22 of the target organ are not aligned, Figure 2 shows a second position where the translucent projection 20 and the image 22 of the target organ are aligned. When the operator believes that the images are aligned, the operator can activate the verification device, in particular the physical verification interface 24, for example including a foot - operated pedal, so as to send a command indicating the alignment between the translucent projection of the 3D model and the image 22 of the target organ on the current image.

[0138] According to another embodiment of the present invention, the verification device may also have no physical contact, for example, using sensors to detect the operator's actions or detect voice commands. According to another embodiment of the present invention, the verification device may also be automatic, and in particular may include a verification module that allows automatic processing of images and automatic detection of the alignment between the translucent projection of the 3D model and the image of the target organ on the current image, thereby prompting the user that the alignment is completed or directly sending a command indicating such alignment.

[0139] Then, the registration method 100 includes:

[0140] - Step 120: Receive a command indicating the alignment between the translucent projection 20 of the 3D model and the image 22 of the target organ on the current image;

[0141] - Step 122: Calculate the position and orientation of the target organ on the current image relative to the reference coordinate system. When a command indicating the alignment between the projection 20 and the image 22 of the target organ is received, the calculation is performed from the current image.

[0142] The registration method 100 further includes step 124: Define a standard coordinate system for the 3D model of the organ, which is defined by an origin and three axes, preferably generated in the preoperative stage; and step 112 of predicting the position and orientation of the target organ can define the transformation between the 3D model of the organ in the standard coordinate system and the reference coordinate system.

[0143] As Figure 4 shown, Figure 4 A cross-section a) and a longitudinal section b) of the uterus 200 are schematically shown. When the target organ is the uterus 200, the uterus 200 particularly includes a uterine fundus 210, an anterior uterine wall 212, and a cervix 214. The standard coordinate system is defined by the origin G that forms the centroid of the distal part of the uterus U , the first left-right axis U of the uterus, and the second axis Y connecting the centroid G F of the uterine fundus and the centroid G C of the cervix, and the third axis defined as the scalar product of the first axis and the second axis.

[0144] In particular, step 124 of defining the standard coordinate system of the uterus 200 includes:

[0145] - A sub-step of calculating the main axis (referred to as the uterine fundus axis N F ) of the inner part of the uterine fundus 210;

[0146] - A sub-step of calculating the main axis (referred to as the anterior wall axis N W ) of the inner part of the anterior uterine wall 212;

[0147] - By the uterine fundus axis N F and the anterior wall axis N WSub-steps for calculating the cross product of the vector product to calculate the left-right axis of the uterus (not shown);

[0148] - Sub-step of calculating the centroid G of the uterine fundus F Sub-step;

[0149] - Sub-step of calculating the centroid G of the cervix C Sub-step;

[0150] - Sub-step of calculating the plane P defined by the plane equidistant from each point to the centroid G of the uterine fundus F and the centroid G of the cervix C Sub-step;

[0151] - Sub-step of determining the two parts of the uterus bounded by the plane P, where the distal part 216a includes the uterine fundus and the anterior wall, and the proximal part 216b is connected to the cervix;

[0152] - Sub-step of calculating the centroid G of the distal part U Sub-step.

[0153] According to these sub-steps, the origin and axis required to define the standard coordinate system can be obtained.

[0154] The registration method 100 may further include step 126: receiving a 3D model (referred to as the intraoperative 3D model) generated from the image captured by the endoscope; and, the registration method 100 may further include: steps of calculating the translation and scaling factors between the 3D model of the target organ and the intraoperative 3D model, including:

[0155] - Sub-step of expressing the 3D model of the target organ and the intraoperative 3D model in a common coordinate system based on the position and orientation of the target organ relative to the reference coordinate system on the current image;

[0156] - Sub-step of selecting the origin in the common coordinate system;

[0157] - Sub-step of generating at least one radius extending from the origin and extending along the optical axis direction of the endoscope;

[0158] - Sub-step of calculating the distance between the origin and the intersection points of each radius and the 3D model of the virtual organ;

[0159] - Sub-step of calculating the distance between the origin and the intersection points of each radius and the intraoperative 3D model;

[0160] - Sub-step of calculating the translation and scaling factors from the distances between the origin and the intersection points of the radius and the 3D model of the virtual organ, and the distances between the origin and the intersection points of the radius and the intraoperative 3D model.

[0161] The endoscope 12 can be configured to acquire a plurality of images, and the registration method 100 can further include a step of selecting, from these images, an image having a quality higher than that of other images for registering the 3D model.

[0162] The present invention is not limited to the described embodiments. The registration system can be integrated into different types of imaging, particularly other types of medical imaging, especially when the position and orientation of the target organ relative to the endoscope can be predicted. In addition, the target organ can be different: in laparoscopy, other organs in the abdomen where surgery is performed through a laparoscope can be the target organ, such as the liver or the kidney.

Claims

1. A method for registering a virtual three-dimensional model of a target organ with at least one image of the target organ in a scene obtained by an endoscope, the virtual three-dimensional model being referred to as a 3D model, the method comprising: - a step (110) of receiving the 3D model of the target organ; - a step (112) of predicting the position and orientation of the target organ relative to a reference coordinate system of the scene, given the position and orientation of the endoscope relative to the scene; - a step (114) of simulating the position and orientation of the 3D model based on the predicted position and predicted orientation of the target organ; - a step (116) of displaying at least one current image from the endoscope on a display device; - a step (118) of superimposing a translucent projection of the 3D model on at least one current image from the endoscope on the display device, based on the predicted position and orientation, the projection being fixed relative to the endoscope; - a step (120) of receiving a command indicating alignment between the translucent projection of the 3D model and the image of the target organ on the current image; - a step (122) of calculating the position and orientation of the target organ on the current image relative to the reference coordinate system.

2. The registration method according to claim 1, wherein The registration method comprises: a step (124) of defining a standard coordinate system of the 3D model of the organ, the standard coordinate system being defined by an origin and three axes; and the step of predicting the position and orientation of the target organ being capable of defining a transformation between the 3D model of the organ in the standard coordinate system and the reference coordinate system.

3. The registration method according to claim 2, wherein The target organ is the uterus (200), particularly comprising a uterine fundus (210), an anterior uterine wall (21), and a cervix (214); and the standard coordinate system is defined by an origin forming the centroid of the distal part of the uterus, a first left-right axis of the uterus, a second axis connecting the centroid of the uterine fundus and the centroid of the cervix, and a third axis which is the scalar product of the first axis and the second axis.

4. The registration method according to claim 3, wherein The step (124) of defining the standard coordinate system comprises: - Sub-step of calculating the principal axis of the medial portion of the uterine fundus (210), the principal axis of the medial portion of the uterine fundus (210) being referred to as the uterine fundus axis (N F ) - Sub-step of calculating the main axis of the inner part of the anterior uterine wall (212), the main axis of the inner part of the anterior uterine wall (212) being referred to as the anterior wall axis (N W ) - The sub-step of calculating the left-right axis of the uterus by the vector product of the uterine fundus axis (N F ) and the anterior wall axis (N W ); - Calculate the centroid (G F ) of the uterine fundus; - Sub-step of calculating the centroid (G C ) of the cervix; - Calculate the sub-step of the plane (P) defined by the plane equidistant from the centroid (G F ) of each point and the fundus of the uterus and the centroid (G C ) of the cervix; - a sub-step of determining two parts of the uterus bounded by the plane (P), a distal part (216a) comprising the uterine fundus and the anterior wall, and a proximal part (216b) connected to the cervix; - Calculate the centroid (G U ) of the distal portion.

5. The registration method according to any one of claims 1 to 4, characterized in that, The registration method comprises: a step (128) of pre-calculating the projection of the 3D model from the 3D model and the predicted position and orientation, before the step (118) of superimposing the projection on at least one current image from the endoscope.

6. The registration method according to any one of claims 1 to 5, characterized in that, The registration method comprises: a step (126) of receiving a 3D model of the scene generated from an image captured by the endoscope, the 3D model of the scene being referred to as an intraoperative 3D model; and The registration method comprises: a step of calculating a translation and a scaling factor between the 3D model of the target organ and the intraoperative 3D model, comprising: - A sub-step of expressing the 3D model of the target organ and the intraoperative 3D model in a common coordinate system based on the position and orientation of the target organ relative to the reference coordinate system on the current image; - A sub-step of selecting an origin in the common coordinate system; - A sub-step of generating at least one radius extending from the origin and along the optical axis direction of the endoscope; - A sub-step of calculating the distances between the origin and the intersection points of each radius and the 3D model of the virtual organ; - A sub-step of calculating the distances between the origin and the intersection points of each radius and the intraoperative 3D model; - A sub-step of calculating the translation and scaling factors from the distances between the origin and the intersection points of the radius and the 3D model of the virtual organ, and the distances between the origin and the intersection points of the radius and the intraoperative 3D model.

7. The registration method according to any one of claims 1 to 6, characterized in that, The registration method includes: obtaining a plurality of images, and a step of registering the 3D model with an image selected from the plurality of images having a quality higher than other images.

8. A system for registering a virtual three-dimensional model of a target organ with an image of the target organ, comprising: - An endoscope (12) configured to capture the image; - A display device (18) configured to display the image; - A verification device (24) configured to provide a command indicating the alignment between the semi-transparent projection of the 3D model and the image of the target organ; - A processing unit (16), The processing unit (16) includes: - A module for receiving the 3D model of the target organ; - A module for predicting the position and orientation of the target organ relative to the reference coordinate system of the scene based on the reference position and orientation of the endoscope relative to the scene; - A module for simulating the position and orientation of the 3D model based on the predicted position and orientation of the target organ; - A module for displaying at least one current image from the endoscope on the display device; - A module for superimposing a semi-transparent projection of the 3D model on at least one current image from the endoscope on the display device based on the predicted position and orientation, the projection being fixed relative to the endoscope; - A module for receiving a command indicating the alignment between the semi-transparent projection of the 3D model and the image of the target organ on the current image; - A module for calculating the position and orientation of the target organ relative to the reference coordinate system on the current image.

9. The registration system according to claim 8, characterized in that The verification device is a physical verification interface (24), including a pedal activated by the operator's foot.

10. The registration system according to claim 8 or 9, characterized in that, The processing unit includes: a module for defining a standard coordinate system of the 3D model of the organ, the standard coordinate system being defined by an origin and three axes; and a module for predicting the position and orientation of the target organ can define the transformation between the 3D model of the organ in the standard coordinate system and the reference coordinate system.

11. The registration system according to any one of claims 8 to 10, characterized in that, The target organ is the uterus (200), particularly including the uterine fundus (210), the anterior uterine wall (21), and the cervix (214); and the standard coordinate system is defined by the origin forming the centroid of the distal part of the uterus, the first left - right axis of the uterus, the second axis connecting the centroid of the uterine fundus and the centroid of the cervix, and the third axis which is the scalar product of the first axis and the second axis.

12. The registration system according to any one of claims 8 to 11, characterized in that, The module for defining the standard coordinate system is configured to: - Calculate the principal axis of the inner part of the uterine fundus (210), and the principal axis of the inner part of the uterine fundus (210) is referred to as the uterine fundus axis (N F ) - Calculate the main axis of the inner part of the anterior uterine wall (212), and the main axis of the inner part of the anterior uterine wall (212) is referred to as the anterior wall axis (N W ) - Calculate the left - right axis of the uterus through the cross - product of the vector of the uterine fundus axis (N F ) and the vector of the anterior wall axis (N W ); - Calculate the centroid (G F ) of the uterine fundus; - Calculate the centroid (G C ) of the cervix; - Calculate the plane (P) defined by the plane equidistant from the centroid (G F ) of each point and the fundus of the uterus and the centroid (G C ) of the cervix; - Determine two parts of the uterus bounded by the plane (P), where the distal part (216a) includes the uterine fundus and the anterior wall, and the proximal part (216b) is connected to the cervix; - Calculate the centroid (G U ) of the distal portion.

13. The registration system according to any one of claims 8 to 12, characterized in that, The processing unit includes: a module for pre - calculating the projection of the 3D model from the 3D model, the predicted position, and the orientation.

14. The registration system according to any one of claims 8 to 13, characterized in that, The processing unit includes: a module for receiving the 3D model of the scene generated from the images captured by the endoscope, the 3D model of the scene being referred to as the intraoperative 3D model; and the processing unit includes: a module for calculating the translation and scaling factors between the 3D model of the target organ and the intraoperative 3D model, configured to: - Express the 3D model of the target organ and the intraoperative 3D model in a common coordinate system based on the position and orientation of the target organ relative to the reference coordinate system on the current image; - Select an origin in the common coordinate system; - Generate at least one radius extending from the origin and along the optical axis direction of the endoscope; - Calculate the distances between the origin and the intersection points of each radius and the 3D model of the virtual organ; - Calculate the distances between the origin and the intersection points of each radius and the intraoperative 3D model; - Calculate the translation and scaling factors from the distances between the origin and the intersection points of the radius and the 3D model of the virtual organ, and the distances between the origin and the intersection points of the radius and the intraoperative 3D model.

15. The registration system according to any one of claims 8 to 14, characterized in that, The processing unit includes: a module for acquiring a plurality of images, and a module for registering the 3D model with the image having a quality higher than other images selected from the plurality of images.

16. A computer program product for registering a virtual three - dimensional model of a target organ with at least one image of the target organ in a scene obtained by an endoscope, the virtual three - dimensional model being referred to as the 3D model, the computer program product including program code instructions for performing the following steps when the computer program product is executed on a computer: - A step (110) of receiving the 3D model of the target organ; - A step (112) of predicting the position and orientation of the target organ relative to the reference coordinate system of the scene, given the position and orientation of the endoscope relative to the scene; - A step (114) of simulating the position and orientation of the 3D model based on the predicted position and the predicted orientation of the target organ; - A step (116) of displaying at least one current image from the endoscope on a display device; - Step (118) of superimposing a translucent projection of the 3D model on at least one current image from the endoscope on the display device based on the predicted position and orientation, the projection being fixed relative to the endoscope; - Step (120) of receiving a command indicating alignment between the translucent projection of the 3D model and the image of the target organ on the current image; - Step (122) of calculating the position and orientation of the target organ on the current image relative to the reference coordinate system.

Images