System and method for bringing a virtual 3D model into register through display in see-through

Abstract

The invention relates to a method for bringing a virtual three-dimensional model of a target organ, which model is called the 3D model, into register with at least one image of said target organ in a scene, said image being obtained by an endoscope (12), the method in particular comprising: a step of predicting a position and orientation of the target organ relative to the scene; a step of superposing at least one current endoscope image with a see-through projection of the 3D model as a function of the predicted position and orientation, on a display device (18); a step of receiving a command indicating an alignment between the see-through projection of the 3D model and the image of the target organ in the current image; and a step of computing the position and orientation of the target organ in the current image relative to the scene.

Description

[0001] DESCRIPTION

[0002] TITLE OF THE INVENTION: RECALIBRATION SYSTEM AND METHOD

[0003] OF A VIRTUAL 3D MODEL BY SEMI-TRANSPARENT DISPLAY

[0004] Technical field of the invention

[0005] The invention relates to a system and method for registering a 3D model on an optical image. In particular, the invention relates to the registration of a virtual 3D model of an organ obtained from preoperative imaging to an image of the organ obtained by optical imaging, in particular by monocular or stereoscopic endoscopic imaging. The invention can be used in particular in a context of laparoscopic and robotic imaging.

[0006] Technological background

[0007] Registration, commonly called "registration" or "alignment" in English, is an operation allowing to obtain a correspondence of position and orientation of a virtual 3D model of an object to an optical image of the same object obtained by a camera. In particular, the objective is to determine the transformation (change of reference and deformation) which must be carried out so that the 3D model of the object and the image of the object having distinct coordinate systems and distinct states are aligned. In practice, knowledge of this transformation resulting from registration makes it possible to display, in an augmented reality context, the virtual 3D model, resulting from precalculated data, on the image obtained by the camera, representing the real world. The objective is then to ensure the tracking or the location of the object on the image to display the 3D model there, after registration.

[0008] In particular, registration methods are implemented in the medical and / or surgical field to match a 3D model, known as preoperative because it is obtained upstream from one or more medical imaging techniques (e.g., radioscopy, ultrasound, MRI, CT, etc.), with a real image obtained by an optical camera, for example, an endoscope. Registration methods are generally divided into two main categories: initialization methods that aim to provide a rapid registration with relatively low accuracy so as to provide an initial solution to the registration, and refinement methods that allow the improvement of a previous registration, the first registration being typically obtained by an initialization method. Refinement methods also allow adaptation to a modification of the shape or position of the object to be tracked.

[0009] The complexity of initialization methods is the lack of a clear idea of ​​the transformation to be implemented, since it is this method that must provide the first recalibration transformations. Known initialization methods can be classified into two categories: automatic methods and manual methods.

[0010] For example, automatic methods use descriptors and visual cues or automatic correspondences to automatically calculate the registration. These methods have variable results, and in a medical and / or surgical context must be systematically validated or corrected by an operator. These automatic methods are also difficult to implement when the 3D model has no texture and is only a shape because few visual cues are available, which is generally the case for the preoperative 3D model.

[0011] Existing manual methods can be classified into several subcategories: methods requiring interactive manipulation of the pre-operative 3D model, allowing, through the use of a user interface, the application of transformations to the 3D model until alignment with the object appearing on the camera image; methods requiring the interactive selection of correspondences of known points, curves or areas on the image of the object obtained by the camera and on the 3D model.

[0012] These methods have several drawbacks.

[0013] Methods requiring model manipulation require acting on a rotation, position and possibly scale of the 3D model, which can be very complex even for an experienced person: some objects have several axes of symmetry which complicate the operation, determining the correct correspondence between scale and depth can be complex, the object may only be partially visible, etc.

[0014] Correspondence point methods can be difficult to implement if the correspondence points are difficult to identify, if the colors of the object seen in the image and the 3D model are different, etc.

[0015] Furthermore, in a medical or surgical context, these methods require operator intervention on an external computer device that may present hygiene or sterilization issues. This intervention also causes the operator to lose concentration, requiring them to shift their attention from the patient to the external device.

[0016] The inventors thus sought to provide a recalibration method for initialization that would simplify operator interaction and limit the risks of initialization errors common in existing methods. The proposed recalibration method would form a new subcategory of manual methods.

[0017] Objectives of the invention

[0018] The invention aims to provide a system and a method for registering a 3D model of a target organ on at least one image of said organ obtained by an endoscope-type camera.

[0019] The invention aims to provide, in at least one embodiment, a system and a method for recalibration making it possible to simplify the initialization of the recalibration while presenting a robust result.

[0020] The invention aims to provide, in at least one embodiment, a registration system and method that can be used for registering a 3D model on an image obtained by an endoscope.

[0021] The invention aims to provide, in at least one embodiment, a system and a method for recalibration not requiring intervention on a computer device external to the medical and / or surgical intervention. Presentation of the invention

[0022] To do this, the invention relates to a method for registering a virtual three-dimensional model of a target organ, called a 3D model, with at least one image of said target organ in a scene obtained by an endoscope, comprising: a step of receiving the 3D model of the target organ, a step of predicting a position and an orientation of the target organ relative to a reference frame of the scene in which the position and the orientation of the endoscope are known, as a function of a reference position and orientation of the endoscope relative to the scene, a step of simulating a position and an orientation of the 3D model as a function of said predicted position and said orientation of the target organ, a step of displaying at least one current image of the endoscope on a display device,a step of superimposing on at least one current image of the endoscope a semi-transparent projection of the 3D model as a function of the predicted position and orientation, on the display device, said projection being fixed relative to the endoscope, a step of receiving a command indicating an alignment between the semi-transparent projection of the 3D model and the image of the target organ on the current image, a step of calculating the position and orientation of the target organ on the current image relative to the reference frame.,

[0023] A registration method according to the invention therefore makes it possible to facilitate the initialization of the registration by proposing a solution making it possible to display in semi-transparency a projection of the virtual 3D model relative to a predefined predicted and reference position and orientation, so that the user manipulates the endoscope until the projection of the 3D model is aligned with the targeted organ. The operator can then indicate that the alignment is done by an interaction with a validation means, for example a physical validation interface or a graphical validation interface. These actions allow the user to concentrate on viewing the displayed image and managing the endoscope while remaining focused on the real world, the patient, without having to manipulate a virtual 3D model on an external system. The physical validation interface is for example a button operated by hand or a pedal operated by the foot by the operator.According to another variant of the invention, the validation means comprises an automatic validation module making it possible to automatically test the alignment of the projection of the 3D model with the targeted organ, and making it possible to suggest to a user that the alignment is correct and / or send the command indicating an alignment between the semi-transparent projection of the 3D model and the image of the target organ on the current image.

[0024] The alignment command allows the calculation of the position and orientation of the target organ on the current image relative to the reference frame from the endoscope position for the current image when the alignment command is received.

[0025] The pair formed by the position and orientation of an object or 3D model is commonly called the pose of that object.

[0026] In conventional registration techniques, the operator interacts with the virtual model to make it correspond with the real world, whereas in the invention, the operator interacts with the real world to make it correspond with the virtual one. The operation can be performed without direct interaction with equipment not used in routine medical and / or surgical practice and therefore very naturally by the operator.

[0027] The method is further particularly suitable when the target organ as visible in the current image and the 3D model of the target organ have distinct states, i.e. they have different shapes, in particular because a deformation is applied to the model or to the target organ as visible in the current image. The registration method works to calculate the pose even in the presence of deformations unlike most methods of the prior art.

[0028] The predicted position and orientation are linked to a prior knowledge of the scene and the reference position and orientation of the endoscope relative to the target organ. The predicted position and orientation can thus be calculated or pre-calculated, manually or automatically, and remain fixed during the rest of the registration process. For example, in the context of a laparoscopy, the endoscope is arranged at a relatively identical standard position and orientation at each laparoscopy and the position of the 3D model can thus be adjusted according to this standard position and orientation. Several pairs of reference position and orientation can be pre-calculated in order to adapt to several possible configurations. For example, the endoscope can be arranged at different entry points depending on the type of intervention, the target organ and the possible pathology to be treated.

[0029] The reference frame is for example a frame of the endoscope, which makes it easy to define a fixed transformation of the 3D model to obtain the projection of the fixed 3D model in the current image. Other frames can be used if the transformation from this frame to the endoscope model is known or calculable.

[0030] The current image may be a 2D image or a stereoscopic image according to the variants of the invention.

[0031] In the context of a medical or surgical procedure, the steps of receiving the 3D model of the target organ, predicting a position and orientation of the target organ and simulating a position and orientation of the 3D model can advantageously be carried out in a “preoperative” manner, i.e. upstream of the image capture procedure. The steps of displaying at least one current image, superimposing a semi-transparent projection of the 3D model, receiving a command and calculating the position and orientation of the target organ are advantageously carried out in parallel with the image capture procedure, in an “intraoperative” manner.

[0032] Advantageously and according to the invention, the method comprises a step of defining a canonical reference frame of the 3D model of the organ, said canonical reference frame being defined by an origin and three axes, and in that the step of predicting a position and an orientation of the target organ makes it possible to define a transformation of the 3D model of the organ between the canonical reference frame and the reference frame.

[0033] According to this aspect of the invention, the transformation of the canonical reference frame to the reference frame makes it possible to define the position and orientation of the 3D model in the reference frame which will be displayed in the current image as a function of the predicted position and orientation. This step can be carried out upstream, that is to say in the preoperative step, before the capture of images of the target organ by the endoscope.

[0034] Advantageously and according to the invention, the target organ is a uterus comprising in particular a uterine fundus, an anterior uterine wall and a cervix, and the canonical reference point is defined by an origin forming the center of mass of a distal part of the uterus, a first left-right axis of the uterus, a second axis connecting a center of mass of the uterine fundus and a center of mass of the cervix and a third axis scalar product of the first axis and the second axis.

[0035] According to this aspect of the invention, the registration method is particularly suitable for registering a 3D model of a uterus on a current image of a uterus as a target organ. The particular shape of the uterus allows the definition of a canonical reference point suitable for implementing the registration method.

[0036] Advantageously and according to the invention, the step of defining the canonical reference frame comprises: a sub-step of calculating a principal axis of the medial part of the uterine fundus, called the uterine fundus axis, a sub-step of calculating a principal axis of the medial part of the anterior uterine wall, called the anterior wall axis, a sub-step of calculating the left-right axis of the uterus by vector product of the uterine fundus axis by the anterior wall axis, a sub-step of calculating the center of mass of the uterine fundus, a sub-step of calculating the center of mass of the cervix, a sub-step of calculating a plane defined by the plane, each point of which is equidistant from the center of mass of the uterine fundus and the center of mass of the cervix, a sub-step of determining two parts of the uterus, delimited by the plane, a distal part comprising the uterine fundus and the anterior wall and a proximal part connected to the cervix,a sub-step of calculating the center of mass of the distal part.,

[0037] According to other variants of the invention, the target organ may be an organ other than the uterus and the canonical reference point is determined based on the general shape of this target organ.

[0038] For example, the target organ may be a liver (whose virtual 3D model is generally obtained by CT imaging) or a kidney (whose virtual 3D model is generally obtained by MRI and / or CT imaging), or other organs.

[0039] When the target organ is a liver or a kidney, the canonical reference frame can be defined by a center of mass forming the origin and by axes fixed according to the principal axes of the 3D model obtained for example by principal component analysis of the vertices of the mesh, a first axis extending along the principal axis of greatest length and being positioned parallel to the image plane of the endoscope, horizontal or vertical depending on the organ considered, a second axis extending along the principal axis of shortest length and being oriented towards the endoscope and a third axis being the scalar product of the first axis and the second axis.

[0040] Advantageously and according to the invention, the method comprises a step of pre-calculating the projection of the 3D model from the 3D model and the predicted position and orientation, upstream of the step of superimposing said projection on at least one current image of the endoscope.

[0041] According to this aspect of the invention, the projection of the 3D model makes it possible to obtain a simple two-dimensional image that can be easily combined with the current image to create the semi-transparency effect. The projection can be complete but can also consist of a silhouette or a complete or partial outline of the target organ. Advantageously and according to the invention, the method comprises a step of receiving a 3D model generated from images captured by the endoscope, called an intraoperative 3D model, and what it comprises a step of calculating translation and scale factor between the 3D model of the target organ and the intraoperative 3D model, comprising: a sub-step of expressing the 3D model of the target organ and the intraoperative 3D model in a common reference frame as a function of the position and orientation of the target organ on the current image relative to the reference frame, a sub-step of selecting an origin point in the common reference frame,a sub-step of generating at least one ray starting from the point of origin and extending in the direction of the optical axis of the endoscope, a sub-step of calculating the distance between the origin and the intersection of each ray with the 3D model of the virtual organ, a sub-step of calculating the distance between the origin and the intersection of each ray with the intraoperative 3D model, a sub-step of calculating the translation and the scale factor from each distance between the origin and the intersection of a ray with the 3D model of the virtual organ and each distance between the origin and the intersection of said ray with the intraoperative 3D model.,

[0042] According to this aspect of the invention, the calculation of translation and scale factor makes it possible to ensure the registration of the 3D model of the target organ, called the preoperative 3D model, with the target organ as modeled in the model of the scene obtained by the endoscope, called the intraoperative 3D model. In particular, these steps are relevant if a difference in scale exists between the preoperative 3D model and the intraoperative 3D model. Indeed, the calculation of the position and orientation of the target organ on the current image makes it possible to obtain the registration of the 3D model on said image but does not guarantee the registration of the preoperative 3D model with the target organ in the intraoperative 3D model due to the partial knowledge of the depth and distance on the basis of a single image.The translation and scale factor calculation steps allow the complete registration of the preoperative 3D model with the intraoperative 3D model by matching the reference points associated with each model, in particular by allowing the use of a metric reference point for each model, making it possible to know the dimensions of each object in each 3D model.

[0043] Using multiple radii and associated distances can increase the accuracy of the calculation, for example by using the median or average of the calculated distances.

[0044] Advantageously and according to the invention, the method comprises the acquisition of several images and a step of selecting from among the images an image of higher quality than the other images for the registration of the 3D model.

[0045] According to this aspect of the invention, the acquisition of several images, for example via the acquisition of a short video extract, makes it possible to guarantee the acquisition of at least one image of sufficient quality which can be used as a key image for the generation of the intraoperative 3D model, and the tracking of the 3D model of the target organ in relation to this intraoperative 3D model.

[0046] The invention also relates to a system for registering a virtual three-dimensional model of a target organ with an image of said target organ, comprising: an endoscope, configured to capture said image, a display device configured to display said image, a validation means, configured to provide a command indicating an alignment between a semi-transparent projection of the 3D model and the image of the target organ, a processing unit, the processing unit comprising: a module for receiving the 3D model of the target organ, a module for predicting a position and an orientation of the target organ relative to a reference frame of the scene in which the position and the orientation of the endoscope are known, as a function of a reference position and orientation of the endoscope relative to the scene,a module for simulating a position and an orientation of the 3D model as a function of said position and said orientation of the predicted target organ, a module for displaying at least one current image of the endoscope on a display device, a module for superimposing on at least one current image of the endoscope a semi-transparent projection of the 3D model as a function of the predicted position and orientation, on the display device, said projection being fixed relative to the endoscope, a module for receiving a command indicating an alignment between the semi-transparent projection of the 3D model and the image of the target organ on the current image, a module for calculating the position and orientation of the target organ on the current image relative to the reference frame.,

[0047] A registration system according to the invention allows the display of the projection of the 3D model on at least one current image and the reception of the command indicating the alignment of the projection of the 3D model with the target organ on the current image. The information relating to the alignment of the projection of the 3D model with the target organ on the current image allows the calculation of the position of the target organ in the current image and thus the registration of the 3D model of the target organ with its image in the current image and in future images captured by the endoscope.

[0048] A module may for example consist of a computing device such as a computer, a set of computing devices, an electronic component or a set of electronic components, or for example a computer program, a set of computer programs, a library of a computer program or a function of a computer program executed by a computing device such as a computer, a set of computing devices, an electronic component or a set of electronic components.

[0049] The validation means may preferably be a physical validation interface with the operator and may for example comprise a physical button, a lever, a pedal intended to be actuated by an operator's foot, etc. The pedal allows interaction by the operator without using his hands. The validation means may also be a graphical validation interface. The validation means may also be without physical contact, for example with a sensor making it possible to detect an operator's gesture or to detect a sound command.Finally, the validation means may also be automatic, in particular may comprise a validation module allowing automatic processing of the images and automatic detection of the alignment between the semi-transparent projection of the 3D model and the image of the target organ on the current image, so as to suggest to the user that the alignment has been made or to directly send the command indicating such an alignment.

[0050] Advantageously, the recalibration system according to the invention is configured to implement the recalibration method according to the invention.

[0051] Advantageously and according to the invention, the recalibration method according to the invention is configured to be implemented by a recalibration system according to the invention.

[0052] Advantageously and according to the invention, the processing unit comprises a module for defining a canonical reference frame of the 3D model of the organ, said canonical reference frame being defined by an origin and three axes, and in that the module for predicting a position and an orientation of the target organ makes it possible to define a transformation of the 3D model of the organ between the canonical reference frame and the reference frame.

[0053] Advantageously and according to the invention, the target organ is a uterus comprising in particular a uterine fundus, an anterior uterine wall and a cervix, and the canonical reference point is defined by an origin forming the center of mass of a distal part of the uterus, a first left-right axis of the uterus, a second axis connecting a center of mass of the uterine fundus and a center of mass of the cervix and a third axis scalar product of the first axis and the second axis.Advantageously and according to the invention, the canonical reference system definition module is configured for: a calculation of a principal axis of the medial part of the uterine fundus, called the uterine fundus axis, a calculation of a principal axis of the medial part of the anterior uterine wall, called the anterior wall axis, a calculation of the left-right axis of the uterus by vector product of the uterine fundus axis by the anterior wall axis, a calculation of the center of mass of the uterine fundus, a calculation of the center of mass of the cervix, a calculation of a plane defined by the plane, each point of which is equidistant from the center of mass of the uterine fundus and the center of mass of the cervix, a determination of two parts of the uterus, delimited by the plane, a distal part comprising the uterine fundus and the anterior wall and a proximal part connected to the cervix, a calculation of the center of mass of the distal part.

[0054] Advantageously and according to the invention, the processing unit comprises a module for pre-calculating the projection of the 3D model from the 3D model and the predicted position and orientation.

[0055] Advantageously and according to the invention, the processing unit comprises a module for receiving a 3D model of the scene generated from images captured by the endoscope, called the intraoperative 3D model, and a module for calculating translation and scale factor between the 3D model of the target organ and the intraoperative 3D model, configured for: an expression of the 3D model of the target organ and the intraoperative 3D model in a common reference frame as a function of the position and orientation of the target organ on the current image relative to the reference frame, a selection of an origin point in the common reference frame, a generation of at least one ray starting from the origin point and extending in the direction of the optical axis of the endoscope, a calculation of the distance between the origin and the intersection of each ray with the 3D model of the virtual organ, a calculation of the distance between the origin and the intersection of each ray with the intraoperative 3D model,a calculation of the translation and the scale factor from each distance between the origin and the intersection of a ray with the 3D model of the virtual organ and each distance between the origin and the intersection of said ray with the intraoperative 3D model.,

[0056] Advantageously and according to the invention, the processing unit comprises a module for acquiring several images and a module for selecting from among the images an image of higher quality than the other images for the registration of the 3D model.

[0057] The advantages of these variants of the recalibration system according to the invention are similar to the advantages of the previously described variants of the recalibration method according to the invention.

[0058] The invention also relates to a computer program product for the registration of a virtual three-dimensional model of a target organ, called a 3D model, with at least one image of said target organ in a scene obtained by the endoscope, said computer program product comprising program code instructions for the execution, when said computer program product is executed on a computer, of the following steps: a step of receiving the 3D model of the target organ, a step of predicting a position and an orientation of the target organ relative to a reference frame of the scene in which the position and the orientation of the endoscope are known, as a function of a reference position and orientation of the endoscope relative to the scene, a step of simulating a position and an orientation of the 3D model as a function of said predicted position and said orientation of the target organ,a step of displaying at least one current image of the endoscope on a display device, a step of superimposing on at least one current image of the endoscope a semi-transparent projection of the 3D model as a function of the predicted position and orientation, on the display device, said projection being fixed relative to the endoscope, a step of receiving a command indicating an alignment between the semi-transparent projection of the 3D model and the image of the target organ on the current image, a step of calculating the position and orientation of the target organ on the current image relative to the reference frame.,

[0059] Advantageously and according to the invention, the recalibration computer program product according to the invention comprises program code instructions for the execution, when said computer program product is executed on a computer, of the steps of the recalibration method according to the invention, in particular of the steps of the recalibration method of all the variants of the invention described previously.

[0060] Advantageously and according to the invention, the recalibration method according to the invention is configured to be implemented by a recalibration computer program product according to the invention.

[0061] The invention also relates to a recalibration system, a recalibration method, and a recalibration computer program product characterized in combination by all or part of the characteristics mentioned above or below.

[0062] List of figures

[0063] Other aims, characteristics and advantages of the invention will appear on reading the following description given solely for non-limiting purposes and which refers to the appended figures in which: [Fig. 1] is a schematic view of a registration system 10 according to an embodiment of the invention, integrated into a coelioscopic imaging system, in a first configuration.

[0064] [Fig. 2] is a schematic view of a registration system 10 according to an embodiment of the invention, integrated into a laparoscopic imaging system, in a second configuration.

[0065] [Fig. 3] is a schematic view of a recalibration method according to one embodiment of the invention.

[0066] [Fig. 4] is a schematic view of a uterus forming a target organ of a registration method according to one embodiment of the invention.

[0067] Detailed description of an embodiment of the invention

[0068] In the figures, scales and proportions are not strictly respected, for the purposes of illustration and clarity.

[0069] Furthermore, identical, similar or analogous elements are designated by the same references in all figures.

[0070] Figures 1 and 2 schematically represent a registration system 10 according to an embodiment of the invention, integrated into a laparoscopic imaging system. The objective of the imaging system is to enable the acquisition and dissemination of images taken in a cavity 50 of the patient's body, here a cavity of the abdomen of a patient (or abdominal cavity 50), in particular in the context of a laparoscopic procedure, for example laparoscopic surgery. The laparoscopic surgery operation may for example be intended for intervention on a target organ 52.

[0071] The laparoscopic imaging system comprises a registration system 10 according to one embodiment of the invention, receiving images provided for example by an endoscope type camera 12 configured to acquire images of the abdominal cavity 50 of the patient. The endoscope used in the context of a laparoscopic operation is commonly called a laparoscope or coelioscope.

[0072] The recalibration system comprises several modules making it possible to implement a method according to the invention, brought together here in a processing unit 16. The processing unit 16 is for example a computer or an electronic card comprising a processor, for example a processor dedicated to the processing of images of the method according to the invention or a general processor configured to, among several functions, execute in particular program instructions for the execution of the steps of the method according to the invention.

[0073] The images acquired from the endoscope 12 are displayed on a display device such as a screen 18 for viewing the registration system intended for an operator. The acquired images may be augmented, that is to say include additional information added by the laparoscopic imaging system, which may come from the registration system or other devices.

[0074] To enable accurate tracking of this additional information on the displayed image, the registration system 10 is configured to determine the position and orientation of the target organ 52 in a reference frame of the scene to enable the display of the additional information based on the position and orientation of the target organ 52. In particular, one objective is to display a 3D model of the target organ on the image of the target organ 52, which requires the registration of the 3D model with the image of the target organ 52.

[0075] To do this, the recalibration system 10 implements a recalibration method as shown in FIG. 3.

[0076] The registration method 100 comprises: a step 110 of receiving the 3D model of the target organ by the processing unit 16, for example provided by an external computer device and having been obtained by medical imaging, in particular of the magnetic resonance imaging (MRI) type. a step 112 of predicting, by the processing unit 16, a position and an orientation of the target organ relative to a reference frame of the scene in which the position and the orientation of the endoscope are known, as a function of a predicted position and orientation of the endoscope relative to the scene, a step 114 of simulating, by the processing unit 16, a position and an orientation of the 3D model as a function of said predicted position and said orientation of the target organ. These steps can be carried out upstream of the medical and / or surgical procedure (preoperative phase).

[0077] The registration method 100 also comprises the following steps, preferably implemented in parallel with the medical and / or surgical procedure (intraoperative phase): a step 116 of displaying at least one current image of the endoscope 12 on the display device 18, a step 118 of superimposing on at least one current image of the endoscope a semi-transparent projection 20 of the 3D model as a function of the predicted position and orientation, on the display device 18, said projection being fixed relative to the endoscope 12. In Figures 1 and 2, the semi-transparent projection 20 is represented in dotted lines and is fixed relative to the image.

[0078] The recalibration method 100 may comprise, upstream of the superposition step 118 and preferably in the pre-operational phase, a step 128 of pre-calculation of the projection of the 3D model from the 3D model and the predicted position and orientation. This phase makes it possible to prepare the projection of the 3D model which will be displayed on the display device 18.

[0079] Following this superposition step 118, the operator in charge of manipulating the endoscope can attempt to align the image 22 of the target organ with the current image with the semi-transparent projection 20. Figure 1 represents a first position where the semi-transparent projection 20 and the image 22 of the target organ are not aligned and Figure 2 represents a second position where the semi-transparent projection 20 and the image 22 of the target organ are aligned. When the images are aligned according to the operator, the latter can actuate a validation means, in particular a physical validation interface 24, for example comprising a pedal actuated by the foot, in order to send a command indicating an alignment between the semi-transparent projection of the 3D model and the image 22 of the target organ with the current image.

[0080] According to another embodiment of the invention, the validation means can also be without physical contact, for example with a sensor making it possible to detect a gesture of the operator or to detect a sound command. According to another embodiment of the invention, the validation means can also be automatic, in particular can comprise a validation module allowing automatic processing of the images and automatic detection of the alignment between the semi-transparent projection of the 3D model and the image of the target organ on the current image, so as to suggest to the user that the alignment is done or to directly send the command indicating such an alignment.

[0081] The registration method 100 then comprises: a step 120 of receiving the command indicating the alignment between the semi-transparent projection 20 of the 3D model and the image 22 of the target organ on the current image, a step 122 of calculating the position and orientation of the target organ on the current image relative to the reference frame. The calculation is carried out from the current image upon receiving the command indicating the alignment between the projection 20 and the image 22 of the target organ.

[0082] The registration method 100 also comprises a step 124 of defining a canonical reference frame of the 3D model of the organ, said canonical reference frame being defined by an origin and three axes, preferably carried out in the preoperative phase, and the step 112 of predicting a position and an orientation of the target organ makes it possible to define a transformation of the 3D model of the organ between the canonical reference frame and the reference frame.

[0083] As visible in Figure 4 schematically representing a uterus 200 in cross section a) and longitudinal section b), when the target organ is a uterus 200 comprising in particular a uterine fundus 210, an anterior uterine wall 212 and a cervix 214, the canonical reference point is defined by an origin Gu forming the center of mass of a distal part of the uterus, a first left-right axis U of the uterus, a second axis Y connecting a center of mass GF of the uterine fundus and a center Gc of the cervix and a third axis scalar product of the first axis and the second axis.

[0084] In particular, step 124 of defining the canonical reference frame for a uterus 200 comprises: a sub-step of calculating a principal axis of the medial part of the uterine fundus 210, called the uterine fundus axis NF, a sub-step of calculating a principal axis of the medial part of the anterior uterine wall 212, called the anterior wall axis Nw, a sub-step of calculating the left-right axis of the uterus (not shown) by vector product of the uterine fundus axis NF by the anterior wall axis Nw, a sub-step of calculating the center GF of mass of the uterine fundus, a sub-step of calculating the center Gc of mass of the cervix, a sub-step of calculating a plane P defined by the plane of which each point is equidistant from the center GF of mass of the uterine fundus and from the center Gc of mass of the cervix, a sub-step of determining two parts of the uterus, delimited by plane P,a distal part 216a comprising the uterine fundus and the anterior wall and a proximal part 216b connected to the cervix, a sub-step of calculating the center of mass Gu of the distal part.,

[0085] Following these sub-steps, the origin and axes necessary for the definition of the canonical reference frame are known.

[0086] The registration method 100 may also comprise a step 126 of receiving a 3D model generated from images captured by the endoscope, called an intraoperative 3D model, and a step of calculating translation and scale factor between the 3D model of the target organ and the intraoperative 3D model, comprising: a sub-step of expressing the 3D model of the target organ and the intraoperative 3D model in a common reference frame as a function of the position and orientation of the target organ on the current image relative to the reference frame, a sub-step of selecting an origin point in the common reference frame, a sub-step of generating at least one ray starting from the origin point and extending in the direction of the optical axis of the endoscope, a sub-step of calculating the distance between the origin and the intersection of each ray with the 3D model of the virtual organ,a sub-step of calculating the distance between the origin and the intersection of each ray with the intraoperative 3D model, a sub-step of calculating the translation and the scale factor from each distance between the origin and the intersection of a ray with the 3D model of the virtual organ and each distance between the origin and the intersection of said ray with the intraoperative 3D model.,

[0087] The endoscope 12 can be configured for the acquisition of several images and the registration method 100 can also comprise a step of selecting from among the images an image of higher quality than the other images for the registration of the 3D model.

[0088] The invention is not limited to the embodiments described. The registration system can be integrated into different types of imaging, in particular other types of medical imaging, in particular when a position and orientation of the target organ relative to the endoscope can be predicted. Furthermore, the target organ can be different: in a laparoscopic context, other organs in the abdomen that can be operated on by laparoscopy can be target organs, for example a liver or a kidney.

Claims

CLAIMS Method for registering a virtual three-dimensional model of a target organ, called a 3D model, with at least one image of said target organ in a scene obtained by an endoscope, comprising: a step (110) of receiving the 3D model of the target organ, a step (112) of predicting a position and an orientation of the target organ relative to a reference frame of the scene in which the position and the orientation of the endoscope are known, as a function of a reference position and orientation of the endoscope relative to the scene, a step (114) of simulating a position and an orientation of the 3D model as a function of said predicted position and said orientation of the target organ, a step (116) of displaying at least one current image of the endoscope on a display device,a step (118) of superimposing on at least one current image of the endoscope a semi-transparent projection of the 3D model as a function of the predicted position and orientation, on the display device, said projection being fixed relative to the endoscope, a step (120) of receiving a command indicating an alignment between the semi-transparent projection of the 3D model and the image of the target organ on the current image, a step (122) of calculating the position and orientation of the target organ on the current image relative to the reference frame. Registration method according to claim 1, characterized in that it comprises a step (124) of defining a canonical frame of reference of the 3D model of the organ, said canonical frame being defined by an origin and three axes,and in that the step of predicting a position and an orientation of the target organ makes it possible to define a transformation of the 3D model of the organ between the canonical reference frame and the reference frame. A recalibration method according to claim 2, characterized in that the target organ is a uterus (200) comprising in particular a uterine fundus (210), an anterior uterine wall (21) and a cervix (214), and the canonical reference point is defined by an origin forming the center of mass of a distal part of the uterus, a first left-right axis of the uterus, a second axis connecting a center of mass of the uterine fundus and a center of mass of the cervix and a third axis scalar product of the first axis and the second axis. A recalibration method according to claim 3, characterized in that the step (124) of defining the canonical reference frame comprises: a sub-step of calculating a principal axis of the medial part of the uterine fundus (210), called the uterine fundus axis (NF), a sub-step of calculating a principal axis of the medial part of the anterior uterine wall (212), called the anterior wall axis (Nw),a sub-step of calculating the left-right axis of the uterus by vector product of the axis (NF) of the uterine fundus by the axis (Nw) of the anterior wall, a sub-step of calculating the center (GF) of mass of the uterine fundus, a sub-step of calculating the center (Gc) of mass of the cervix, a sub-step of calculating a plane (P) defined by the plane of which each point is equidistant from the center (GF) of mass of the uterine fundus and the center (Gc) of mass of the cervix, a sub-step of determining two parts of the uterus, delimited by the plane (P), a distal part (216a) comprising the uterine fundus and the anterior wall and a proximal part (216b) connected to the cervix, a sub-step of calculating the center (Gu) of mass of the distal part. Registration method according to one of claims 1 to 4, characterized in that it comprises a step (128) of pre-calculating the projection of the 3D model from the 3D model and the predicted position and orientation,upstream of the step (118) of superimposing said projection on at least one current image of, the endoscope. Registration method according to one of claims 1 to 5, characterized in that it comprises a step (126) of receiving a 3D model of the scene generated from images captured by the endoscope, called an intraoperative 3D model, and that it comprises a step of calculating translation and scale factor between the 3D model of the target organ and the intraoperative 3D model, comprising: a sub-step of expressing the 3D model of the target organ and the intraoperative 3D model in a common reference frame as a function of the position and orientation of the target organ on the current image relative to the reference frame, a sub-step of selecting an origin point in the common reference frame, a sub-step of generating at least one ray starting from the origin point and extending in the direction of the optical axis of the endoscope, a sub-step of calculating the distance between the origin and the intersection of each ray with the model 3D of the virtual organ,a sub-step of calculating the distance between the origin and the intersection of each ray with the intraoperative 3D model, a sub-step of calculating the translation and the scale factor from each distance between the origin and the intersection of a ray with the 3D model of the virtual organ and each distance between the origin and the intersection of said ray with the intraoperative 3D model. Registration method according to one of claims 1 to 6, characterized in that it comprises the acquisition of several images and a step of selecting from among the images an image of higher quality than the other images for the registration of the 3D model. System for registration of a virtual three-dimensional model of a target organ with an image of said target organ, comprising: an endoscope (12), configured to capture said image, a display device (18) configured to display said image, a validation means (24), configured to provide a command indicating an alignment between a semi-transparent projection of the 3D model and the image of the target organ, a processing unit (16), the processing unit (16) comprising: a module for receiving the 3D model of the target organ, a module for predicting a position and an orientation of the target organ relative to a reference frame of the scene in which the position and the orientation of the endoscope are known, as a function of a reference position and orientation of the endoscope relative to the scene, a module for simulating a position and an orientation of the 3D model as a function of said predicted position and said orientation of the target organ, a module for displaying at least one current image of the endoscope on a display device,a module for superimposing on at least one current image of the endoscope a semi-transparent projection of the 3D model as a function of the predicted position and orientation, on the display device, said projection being fixed relative to the endoscope, a module for receiving a command indicating an alignment between the semi-transparent projection of the 3D model and the image of the target organ on the current image, a module for calculating the position and orientation of the target organ on the current image relative to the reference frame. Registration system according to claim 8, characterized in that the validation means is a physical validation interface (24) comprising a pedal intended to be actuated by an operator's foot., Registration system according to one of claims 8 or 9, characterized in that the processing unit comprises a module for defining a canonical reference frame of the 3D model of the organ, said canonical reference frame being defined by an origin and three axes, and in that the module for predicting a position and an orientation of the target organ makes it possible to define a transformation of the 3D model of the organ between the canonical reference frame and the reference frame. Registration system according to one of claims 8 to 10, characterized in that the target organ is a uterus (200) comprising in particular a uterine fundus (210), an anterior uterine wall (21) and a cervix (214), and the canonical reference frame is defined by an origin forming the center of mass of a distal part of the uterus, a first left-right axis of the uterus,a second axis connecting a center of mass of the uterine fundus and a center of mass of the cervix and a third axis scalar product of the first axis and the second axis. Registration system according to one of claims 8 to 11, characterized in that the module for defining the canonical reference frame is configured for: a calculation of a principal axis of the medial part of the uterine fundus (210), called the uterine fundus axis (NF), a calculation of a principal axis of the medial part of the anterior uterine wall (212), called the anterior wall axis (Nw), a calculation of the left-right axis of the uterus by vector product of the uterine fundus axis (NF) by the anterior wall axis (Nw), a calculation of the center (GF) of mass of the uterine fundus, a calculation of the center (Gc) of mass of the cervix, a calculation of a plane (P) defined by the plane of which each point is equidistant from the center (GF) of mass of the uterine fundus and from the center (Gc) of mass of the cervix, a determination of two parts of the uterus,delimited by the plane (P), a distal part (216a) comprising the uterine fundus and the anterior wall and a proximal part (216b) connected to the cervix, a calculation of the center (Gu) of mass of the distal part., ZI Registration system according to one of claims 8 to 12, characterized in that the processing unit comprises a module for pre-calculating the projection of the 3D model from the 3D model and the predicted position and orientation. Registration system according to one of claims 8 to 13, characterized in that the processing unit comprises a module for receiving a 3D model of the scene generated from images captured by the endoscope, called an intraoperative 3D model, and a module for calculating translation and scale factor between the 3D model of the target organ and the intraoperative 3D model, configured for: an expression of the 3D model of the target organ and the intraoperative 3D model in a common reference frame as a function of the position and orientation of the target organ on the current image relative to the reference frame, a selection of an origin point in the common reference frame,a generation of at least one ray starting from the point of origin and extending in the direction of the optical axis of the endoscope, a calculation of the distance between the origin and the intersection of each ray with the 3D model of the virtual organ, a calculation of the distance between the origin and the intersection of each ray with the intraoperative 3D model, a calculation of the translation and the scale factor from each distance between the origin and the intersection of a ray with the 3D model of the virtual organ and each distance between the origin and the intersection of said ray with the intraoperative 3D model. Registration system according to one of claims 8 to 14, characterized in that the processing unit comprises a module for acquiring several images and a module for selecting from among the images an image of higher quality than the other images for the registration of the 3D model., Computer program product for the registration of a virtual three-dimensional model of a target organ, called a 3D model, with at least one image of said target organ in a scene obtained by an endoscope, said computer program product comprising program code instructions for the execution, when said computer program product is executed on a computer, of the following steps: a step (110) of receiving the 3D model of the target organ, a step (112) of predicting a position and an orientation of the target organ relative to a reference frame of the scene in which the position and the orientation of the endoscope are known, as a function of a reference position and orientation of the endoscope relative to the scene, a step (114) of simulating a position and an orientation of the 3D model as a function of said predicted position and said orientation of the target organ,a step (116) of displaying at least one current image of the endoscope on a display device, a step (118) of superimposing on at least one current image of the endoscope a semi-transparent projection of the 3D model as a function of the predicted position and orientation, on the display device, said projection being fixed relative to the endoscope, a step (120) of receiving a command indicating an alignment between the semi-transparent projection of the 3D model and the image of the target organ on the current image, a step (122) of calculating the position and orientation of the target organ on the current image relative to the reference frame.,

Images