Augmented Reality Navigation System for Medical Robots

Abstract

The augmented reality navigation system (10) is used to assist a medical practitioner (31) during a surgical procedure on an anatomical structure of a patient (30). The procedure is planned on preoperative medical images. The navigation system includes a camera (11) intended to be worn on the head of the medical practitioner, a display device (12) for displaying real images acquired by the camera and augmented reality content overlaid on the real images in real time, and a control unit (13) connected to the camera and the display device. The control unit is configured to generate a predictive model of the motion of markers (20) placed near the anatomical structure based on the motion followed by the markers during one or more respiratory cycles of the patient. The predictive model is used to determine the appropriate moment for performing the planned surgical procedure on the preoperative images and / or to display a three-dimensional anatomical model of the anatomical structure in augmented reality.

Description

Technical Field

[0001] Field of the Invention The present invention relates to the field of devices used in the context of robot-assisted minimally invasive surgery. In particular, the present invention relates to an augmented reality navigation system intended to cooperate with a medical robot to assist a medical practitioner during a surgical procedure.

Background Art

[0002] Prior Art Minimally invasive surgery essentially consists of inserting one or more medical devices into the relevant biological structure of a patient. For example, minimally invasive surgery can aim to take a sample or excise a lesion such as a tumor.

[0003] In minimally invasive surgery, the relevant biological structure of the patient is usually not visible to the naked eye of the medical practitioner. The insertion of the medical device is usually guided by a medical image. To improve the accuracy with which the medical device can be inserted, minimally invasive procedures can be assisted by a robotic device. To make the relevant biological structure visible to the medical practitioner, it is possible to convert the content of a medical image of the relevant biological structure into augmented reality content overlaid on the patient's body. However, to monitor the insertion of the medical device into the relevant biological structure during the surgical procedure, it is necessary to acquire numerous medical images, and this involves using invasive means (e.g., an endoscope) or, otherwise, using a medical imaging device that exposes the patient to radiation over the duration of the surgical procedure.

[0004] In the case of certain soft organs, the medical image cannot reproduce the movement associated with breathing due to local deformation of the organ as a result of the insertion of the medical device or, otherwise, due to involuntary movement of the patient at the moment of the procedure. Thus, the position of the lesion within the relevant biological structure can differ between the time of acquisition of the medical image and the time of the procedure. If the insertion of the medical device is planned based on the medical image, there is a risk that the lesion may not be reached precisely by the medical device.

[0005] Therefore, current means for assisting medical practitioners during minimally invasive medical treatment cannot enable medical practitioners to consider, in a simple and reliable manner, the movement of a patient associated with the patient's respiration, internal deformation of the living structure, or involuntary movement during the period of medical treatment. SUMMARY OF THE INVENTION MEANS FOR SOLVING THE PROBLEM

[0006] SUMMARY OF THE INVENTION An object of the present invention is to overcome all or some of the disadvantages of the prior art (in particular those described above).

[0007] To that end, and according to a first aspect, the present invention proposes an augmented reality navigation system for assisting a medical practitioner when performing a medical treatment on a living structure of a patient. The system particularly includes the following: - A camera for acquiring a real image, the camera being intended to be worn above the head by a medical practitioner, - A display device for displaying, in real time, the real image acquired by the camera together with augmented reality content overlaid on the real image, - A control unit connected to the camera and the display device.

[0008] The control unit of the augmented reality navigation system is configured to do the following: - Detect the position of a marker placed on a patient in the vicinity of the living structure on the real image, - Record the movement followed by the marker during a recording period corresponding to one or more respiratory cycles of the patient, - Determine the position of the marker at a first instant belonging to the recording period, the first instant corresponding to the instant of acquisition of a pre-medical image of the living structure of the patient by a medical imaging device, - Formulate a prediction model of the movement of the marker from the movement followed by the marker during the recording period.

[0009] In this application, and unless otherwise indicated, the term "position" generally encompasses the concepts of both the location and orientation of an object within a given reference system, which is typically a three-dimensional coordinate system. The term "pose" is used in the English literature to denote this combination of the position and orientation of an object in space. Thus, in this application and unless otherwise indicated, the term "position" is equivalent to the term "position / orientation".

[0010] Minimally invasive medical treatment essentially consists of inserting one or more medical devices into the relevant biological structure of a patient (such as a soft organ like the liver, lung, kidney, or a rigid structure like bone) for, for example, taking a specimen, dissecting a lesion (such as a tumor), positioning an implant (such as a screw, plate, artificial organ), or inserting another material (such as cement or an artificial disc). In this type of treatment, the relevant biological structure is not visible to the naked eye.

[0011] Advantageously, the camera of the augmented reality navigation system is mounted above the head of the medical practitioner such that the camera maintains the direction of the direct field of view of the patient (and in particular such that the medical practitioner cannot obstruct this direct field of view).

[0012] The augmented reality navigation system enables information to be displayed on a display device in the form of augmented reality content. This means that this information is overlaid on the real image and would otherwise not be visible to the medical practitioner. Advantageously, the display device can also be mounted above the head of the medical practitioner in front of the eyes of the medical practitioner (for example, in the form of a headset or the mask / facepiece of glasses / goggles). Such a device enables the medical practitioner to read the information from the display device without turning their gaze away from the patient. This also makes it possible to limit the size of the room in which the treatment is being performed. However, nothing prevents the display device from being, in a variant form, a screen visible to some individuals present in the room where the treatment is being performed (for example, a screen fixed to the operating table above the patient).

[0013] The control unit can also be worn on the head by a medical practitioner. Alternatively, the control unit can belong to an entity different from the entity worn on the head by the medical practitioner. The control unit is connected to the camera and the display device via, for example, wired communication means or wireless communication means.

[0014] The marker placed on the patient in the vicinity of the biological structure can be identified by an augmented reality navigation system thanks to one or more optical markers of known geometry. This marker also includes a radiation-opaque marker of known geometry that is visible on a medical image obtained by a medical imaging device (e.g., by computed tomography, magnetic resonance, ultrasound, tomography, positron emission tomography, etc.). The pre-diagnostic medical image is obtained, for example, at the moment when the patient's breathing is interrupted. However, in another example, there is nothing to prevent the pre-diagnostic medical image from being obtained at the moment when the patient is breathing freely (e.g., when the breathing reaches a stationary state at the end of inspiration or at the end of expiration) (the movement related to the patient's breathing can be ignored for 2-3 seconds when reaching one of these stationary states).

[0015] It is possible to plan a surgical treatment based on the pre-diagnostic medical image, and in particular, it is possible to define the position that the medical device needs to adopt with respect to the position of the marker for performing the surgical treatment. However, the position of the biological structure can vary with respect to the position of the marker due to the patient's breathing, internal deformation of the organs, or movement due to the involuntary movement of the patient at the moment of the treatment. Therefore, when the surgical treatment is performed, it is appropriate to ensure that the patient is in the same position or the same phase of the breathing cycle as the position at the moment when the pre-diagnostic medical image was obtained.

[0016] In the present invention, the control unit is configured to record the movement tracked by the marker during one or more breathing cycles of the patient in order to determine the position of the marker at the moment when the pre-diagnostic medical image was obtained and to formulate a prediction model of the movement of the marker from the recorded movement.

[0017] Such measures enable, in particular, planning a surgical procedure on a pre-operative medical image and ensuring that at the moment of insertion of the medical device the marker is in the same position as at the moment the pre-operative medical image was acquired (or, in other words, ensuring that at the moment of insertion of the medical device the patient is in the same position or phase of the respiratory cycle as at the moment the pre-operative medical image was acquired).

[0018] Furthermore, the prediction model of the movement of the marker can be used to continuously and in real time update the three-dimensional anatomical model of the living structure (generated from the pre-operative medical image). Next, this anatomical model can be displayed on a display device in the form of augmented reality content overlaid on the real image.

[0019] In its simplest form, the prediction model corresponds to the movement of the marker recorded during a respiratory cycle or several respiratory cycles. However, advantageously, the prediction model can be formulated by using statistical calculations regarding the positions adopted by the marker during several consecutive respiratory cycles. It is particularly contemplated to rely on machine learning algorithms to formulate the prediction model.

[0020] The medical imaging device is a different entity from the augmented reality navigation system. There are various conceivable synchronization means for the augmented reality navigation system to detect the moment the pre-operative medical image was acquired.

[0021] The augmented reality navigation system detects the position of the marker on the real image. This detection of the position of the marker can likely be facilitated or made more reliable by an optical navigation device (such as a stereo camera, a time-of-flight camera, etc.) or an electromagnetic navigation device.

[0022] In certain embodiments, the invention may further include one or more of the following features, considered in isolation or in any technically possible combination.

[0023] In certain embodiments, the control unit is further configured to: - Determine from a prediction model a second moment corresponding to a candidate moment of insertion of a medical device into the patient's living body structure; - Compare the position of the marker at the first moment with the position of the marker at the second moment; and - Display the result of the comparison on a display device in the form of augmented reality content overlaid on the real image.

[0024] Comparing the position of the marker at the first moment (corresponding to the moment of acquisition of the pre-treatment medical image during the recording period) with the position of the marker at the second moment (corresponding to the insertion of the medical device into the living body structure) makes it possible to check whether the marker is in the same position as the position occupied by the marker at the moment the pre-treatment medical image was acquired (or, in other words, to check whether the patient is in the same phase of the breathing cycle as when the pre-treatment medical image was acquired at the moment of insertion of the medical device). The second moment corresponds, for example, to the moment when the patient's breathing is interrupted. However, nothing prevents the patient from breathing freely during the treatment. The second moment can be determined by using the prediction model by synchronizing the movement of the marker caused by the patient's breathing with the prediction model.

[0025] In certain embodiments, the control unit is further configured to receive a pre-treatment medical image from a medical imaging device, generate a three-dimensional anatomical model of the patient's living body structure from the pre-treatment medical image, and display the anatomical model on a display device in the form of augmented reality content overlaid on the real image. The position of the anatomical model overlaid on the real image is updated continuously and in real time according to the prediction model of the movement of the marker and according to the biomechanical model of the anatomical structure of the human body.

[0026] ​What is meant by a "biomechanical model" is a mathematical model of the various anatomical structures of the human body (and thus of the patient within the anatomical region being considered) (muscles, tendons, skeleton, organs, vascular network, etc.), which enables the deformation of said anatomical structures to be modeled together with the mechanical interactions between said anatomical structures. Thus, such a biomechanical model makes it possible, in particular, to determine the deformations and mechanical interactions (and thus movements) of the patient's internal anatomical structures caused, for example, by modifications to the patient's skin, modifications to the position of the blood vessels of an organ, modifications to the skin of an organ, etc. Such modifications can be caused, for example, by the patient's breathing (movement of the organs caused by the movement of the ribs and diaphragm), by a change in the patient's position (movement of the organs caused by gravity), by contact with a medical device (local deformation), etc.

[0027] The movement of the markers indicates the movement of the patient's ribs, such as that caused by the patient's breathing. Thus, the biomechanical model makes it possible to define how the position of the anatomical model is affected by these movements during the course of the patient's breathing cycle.

[0028] By using such measures, the medical practitioner can view the biological structure on the display device "through" the outer skin of the patient, even though the biological structure is not visible to the naked eye as it is inside the patient's body.

[0029] In a particular embodiment, the control unit is further configured to display, on the anatomical model, in the form of augmented reality content overlaid on a real image, a predetermined path that is followed by the medical device and updated in real time according to the position of the anatomical model.

[0030] The path followed by the medical device and determined, for example, from a pre-operative medical image is data from the surgical treatment plan. This path in particular defines the positions that the medical device will adopt with respect to the positions of the markers in order to perform said surgical treatment. This path includes, for example, the target points reached within the region to be treated (e.g., a tumor) within the living structure and the entry points of the medical device on the surface of the living structure.

[0031] In a particular embodiment, the predetermined path followed by the medical device is predefined on a pre-operative medical image, and the control unit is configured to receive said pre-operative medical image and realign the pre-operative image to the pre-operative medical image in order to display the path on the anatomical model.

[0032] What is meant by a "pre-operative" image is an image in which the condition was diagnosed several days, weeks, or months prior to the surgical treatment for treating the condition. What is meant by a "pre-treatment" image is an image acquired at the moment of the surgical treatment when the patient is placed on the treatment table but before the surgical act is performed (i.e., before the medical device is inserted).

[0033] In a particular embodiment, the system further includes an interaction device that enables a medical operator to target a specific location on the anatomical model, and the predetermined path followed by the medical device is defined by the medical operator by using the interaction device.

[0034] In a particular embodiment, the control unit is configured to segment at least one element on the anatomical model from among the following elements: - various anatomical structures within the living structure, - the region to be treated within the living structure, - the dissection region estimated from the surgical treatment parameters, - The detachment margin of the region to be treated within the biological structure, which is determined by comparing the region to be treated with the estimated detachment region, and the segmented elements are configured to be displayed on the anatomical model in the form of augmented reality content overlaid on the real image. Next, the segmented elements are updated in real time according to the position of the anatomical model.

[0035] What is meant by "anatomical structure" is, for example, organs, bones, blood vessels, etc. The region to be treated corresponds to, for example, the tumor to be detached. The detachment region can be estimated according to surgical treatment parameters such as, for example, the type of treatment, the type of medical device used for treatment, the duration of treatment, the power of treatment, etc.

[0036] In a particular embodiment, the control unit is configured to display at least one virtual configuration object regarding surgical treatment parameters on the display device in the form of augmented reality content. The system includes an interaction device that enables a medical practitioner to interact with the virtual configuration object to select a specific value of the parameter.

[0037] The virtual configuration object corresponds to, for example, a menu, button, multi - option selection list, etc. that enables the definition of a specific parameter value of the treatment to be performed.

[0038] In a particular embodiment, the display device is intended to be worn at eye level by a medical practitioner.

[0039] In a particular embodiment, the control unit is configured to correct the position of the augmented reality content according to the position of the medical practitioner's head.

[0040] In a particular embodiment, the position of the medical practitioner's head is determined by using a medical practitioner head movement sensor or directly determined from the position of a marker on the real image.

[0041] A second aspect of the present invention relates to an assembly formed by an augmented reality navigation system according to any one of the foregoing embodiments and a medical robot for assisting a medical practitioner in a surgical treatment on a living body structure of a patient. The medical robot includes a movable base, an articulated arm, and a control unit. One end of the articulated arm is fixed to the movable base, and the other end has a tool guide intended to hold a medical device. The control unit of the medical robot is configured to determine a configuration of the articulated arm that enables a surgical act to be performed by the medical device along a predetermined path. The configuration of the articulated arm is related to the position of the marker and is determined according to information transmitted by the navigation system.

[0042] The augmented reality navigation system and the medical robot cooperate with each other. The augmented reality navigation system and the medical robot include, for example, communication means for exchanging messages. These communication means can be of a wired type or a wireless type.

[0043] The augmented reality navigation system transmits medical robot information related to the position of the marker so that the medical robot can arrange its articulated arm in a configuration that enables the medical device to perform a surgical act along a predetermined path. The path followed by the medical device is, for example, data from a surgical treatment plan determined based on pre-treatment medical images.

[0044] The medical act can be performed by a medical practitioner (and in this case, the articulated arm of the medical robot is mainly used to guide the medical device to assist the medical practitioner in performing the medical act), or can be directly performed by the medical robot.

[0045] In a particular embodiment, the present invention may further include one or more of the following features considered in isolation or in any technically possible combination.

[0046] In a particular embodiment, the information related to the position of the marker is an indication that the result of the comparison between the position of the marker at a first moment (corresponding to the moment of acquisition of the pre-diagnostic medical image during the recording period) and the position of the marker at a second moment (corresponding to a candidate moment of insertion of the medical device into the living structure) satisfies a specific criterion.

[0047] For example, the patient's respiration can be interrupted to insert the medical device. If, at the moment when the respiration is interrupted, the position of the marker is close enough to the position of the marker at the moment when the pre-diagnostic medical image was acquired, the augmented reality navigation system sends an instruction to the medical robot that the diagnosis can occur as planned based on the pre-diagnostic medical image (since the patient is at the same phase of the same respiration cycle as at the moment when the pre-diagnostic medical image was acquired).

[0048] This instruction can be sent by the augmented reality navigation system after the medical practitioner has verified the result of the comparison, or otherwise automatically, for example, if the difference in position is less than a predetermined threshold.

[0049] In a particular embodiment, the information related to the position of the marker corresponds to a prediction model that indicates the position of the anatomical model during the patient's respiration cycle. The control unit of the medical robot is configured to continuously and in real time adjust the configuration of the multi-joint arm according to the prediction model that indicates the position of the anatomical model.

[0050] By such measures, the multi-joint arm is positioned in a configuration that enables the surgical procedure to be performed at any moment within the patient's respiration cycle.

[0051] In a particular embodiment, the control unit of the augmented reality navigation system is configured to model the outer skin of the patient's body, to determine the position of the multi-joint arm or the tool guide of the medical robot, and to detect a situation where there is a risk of collision if the distance between the tool guide and the outer skin of the patient is less than a predetermined threshold.

[0052] The outer skin of the patient's body is modeled, for example, based on the detection of the contour of the patient's body in the real image. Such measures may enable the continuous and real-time measurement of the distance between the multi-joint arm and the patient and the distance between the tool guide and the patient. Next, the current measures can be adopted to avoid unwanted contact between the multi-joint arm or the tool guide and the patient when positioning the multi-joint arm (for example, when the patient's weight is heavier than the weight estimated when planning the treatment).

[0053] In a particular embodiment, the control unit immediately blocks the movement of the multi-joint arm if the distance between the multi-joint arm or the tool guide and the outer skin of the patient is insufficient (for example, less than 5 cm or even less than 1 cm).

[0054] In a particular embodiment, the control unit of the augmented reality navigation system is configured to detect a situation where there is a risk of injury from the medical device if the deviation of the position of the marker from the predicted model of the movement of the marker exceeds a predetermined threshold. Such measures make it possible to detect situations where the patient makes an unexpected movement during the insertion of the medical device or when the medical device has been inserted but not released from the tool guide. Such situations can in fact lead to the patient being damaged by the medical device (for example, damage to healthy tissue of the biological structure or another part of the patient's body by the medical device). At this time, current measures can be adopted to avoid injuring the patient if such a situation is detected.

[0055] In a particular embodiment, the tool guide of the medical robot includes an actuator that enables the medical device to be instantaneously released. The actuator is commanded by the control unit of the medical robot. The control unit of the augmented reality navigation system is configured to send a command to instantaneously release the medical device to the control unit of the medical robot if a risk of injury situation is detected.

[0056] In certain embodiments, the multi-joint arm has at least six degrees of freedom such that several different candidate configurations of the multi-joint arm enable a surgical procedure to be performed by a medical device along a predetermined path. The control unit of the augmented reality navigation system is configured to display the candidate configurations on a display device in the form of augmented reality content overlaid on a real image and to receive an instruction from a medical practitioner related to the selection of one specific configuration from among the various candidate configurations.

[0057] The various candidate configurations are overlaid on the real image and enable the medical practitioner to select one specific configuration such that the presence of the medical robot does not or hardly interferes during the surgical treatment.

[0058] In certain embodiments, when a medical device is inserted into a patient's body along a predetermined path, the control unit of the augmented reality navigation system is configured to display a part of the medical device inserted into the patient's body on a display device in the form of augmented reality content overlaid on a real image. To achieve this, the position of the medical device is detected by the control unit of the augmented reality navigation system on a real image acquired by a camera, for example, by using a known algorithm of the "Computer Vision" type.

[0059] By using such an arrangement, the medical practitioner can continuously and in real time monitor the insertion of the medical device into the relevant biological structure of the patient as long as a part of the medical device that is not visible to the naked eye is overlaid on the real image.

[0060] In certain embodiments, the control unit of the augmented reality navigation system is configured to calculate the distance between the medical device and a target point on the biological structure and to display this on a display device in the form of augmented reality content overlaid on a real image and / or to detect the moment when the medical device reaches the target point.

[0061] Description of the Drawings The present invention will be better understood by reading the following description, given by way of non-limiting example and made with reference to FIGS. 1-10.

Brief Description of the Drawings

[0062]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Modes for Carrying Out the Invention

[0063] In these figures, references that remain the same from one figure to another refer to identical or similar elements. For the sake of clarity, the elements depicted are not necessarily drawn to the same scale unless otherwise stated.

[0064] Detailed Description of an Embodiment of the Invention FIG. 1 schematically depicts an embodiment of an augmented reality navigation system 10 for assisting a medical practitioner 31 in performing a surgical procedure on a patient 30 lying on a diagnostic table 32 with respect to the relevant biological structure.

[0065] The augmented reality navigation system 10 includes a camera 11 for acquiring real images. The camera 11 is intended to be worn by the medical practitioner above his or her head. In the example considered and shown in FIG. 1, the system 10 takes the form of an augmented reality headset. Accordingly, the camera 11 is incorporated into the headset. Such an arrangement enables the navigation system 10 to maintain the direct field of view direction of the camera 11 with respect to the patient 30. In particular, the medical practitioner 31 cannot obstruct this direct field of view direction (which could be obstructed if it were mounted on a pillar in the room where the procedure is being performed).

[0066] The augmented reality navigation system 10 also includes a display device 12 for displaying in real time the real images acquired by the camera 11 together with augmented reality content overlaid on the real images. Advantageously, the display device 12 can also be worn by the medical practitioner above his or her head directly in front of the eyes of the medical practitioner 31, as is the case in the example shown in FIG. 1 where the display device is a display screen 12a incorporated into the augmented reality headset. Such an arrangement enables the medical practitioner 31 to read the information on the display device without having to divert his or her gaze from the patient 30. Alternatively or additionally, the display device 12 can also take the form of a display screen 12b fixed to, for example, the diagnostic table 32 and visible to some individuals present in the room where the procedure is being performed.

[0067] The augmented reality navigation system 10 also includes a control unit 13 connected to the camera 11 and the display device 12. Conventionally, the control unit can be connected to the camera and the display device 12 via wired communication means (especially when all of these elements are incorporated into an augmented reality headset worn by a medical practitioner), or can be connected to the camera and the display device 12 via wireless communication means (for example, when the display device 12b and / or the control unit 13 are located at a certain distance).

[0068] The control unit 13 includes one or more processors for executing an augmented reality algorithm that operates based on the "Computer Vision" principle to detect individual elements in the real world and then superimpose virtual content thereon. Alternatively or additionally, the control unit 13 includes one or more programmable logic circuits (such as FPGA, PLD, etc.) and / or one or more application-specific integrated circuits (ASIC) and / or a group of individual electronic components, etc. for executing these algorithms. In other words, the control unit is composed of software and / or hardware means for executing these augmented reality algorithms.

[0069] The control unit 13 is particularly configured to detect the position of the marker 20 placed on the patient 30 in the vicinity of the living body structure on the real image.

[0070] The control unit 13 can also be configured to detect the position of the medical device operated by the medical practitioner, or the position of the medical device held by the tool guide fixed to the end of the multi-joint arm of the medical robot used to assist the medical practitioner on the real image.

[0071] FIG. 2 schematically depicts such a marker 20 that is intended to be placed on a patient in the vicinity of the biological structure. In the example considered and shown in FIG. 2, the marker 20 includes three optical markers 21 whose geometry and respective positions are known. Such an arrangement enables the position of the marker 20 in the fixed reference system of the navigation system 10 to be determined with high precision. This reference system is a three-dimensional coordinate system defined based on a fixed object such as a diagnostic table 32, for example.

[0072] The marker 20 also includes a radiation-opaque marker 22 whose geometry and respective positions are known and which is visible in a pre-diagnostic medical image acquired by a medical imaging device (for example, by using computed tomography, magnetic resonance, ultrasound, tomography, positron emission tomography, etc.). Therefore, it is possible to accurately determine the position of the target anatomical region of the biological structure relative to the position of the marker 20 and to plan a surgical procedure based on the pre-diagnostic medical image. In particular, it is possible to define the position that a medical device will adopt relative to the position of the marker 20 for performing a surgical procedure.

[0073] However, the position of the biological structure can vary relative to the position of the marker 20 as a result of movements associated with the patient's breathing, internal organ deformations, or involuntary patient movements at the moment of the procedure. Therefore, it is appropriate to ensure that the patient is in the same position or the same phase of the breathing cycle as when the pre-diagnostic medical image was acquired when the surgical procedure is performed.

[0074] To achieve this objective, the control unit 13 is configured to determine the position of the marker 20 at the moment when the pre-diagnostic medical image was acquired, so as to record the movement followed by the marker 20 during one or more respiratory cycles of the patient 30 (implicitly, this means that the control unit is configured to, firstly, determine the moment of acquisition of the pre-diagnostic medical image and, secondly, determine the position of the marker at said moment of capture), and to formulate a prediction model of the movement of the marker from the recorded movement. Next, the prediction model of the movement of the marker enables the medical device to determine the second moment when it is inserted into the relevant biological structure of the patient. The second moment is determined in such a way that the position of the marker 20 at the second moment is approximately the same as the position of the marker 20 at the first moment. This makes it possible to ensure that at the moment of insertion of the medical device (the second moment), the patient is in the same position or in the same phase of the respiratory cycle as at the moment when the pre-diagnostic medical image was acquired (the first moment). The control unit 13 may be configured to compare the position of the marker at the first moment with the position of the marker at the second moment and to display the result of this comparison on the display device in the form of augmented reality content overlaid on the real image.

[0075] Figure 3 depicts, as an example, a recording of the movement 40 followed by the marker 20 during a recording period of a predetermined period corresponding to several respiratory cycles of the patient. Each point corresponds to the position adopted by the marker 20 over time within the XY plane of the coordinate system of the navigation system 10. It may be noted that in this example the movement of the marker 20 mainly follows the axis 44 plotted by the dotted line in Figure 3.

[0076] The movement 40 of the marker 20 indicates the movement of the patient's rib cage as caused by the patient's breathing. For a better interpretation of the movement of the marker, and by analogy with the patient's breathing cycle, it is preferable to obtain a one-dimensional curve showing the oscillatory movement of the marker over time. There are various ways to enable obtaining this one-dimensional curve. For example, it may be considered that the movement of the marker is mainly in the vertical direction, and thus only the Y-axis is taken into account. However, in such a case, a part of the amplitude of the movement of the marker will be lost. In another example, it may be considered to analyze the position of the marker in terms of its significant components. The position of the marker can be displayed, in particular, according to the significant components corresponding to the main axis 44 of the movement of the marker.

[0077] Figure 4 depicts, with a continuous line, the movement 40 of the marker 20 during the recording period 42 along the main axis 44 over time. The position p of the marker 20 along the main axis 44 is depicted along the vertical axis, and time is depicted on the horizontal axis. The recording period includes several breathing cycles 41 of the patient.

[0078] Figure 4 also depicts, with a broken line, a prediction model 43 of the movement of the marker 20, which is formulated from the movement 40 performed by the marker 20 during the recording period 42. In the example considered and shown in Figure 4, the prediction model 43 is a simple repetition of the movement 40 of the marker 20 recorded during the recording period 42. However, it should be noted that other methods for obtaining a prediction model of the movement of the marker can be envisioned. For example, it may be considered to model a breathing cycle corresponding to the average of the breathing cycles observed during the recording period. In a variant form, a machine learning algorithm can be used to formulate a prediction model of the movement of the marker from the movement of the marker recorded during the recording period.

[0079] Figure 5 shows a determination of a first instant t1 belonging to the recording period 42, corresponding to the instant at which a pre-diagnostic medical image of the patient's relevant biological structure was acquired. It should be recalled that the pre-diagnostic medical image is acquired by a medical imaging device that does not form part of the navigation system according to the present invention.

[0080] There are probably various ways that could be used so that the control unit 13 of the augmented reality navigation system 10 can determine the first moment t1. According to the first example, the medical imaging device can be connected to the control unit 13 of the navigation system 10 via, for example, wired communication means or wireless communication means. In this case, the medical imaging device can send an instruction to the control unit 13 at the moment when the pre-diagnostic medical image is acquired. According to the second example, the position of the marker 20 at the first moment t1 is automatically recorded by the control unit 13 when the patient enters the apnea state. In this case, a breathing apparatus or a ventilator device is connected to the control unit 13 of the navigation system 10 via, for example, wired communication means or wireless communication means, and immediately sends an instruction to the control unit 13 when the patient's breathing is interrupted or the patient enters the apnea state. According to the third example, the position of the marker 20 at the first moment t1 is manually recorded by the operator when the patient is placed in the apnea state. According to the fourth example, the position of the marker 20 at the first moment t1 is automatically recorded by the control unit 13 immediately when the marker 20 is substantially stationary for more than 2 seconds (for example, a period corresponding to a stagnant state within the breathing cycle at the end of inhalation or exhalation). According to the fifth example, an X-ray detector (for example, a dosimeter or a scintillator) can be connected to the control unit 13 of the navigation system 10. In this case, the X-ray detector can send an instruction to the control unit 13 at the moment when the pre-diagnostic medical image is acquired. Regarding the second, third, and fourth examples, the pre-diagnostic medical image is acquired at the moment when the patient's breathing is interrupted. Regarding the first and fifth examples, it is not essential that the pre-diagnostic medical image is acquired at the moment when the patient's breathing is interrupted.

[0081] In the example shown in FIG. 5, the first instant t1 corresponds to the central instant within the period of acquisition of the medical image. Therefore, it is possible to determine the position adopted by the marker 20 at the instant t1. However, it should be noted that the first instant t1 may have a certain period (the X-ray exposure time corresponding to the time taken to acquire the medical image or even several consecutive medical images). In this case, the position taken by the marker 20 at the first instant t1 can be determined from the average of the positions adopted by the marker 20 during the X-ray exposure time. It is also conceivable that this average is weighted by the X-ray dose received at each position.

[0082] In the example shown in FIG. 5, the pre-diagnosis medical image was acquired at the instant when the patient's breathing was interrupted. However, in another example, there is nothing to prevent the pre-diagnosis medical image from being acquired at the instant when the patient is breathing freely.

[0083] FIG. 6 shows the determination of the second instant t2 corresponding to the candidate instant for the insertion of the medical device, and the comparison between the position of the marker at the first instant t1 ("target" position) and the position of the marker at the second instant t2 ("candidate" position). The second instant t2 is determined from the prediction model such that the candidate position corresponds to the target position to some extent. As shown in FIG. 6, the actual movement of the marker 20 is continuously monitored. Therefore, the comparison between the candidate positions of the marker 20 at the instant t2 can be made with respect to the target position of the marker 20 at the instant t1. In the example shown in FIG. 6, by chance, the candidate position is far from the target position. Therefore, a new second instant t2' is determined from the prediction model. The position of the marker at the instant t2' is now sufficiently close to the target position. This means that the surgical act (insertion of the medical device) can occur at the instant t2' planned on the pre-diagnosis medical image.

[0084] A tolerance band is used to compare the candidate position and the target position. For example, if the candidate position is within ±10% of the target position, it is considered to be approximately the same as the target position. Preferably, a tolerance band of ±5% of the target position is used.

[0085] The result of the comparison is displayed on the display device 12 to instruct the medical practitioner 31 whether the current time is a convenient time to proceed with inserting the measuring instrument at the candidate moment.

[0086] In certain embodiments, the augmented reality navigation system 10 is configured to cooperate with a medical robot used to assist a medical practitioner during a surgical procedure or to autonomously perform the surgical procedure directly. Next, the result of the comparison is transmitted by the navigation system 10 to the medical robot, when appropriate, to activate the positioning of the multi-joint arm of the medical robot for performing the surgical procedure preferably.

[0087] Independently of the determination of the appropriate moment for performing the surgical procedure and as shown in FIGS. 7 and 8, the prediction model 43 of the movement of the marker 20 can also be used to continuously and in real time update the three-dimensional anatomical model 51 of the living body structure generated by the control unit 13 from the pre-operative medical image 50. Next, this anatomical model can be displayed on the display device 12 in the form of augmented reality content overlaid on the real image.

[0088] To receive the pre-operative medical image acquired by the medical imaging device, the control unit is connected to the imaging device via, for example, wired communication means or wireless communication means. According to another example, the control unit can be connected to a peripheral memory device (for example, a USB (acronym for "Universal Serial Bus") stick storing the pre-operative medical image).

[0089] In the example considered and shown in FIGS. 7 and 8, the living body structure is the patient's liver, and the surgical procedure attempts to excise a tumor present within the living body structure. The pre-operative medical image 50 is acquired by performing a computed tomography (CT) scan. The moment when the medical image is acquired and the corresponding position of the marker 20 at this moment are determined by the control unit 13 of the navigation system 10.

[0090] The control unit 13 is configured to display the anatomical model 51 on the display device 12 in the form of augmented reality content overlaid on the real image. However, the movement caused by the patient's breathing moves the living structure. As shown in FIG. 8, the control unit is configured to enable the position of the anatomical model 51 to be continuously and real-time updated according to the prediction model 43 of the movement of the marker and according to the biomechanical model 60 of the anatomical structure of the human body. Therefore, the anatomical model 51 displayed on the display device 12 as an overlay on the real image moves in a manner synchronized with the patient's breathing.

[0091] The position of the anatomical model 51 at the moment when the pre-diagnosis medical image is acquired corresponds to the reference position of the anatomical model 51 with respect to the specific position of the marker 20. Thanks to the prediction model 43 of the movement of the marker 20, it is possible to predict the position that the marker 20 will adopt over time. Thanks to the biomechanical model 60, it is possible to adjust the position of the anatomical model 51 according to the position adopted by the marker 20 over time.

[0092] The biomechanical model 60 preferably includes models of the main anatomical structures (chest and abdominal walls, muscles, tendons, bones and joints, organs, vascular networks, etc.) in the chest-abdomen-pelvis region and their deformations and mechanical interactions. The biomechanical model 60 also preferably takes into account the influence of gravity due to the position of the patient 30. Such biomechanical models are known from the scientific literature, for example, see the following publications: - “SOFA: A Multi-Model Framework for Interactive Physical Simulation”, F. Faure et al., Soft Tissue Biomechanical Modeling for Computer Assisted Surgery - Studies in Mechanobiology, Tissue Engineering and Biomaterials, Volume 11, Springer、 - "A Personalized Biomechanical Model for Respiratory Motion Prediction", B. Fuerst et al., International Conference on Medical Image Computing and Computer Assisted Intervention, 2012、 - "Patient-Specific Biomechanical Model as Whole-Body CT Image Registration Tool", Mao Li et al., Medical Image Analysis, 2015, May, pages 22-34。

[0093] It should be noted that the biomechanical model of the human body may not necessarily be specific to the patient in question, but rather a biomechanical model of a general patient, such as having the same gender, size, and body mass as the patient on whom the surgical procedure is being performed.

[0094] The control unit 13 incorporates an algorithm for harmonizing the biomechanical model 60 with the positions of the markers 20 located on the skin of the patient 30 near the biological structure. For example, one algorithm enables the propagation of the movement of the muscle surface into the internal volume and the correct calculation of the positions of the internal anatomical structures.

[0095] The control unit 13 may also be configured to display a predetermined path followed by the medical device on the anatomical model 51 in the form of augmented reality content overlaid on the real image. Next, the path is updated in real time according to the position of the anatomical model. In the example shown in FIG. 8, the path includes a target point 52 on the biological structure. This target point 52 corresponds, for example, to the center of the tumor to be treated. The path may also include the entry point of the medical device on the surface of the biological structure. The predetermined path followed by the medical device is generally defined based on the pre-diagnostic medical image 50.

[0096] However, it is also conceivable that this path is defined on the pre-operative image several days or even months prior to the medical treatment. In this case, the control unit 13 is configured to realign the pre-operative image and the pre-medical treatment image. To achieve this purpose, the control unit may execute a conventional medical image realignment and / or fusion algorithm.

[0097] For example, the control unit receives the pre-operative medical image via a wired communication means or a wireless communication means. According to another example, the control unit may be connected to a peripheral memory device (e.g., a USB stick) in which the pre-operative medical image is stored.

[0098] Some elements (e.g., the region to be treated (the tumor to be dissected), the dissection region estimated based on the treatment parameters, or the dissection margin of the region to be treated (the comparison between the region to be treated and the estimated dissection region), etc.) can be segmented on the anatomical model 51. These elements can also be displayed in the form of augmented reality content. Next, their positions are updated in real time according to the position of the anatomical model.

[0099] In certain embodiments, the augmented reality navigation system 10 includes an interaction device that enables a medical practitioner 31 to interact with the system 10. The interaction device may take the form of, for example, a virtual reality glove that can capture an exact position and the pressure applied by a finger, the pressure providing haptic feedback that enables the user to feel a virtual object. In another example, the interaction device may take the form of a stylus that can target a virtual object or a specific location on an anatomical model. In particular, the interaction device may enable a medical practitioner to define or modify a predetermined path to be followed by a medical device. The interaction device may also enable interaction with virtual objects such as a setup menu, buttons, multi-choice lists, etc., to define specific parameter values of the treatment being performed (e.g., the type of treatment being performed, the duration of the treatment, the power or dose used for the treatment, etc.). Alternatively, the path and / or treatment parameters to be followed by the medical device are automatically determined by a machine learning algorithm. The medical practitioner may then verify or modify the proposed path and / or treatment parameters.

[0100] The position of the augmented reality content displayed on the display device 12 may be adapted according to the position of the medical practitioner's head. For example, when the medical practitioner's head is tilted to the left, some virtual constituent objects displayed within the augmented reality (in particular, virtual objects in the form of text) may be rotated counterclockwise so that these elements appear to the medical practitioner as if the head had not been tilted. The position of the medical practitioner's head may be determined, in particular, by using a medical practitioner head movement sensor or, alternatively, directly from the position of the marker 20 on the real image.

[0101] As shown in FIG. 9, the augmented reality navigation system 10 can cooperate with a medical robot 70 used to assist a medical practitioner 31 during a surgical procedure. The medical robot includes a movable base 71, an articulated arm 72, and a control unit 75. One end of the articulated arm 72 is fixed to the movable base 71, and the other end has a tool guide 73 intended to hold a medical device 74. The control unit 75 of the medical robot is configured to determine a configuration of the articulated arm that enables the medical device to perform a surgical act along a predetermined path. The configuration of the articulated arm is determined according to information related to the position of the marker 20 and transmitted by the navigation system 10. The augmented reality navigation system 10 and the medical robot 70 include, for example, communication means for exchanging messages. These communication means can be of a wired type or a wireless type.

[0102] The tool guide 73 is composed of, for example, two grippers driven by a linear actuator via two link rods so as to hold or release the medical device 74. The linear actuator can be reversible (the tool guide 73 can then be opened manually or automatically by an order from the control unit 75), or non-reversible (the tool guide 73 can only be opened automatically by an order from the control unit 75). Advantageously, the tool guide 73 can guide medical devices 74 of various diameters while maintaining the position of the guide axis of the medical device 74 and release the medical device 74 laterally at any time during the procedure. For example, such a tool guide 73 can guide an instrument having an outer diameter of 11 to 21 gauge (G) (gauge is a unit of measurement commonly used to define the outer diameter of a medical device such as a needle, a probe, or a catheter; 11 gauge corresponds to a diameter of 2.946 mm, and 21 gauge corresponds to a diameter of 0.812 mm).

[0103] The medical act can be performed by the medical practitioner 31 (in this case, the articulated arm 72 of the medical robot 70 is mainly used to guide the medical device 74 to assist the medical practitioner during the medical act), or can be performed directly by the medical robot 70.

[0104] In the example under consideration, the information related to the position of the marker 20 corresponds to a prediction model that indicates the position of the anatomical model 51 during the course of the patient's respiratory cycle. Accordingly, the control unit 75 of the medical robot 70 continuously and in real time adjusts the configuration of the articulated arm 72 according to the prediction model that indicates the position of the anatomical model. Accordingly, the articulated arm 72 of the medical robot 70 always remains in a configuration that enables the planned surgical act to be performed.

[0105] In a particular embodiment, and as shown in FIG. 9, the articulated arm 72 has at least six degrees of freedom such that several different candidate configurations 81, 82 of the articulated arm enable the medical device 74 to perform a surgical act along a planned path. The control unit 13 of the augmented reality navigation system 10 is configured to display the candidate configurations 81, 82 on the display device 12 in the form of augmented reality content overlaid on a real image. Next, the medical practitioner may select one of these candidate configurations by using an interaction device (for example, a virtual reality glove or a stylus). Next, the control unit 13 of the navigation system 10 may transmit the selected configuration to the control unit 75 of the medical robot 70.

[0106] It may happen that the selected configuration of the articulated arm is not preferable because the patient's weight is too heavy. Also, during the course of the medical treatment, the patient 30 may unexpectedly make a sudden movement. In such a case, it is appropriate to ensure that the patient is not injured by an undesirable contact between the outer skin of the patient's body and the medical device or the articulated arm.

[0107] To achieve this objective, in certain embodiments, the control unit 13 of the augmented reality navigation system 10 is configured to model the outer skin of the patient 30. The outer skin of the patient's body can be generated from the pre-operative medical image 50 or the anatomical model 51, perhaps with the help of data related to the patient's structure. Alternatively or additionally, the outer skin of the patient's body can be determined on the real image acquired by the camera 11 of the navigation system 10 (e.g., via a contour detection algorithm). The control unit 13 of the navigation system 10 is also configured to determine the position of the multi-joint arm 72 or the tool guide 73 of the medical robot 70 and to detect a collision risk situation when the distance between the multi-joint arm 72 or the tool guide 73 and the outer skin of the patient 30 is less than a predetermined threshold. Next, the control unit 75 can be configured to immediately block the movement of the multi-joint arm 72 when the distance between the multi-joint arm 72 or the tool guide 73 and the outer skin of the patient is insufficient.

[0108] Alternatively or additionally, in certain embodiments, the control unit 13 of the augmented reality navigation system 10 is configured to measure the deviation of the position of the marker 20 with respect to the prediction model 43 of the movement of the marker 20 (the deviation at a given instant corresponds, for example, to the distance between the actual position of the marker at said instant and the position of the marker in the prediction model 43 at the corresponding instant within the patient's respiratory cycle). Next, the control unit 13 of the augmented reality navigation system 10 is configured to detect a situation where there is a risk of injury from the medical device 74 when the deviation measured in this way exceeds a predetermined threshold. Next, measures can be taken to avoid injuring the patient when such a situation is detected. When a situation of risk of injury is detected, the control unit 13 of the augmented reality navigation system 10 immediately transmits this information to the control unit 75 of the medical robot 70. Next, the control unit 75 can be configured to perform a specific action (e.g., instructing the actuator of the tool guide 73 to immediately release the medical device 74, etc.) when a situation of risk of injury is detected.

[0109] When the medical device 74 is inserted into the patient 30's body along a predetermined path, the control unit 13 of the augmented reality navigation system 10 can be configured to display a part of the medical device 74 inserted into the interior of the patient 30's body on the display device 12 in the form of augmented reality content overlaid on the real image. It is also conceivable to calculate and display the distance between the medical device 74 and the target point 52 on the living body structure. It is also possible to detect the moment when the medical device 74 reaches the target point 52 and display an instruction for advising the medical practitioner about this.

[0110] The foregoing description clearly shows, through its various features and advantages, that the present invention achieves the stated object of providing a solution for assisting medical practitioners in minimally invasive medical treatment while reliably taking into account movements associated with the patient's breathing or involuntary movements of the patient during surgical intervention.

Claims

1. An augmented reality navigation system (10) for assisting a medical practitioner (31) when performing a surgical treatment on the biological structure of a patient (30), wherein the system (10) comprises: a camera (11) for acquiring a real image, the camera (11) being intended to be worn above the head by the medical practitioner (31); a display device (12) for displaying in real time the augmented reality content overlaid on the real image together with the real image acquired by the camera (11); a control unit (13) connected to the camera (11) and the display device (12), wherein the control unit (13) is configured to: detect on the real image the position and orientation of a marker (20) placed on the patient (30) in the vicinity of the biological structure; record the movement (40) followed by the marker (20) during a recording period (42) corresponding to one or more respiratory cycles (41) of the patient (30); determine the position and orientation of the marker (20) at a first instant (t1) belonging to the recording period (42) and corresponding to the instant of acquisition of a pre-medical image of the biological structure of the patient by a medical imaging device; formulate a prediction model (43) of the movement of the marker (20) from the movement (40) followed by the marker during the recording period (42), wherein the control unit (13) is further configured to: determine from the prediction model (43) a second instant (t2) corresponding to a candidate instant for insertion of a medical device into the biological structure of the patient; compare the position and orientation of the marker (20) at the first instant (t1) with the position and orientation of the marker (20) at the second instant (t2); and display the result of the comparison on the display device (12) in the form of augmented reality content overlaid on the real image. An augmented reality navigation system (10).

2. The control unit (13) is further configured to: receive the pre-medical image (50) from the medical imaging device; generate a three-dimensional anatomical model (51) of the biological structure of the patient (30) from the pre-medical image (50); and display the three-dimensional anatomical model (51) on the display device (12) in the form of augmented reality content overlaid on the real image. The position and orientation of the three-dimensional anatomical model (51) overlaid on the real image are continuously and real-time updated according to the prediction model (43) of the movement of the markers and according to the biomechanical model (60) of the anatomical structure of the human body, the system (10) according to claim 1.

3. The control unit (13) is further configured to display, on the three-dimensional anatomical model (51), in the form of augmented reality content overlaid on the real image, a predetermined path that is tracked by a medical device and updated in real time according to the position and orientation of the three-dimensional anatomical model (51), the system (10) according to claim 2.

4. The predetermined path tracked by the medical device is predefined on a pre-operative medical image, and the control unit (13) is configured to receive the pre-operative medical image and realign the pre-operative medical image to the pre-operative medical image (50) for displaying the path on the three-dimensional anatomical model (51), the system (10) according to claim 3.

5. The system (10) according to claim 3, further comprising an interaction device that enables the medical practitioner (31) to target a specific location on the three-dimensional anatomical model (51), wherein the predetermined path tracked by the medical device is defined by the medical practitioner by using the interaction device.

6. The control unit (13) includes the following elements various anatomical structures within the living structure, the region to be treated within the living structure, the dissection region estimated from the parameters of the surgical treatment, and the dissection margin of the region to be treated within the living structure, which is determined by comparing the region to be treated with the dissection region estimated from the parameters of the surgical treatment, segmenting at least one element on the three-dimensional anatomical model (51) from among the above, and configured to display the segmented elements on the three-dimensional anatomical model (51) in the form of augmented reality content overlaid on the real image, and the segmented elements are updated in real time according to the position and orientation of the three-dimensional anatomical model, the system (10) according to any one of claims 2 to 5.

7. The control unit (13) is configured to display at least one virtual configuration object related to surgical treatment parameters on the display device (12) in the form of the augmented reality content, and the system (10) includes an interaction device that enables the medical practitioner (31) to interact with the virtual configuration object to select a specific value of the surgical treatment parameter. The system (10) according to any one of claims 1 to 6.

8. The display device (12) is a display screen (12a) incorporated into an augmented reality headset. The system (10) according to any one of claims 1 to 7.

9. The control unit (13) is configured to correct the position and orientation of the augmented reality content according to the position and orientation of the head of the medical practitioner (31). The system (10) according to any one of claims 1 to 8.

10. The position and orientation of the head of the medical practitioner (31) are determined by using a medical practitioner head movement sensor. The system (10) according to claim 9.

11. An assembly formed by the augmented reality navigation system (10) according to any one of claims 1 to 10 and a medical robot (70) for assisting a medical practitioner (31) in surgical treatment on a living structure of a patient (30), wherein the medical robot (70) includes: - A movable base (71), - A multi-joint arm (72) having a tool guide (73) with one end fixed to the movable base (71) and the other end intended to hold a medical device (74), and - A control unit (75) configured to determine a configuration of the multi-joint arm (72) that enables a surgical act to be performed by the medical device (74) along a predetermined path, wherein the configuration of the multi-joint arm is determined according to information related to the position and orientation of the marker (20) and transmitted by the navigation system (10). An assembly and a medical robot (70) including the control unit (75).

12. The information related to the position and orientation of the marker (20) is an indication that the difference between the position and orientation of the marker (20) at the first moment (t1) and the position and orientation of the marker (20) at the second moment (t2) is less than a predetermined threshold. The assembly according to claim 11 combined with claim 1.

13. The information related to the position and orientation of the marker (20) corresponds to a prediction model that indicates the position and orientation of the three-dimensional anatomical model (51) during the respiratory cycle of the patient, and the control unit (75) of the medical robot (70) is configured to continuously and in real time adjust the configuration of the articulated arm (72) according to the prediction model that indicates the position and orientation of the three-dimensional anatomical model. The assembly according to claim 11 in combination with claim 2.

14. The control unit (13) of the augmented reality navigation system (10) models the outer skin of the body of the patient (30), determines the position and orientation of the articulated arm (72) or the tool guide (73) of the medical robot (70), and is configured to detect a situation where there is a risk of collision when the distance between the articulated arm (72) or the tool guide (73) of the medical robot (70) and the outer skin of the patient (30) is less than a predetermined threshold value. The assembly according to any one of claims 11 to 13.

15. The control unit (13) of the augmented reality navigation system (10) is configured to detect a situation where there is a risk of injury from the medical device (74) when the deviation of the position and orientation of the marker (20) from the prediction model (43) of the movement of the marker (20) exceeds a predetermined threshold value. The assembly according to any one of claims 11 to 14.

16. The tool guide (73) of the medical robot (70) includes an actuator that enables the medical device (74) to be instantaneously released, the actuator is commanded by the control unit (75) of the medical robot (70), and the control unit (13) of the augmented reality navigation system (10) is configured to send a command to instantaneously release the medical device (74) to the control unit (75) of the medical robot (70) when the risk of injury situation is detected. The assembly according to claim 15.

17. The articulated arm (72) has at least six degrees of freedom such that several different candidate configurations (81, 82) of the articulated arm (72) enable the surgical act to be performed by the medical device (74) along the predetermined path. The control unit (13) of the augmented reality navigation system (10) is configured to display the candidate configuration (81, 82) on the display device (12) in the form of augmented reality content overlaid on the real image and to receive an instruction related to the selection of one specific configuration from among the various candidate configurations (81, 82) from the medical practitioner (31). The assembly according to any one of claims 11 to 16.

18. When the medical device (74) is inserted into the patient (30)'s body along the predetermined path, the control unit (13) of the augmented reality navigation system (10) is configured to display a part of the medical device (74) inserted into the patient (30)'s body on the display device (12) in the form of augmented reality content overlaid on the real image. The assembly according to any one of claims 11 to 17.

19. The control unit (13) of the augmented reality navigation system (10) is configured to calculate the distance between the medical device (74) and the target point (52) on the biological structure and display it on the display device (12) in the form of augmented reality content overlaid on the real image, and / or to detect the moment when the medical device (74) reaches the target point (52). The assembly according to claim 18.

Images