Method for producing and displaying 3D ultrasound images

Abstract

A method for producing and displaying 3D ultrasound images, a method for needle insertion using same and equipment for the implementation of these methods. The invention relates to an assisted and interactive method for producing and displaying 3D ultrasound images, by controlled acquisition and processing of 2D images acquired by way of a 2D ultrasound probe (1) that can be moved on a living subject (S) by a robotic arm (2), or by a practitioner with the assistance of a robotic arm (2), in particular before and during a procedure for inserting a needle (3) into said subject (S), the procedure being performed, for example, by means of or with the assistance of another robotic arm (4) equipped with a needle holder (8). The method is used during a phase of planning the insertion, during a re-planning phase just before insertion, and during the insertion, and displays relevant volumes in a manner adapted to the different phases, while also showing the trajectory of the needle (3), the fragile structures (OVT) and the target (C).

Description

Description METHOD FOR PRODUCING AND DISPLAYING 3D ECHOGRAPHIC IMAGES

[0001] The present invention relates to the fields of medical imaging and assistance in the positioning of needles under ultrasound guidance assisted by a robotic installation, and more particularly the creation and exploitation of 3D ultrasound images from the autonomous or semi-autonomous acquisition of a set of 2D ultrasound images and its advantageous use in relation to procedures for inserting and implanting sampling, injection or localized (target) treatment needles in subjects.

[0002] Its objects are an assisted and interactive method for producing and displaying 3D ultrasound images by controlled acquisition and processing of 2D images, a needle insertion method, automatic or preferably assisted or collaborative, using the aforementioned method, and a robotic medical installation for implementing these two methods.

[0003] A medical technique currently under development is percutaneous surgery using a needle inserted into the patient, both for the treatment of cancers and for injection or localized sampling.

[0004] In the case of treating cancers located in the abdominal region, it consists of positioning the tip of one or more needles in a tumor, passing through the abdominal wall. The destruction of the tumor takes place in particular by generating significant temperature variations (hot or cold) at the tip of the needle. It is thus possible, for example, to treat tumors by radiofrequency, cryoablation, microwaves or even by electroporation.

[0005] This approach is gaining popularity because it is inexpensive, minimally invasive, and advances in ablation techniques allow for the treatment of targets of increasingly diverse sizes, shapes, and locations. This allows this type of treatment to be performed most often on an outpatient basis, saving money for the healthcare system and also improving patient well-being.

[0006] The main challenge associated with this healing technique is the precise positioning of the needles, while preserving the surrounding fragile structures. Correct needle positioning is a procedure that requires unique expertise.

[0007] In the case of the pancreas, kidney and liver, these organs being made of soft tissue and moreover located in the abdominal cavity, they are subject to displacements as well as deformations. These displacements and deformations are notably induced by physiological movements, mainly respiratory movement, and the insertion of the needle itself.

[0008] It is essential to note that if the patient is required to hold his breath during needle insertion, two successive apneas do not guarantee that the target will be located in a similar manner. The same applies to structures (tissues, organs, vessels) located between the skin and the target.

[0009] Currently, various technical solutions are known for performing needle insertion procedures (assisted or guided) in patients or subjects, based on various 3D medical imaging technologies (scanner, MRI or 3D ultrasound). They all have more or less significant drawbacks, in terms of costs, complexity of implementing the necessary installations, duration of the procedure, harmful exposure to radiation, compatibility with an environment presenting a strong magnetic field, comfort of use for the practitioner and / or precision of execution, in particular linked to the certain obsolescence of the imaging data at the time of the intervention since they were acquired well before the intervention during the planning phase.

[0010] Thus, known methods using X-ray imaging suffer from a complex installation configuration (markers, cameras), a high cost of imaging equipment (CT scanner), the impact of a high dose of X-rays and, above all, the obsolescence of imaging data when they are actually used, due to the acquisition and processing times of the latter.

[0011] The same defects are identified for solutions implementing an MRI process, except for the absence of X-ray dose, but for MRI, the presence of a high intensity magnetic field significantly limits the availability of compatible processing equipment and has a major impact on its cost.

[0012] Finally, methods using 3D ultrasound scanners to assist or guide the insertion of needles are also already known, but they also have numerous limitations.

[0013] Thus, these 3D ultrasound scanners are expensive, lack precision (poor quality images compared to 2D ultrasound imagers), use a bulky and tedious probe to handle, also require a spatial tracking and referencing system (camera or electromagnetic sensor) and are not optimized in terms of data processing (quantity and relevance of the data acquired and processed). In addition, the problem of data obsolescence also affects the images provided by these 3D ultrasound imagers, unlike 2D ultrasound imagers (which are not always suitable for use in the aforementioned context and require controlled movement to provide usable images).

[0014] There are also various solutions to assist needle insertion using robotic devices, most of which involve the use of a scanner-type imager and a camera. However, these solutions require alignment between the imager's reference frame and that of the robot arm, and the use of markers in the scanner and on the robot that can be identified by the camera and the scanner.

[0015] Furthermore, these known solutions also do not take into account the highly probable displacement of the target and the soft and fragile structures (affected by the insertion) between two apnea phases, which makes blind insertion performed within the framework of these solutions problematic and risky, also due to the obsolescence of the images taken previously and which are exploited just before the start of the insertion. In addition, the ergonomic difficulties linked to the necessary use of a specific camera represent a significant challenge, which can impact the efficiency and fluidity of procedures in the operating room. Furthermore, it should be considered that the trajectory chosen by the clinician for the insertion of the needle is not necessarily the best one: this limitation raises questions about the flexibility and adaptability of the method depending on the anatomical particularities of each patient. Finally, the economic aspect of these solutions cannot be ignored: its high cost, linked both to the qualified personnel and the specific infrastructures necessary for its implementation, represents a major obstacle to its widespread adoption, particularly in a context of limited resources.

[0016] The present invention aims to propose a new solution making it possible to overcome at least some of the main limitations mentioned above, in particular those linked to the obsolescence of the images used, in particular at the end of the preparatory period preceding the start of insertion, to the presence of a camera, to the absence of consideration of changes occurring between two apneas and to limited collaboration between the robotic devices and the practitioner, all while exploiting the advantages linked to the use of ultrasound.

[0017] According to a first and main aspect of the invention, the invention aims to provide a relevant, ergonomic and interactive visual aid for the practitioner, at least before an intervention involving needle insertion with the aid of a needle holder or needle guide.

[0018] To this end, its main object is an assisted and interactive method for producing and displaying 3D ultrasound images, by controlled acquisition and processing of 2D images acquired by means of a 2D ultrasound probe which can be moved over a living subject by a robotic arm, or by a practitioner with the assistance of a robotic arm, in particular before a needle insertion operation using a needle holder or needle guide in said subject, during a preparatory planning phase, the insertion operation being carried out for example by means of or with the assistance of another robotic arm equipped with a needle holder, said method implementing a medical installation comprising, on the one hand, at least one human-machine interface with a display device and command or control means and, on the other hand, at least the robotic arm carrying said 2D ultrasound probe,whose position and orientation are known in real time and can be used and recorded, concomitantly with the 2D images generated at each exploration, by said installation, method characterized in that it comprises at least the steps consisting of:, a) locating one or more targets to be treated in the subject by exploratory movement of the probe on the subject's skin and analysis of the acquired 2D ultrasound images, by a practitioner possibly assisted by suitable detection software; b) producing a first 3D image of a volume region of the subject, extending from the skin to the target concerned and containing the latter and the reasonable possible trajectories for introducing a needle to reach said target estimated by the practitioner, this 3D image being obtained from a set of 2D images acquired, during a phase of apnea of ​​the subject, by movement of the 2D ultrasound probe by the practitioner with the assistance of the robotic arm and potentially by using the location information from step a);c) automatically determining, during or after step b), the parameters and the path of an optimized and reproducible trajectory for the 2D ultrasound probe allowing the production of substantially standardized and identically framed 3D images of the aforementioned volume region; d) producing one, or successively several, substantially standardized 3D image(s) of this volume region, in quasi-real time and automatically or at the command of the practitioner, during a breath-hold phase of the subject and on the basis of 2D images freshly acquired following a new exploration with the 2D ultrasound probe carried out by applying the parameters and following the path which were determined during step c);e) visualizing, in this or these substantially standardized 3D images of the volume region of the subject, the organs, vessels and / or similar tissues or soft structures present between the skin and said target, as well as the latter two, in order to determine the positioning and orientation of a needle holder for at least one possible optimized insertion trajectory of the needle; f) repeating steps d) and e) just before the start of the insertion of the needle, which is then positioned in accordance with the optimized insertion trajectory selected by the practitioner and visualized in the current 3D image, and automatically determining and visualizing the position of the target at this instant in order to possible adjust the positioning and / or orientation of the needle holder, in particular just before the actual insertion of the needle.;

[0019] It also has as objects, according to two other aspects, a needle insertion method using the aforementioned 3D ultrasound image production and display method, and a medical installation for implementing these two methods.

[0020] The invention will be better understood from the following description, which relates to a preferred embodiment, given as a non-limiting example, and explained with reference to the appended schematic drawings, in which:

[0021] [Fig. 1] schematically represents a medical installation according to a preferred embodiment, for implementing the imaging and needle insertion methods according to the invention;

[0022] [Fig. 2] represents the installation of figure 1 during an intervention on a subject, here a human patient;

[0023] [Fig. 3] is a detailed schematic representation in section of a part of the subject's body just before the insertion of a needle;

[0024] [Fig. 4] is a representation similar to that of Figure 3 during the insertion of the needle, for example as a partial and detailed view (on another scale) of the scene represented in Figure 2;

[0025] [Fig. 5] is a representation of a set of 2D ultrasound images acquired, automatically or collaboratively with the practitioner, via the probe carried by the first robotic arm forming part of the installation according to the invention;

[0026] [Fig. 6] is a representation of a volumetric region of a patient's abdomen in the form of a planisphere showing the fragile / soft structures and allowing to estimate the different possible trajectories for the insertion of a needle in order to reach the target;

[0027] [Fig. 7A] and [Fig. 7B] are 3D images of a tubular / truncated volumetric region of the patient's body between the skin and the target and surrounding a needle trajectory, viewed from two different directions;

[0028] [Fig. 8] is an image showing the tubular / truncated region of Figures 7A and 7B viewed along the insertion direction, with the entry point marked by a cross, and,

[0029] [Fig. 9] is a 3D image showing a needle hitting the target.

[0030] The invention relates, according to a first aspect of the invention, to an assisted and interactive method for producing and displaying 3D ultrasound images, by controlled acquisition and processing of 2D images acquired by means of a 2D ultrasound probe (1) which can be moved over a living subject (S) by a robotic arm (2) carrying it, or by a practitioner with the assistance of a robotic arm (2) moved by the latter and holding said probe (1), in particular before an operation of inserting a needle (3) using a needle holder or needle guide (8) into said subject (S), during a preparatory planning phase, the operation being carried out for example by means of or with the assistance of another robotic arm (4) equipped with the needle holder or needle guide (8).

[0031] This method implements a medical installation (5) comprising, on the one hand, at least one human-machine interface with a display device (6) and command or control means (7) and, on the other hand, at least the robotic arm (2) carrying said 2D ultrasound probe (1), the position and orientation of which are known in real time and can be used and recorded, concomitantly with the 2D images generated at each exploration, by said installation (5).

[0032] This method is characterized in that it comprises at least the steps consisting of: a) locating one or the target (C) to be treated in the subject (S) by exploratory movement of the probe (1) on the skin (P) of the latter and analysis of the acquired 2D ultrasound images, by a practitioner possibly assisted by suitable detection software; b) producing a first 3D image of a volume region (RV) of the subject (S), extending from the skin (P) to the target (C) concerned and containing the latter and the reasonable possible introduction trajectories of a needle (3) to reach said target (C) estimated by the practitioner, this 3D image being obtained from a set of acquired 2D images, during an apnea phase of the subject (S), by movement of the 2D ultrasound probe (1) by the practitioner with the assistance of the robotic arm (2) and potentially by exploitation of the location information from step a); c) automatically determining, during or after step b), the parameters and the path of an optimized and reproducible trajectory for the 2D ultrasound probe (1) allowing the production of substantially standardized and identically framed 3D images of the aforementioned volume region (RV); d) producing one, or successively several, substantially standardized 3D image(s) of this volume region (RV), in quasi-real time and automatically or at the command of the practitioner, during a breath-hold phase of the subject (S) and on the basis of 2D images freshly acquired following a new exploration with the 2D ultrasound probe (1) carried out by applying the parameters and following the path which were determined during step c);e) visualizing, in this or these substantially standardized 3D images of the volume region (RV) of the subject (S), the organs, vessels and / or similar tissues or soft structures (OVT) present between the skin (P) and said target (C), as well as the latter two, in order to determine the positioning and orientation of a needle holder (8) for at least one possible optimized insertion trajectory of the needle (3); f) repeating steps d) and e) just before the start of the insertion of the needle (3), which is then positioned in accordance with the optimized insertion trajectory selected by the practitioner and visualized in the current 3D image, and automatically determining and visualizing the position of the target (C) at this time with a view to a possible adjustment of the positioning and / or orientation of the needle holder (8), in particular just before the actual insertion of the needle.

[0033] Thanks to the aforementioned provisions of the invention, it is possible to propose a solution making it possible to achieve the set goal. Indeed, the method in question makes it possible to provide continuous, collaborative and interactive assistance to the practitioner at least during the preparatory phase of planning the intervention and during the fine adjustment phase just before the start of the insertion of the needle with the help of the needle holder, which determines the insertion trajectory and will guide the needle during its insertion. This method exploits the advantages of 2D ultrasound, while allowing 3D visualization, this by means of particular provisions. The method also takes advantage of preliminary targeted learning phases, provided by the practitioner for the robotic arm (2) carrying the probe (1) and the software means forming part of the installation (5), to subsequently assist effectively the said practitioner by automation or semi-automation of ultrasound explorations and automation of target recognition and tracking.

[0034] According to a possible additional aspect of the invention, the proposed solution can further provide continuous, collaborative and interactive assistance to the practitioner throughout the insertion period until the target is reached.

[0035] For this purpose, the aforementioned method may also comprise, after step f) and during the insertion of the needle (3), a step g) consisting of: visualizing in quasi-real time the progressive insertion of the needle (3) by repeated creation and visualization of refreshed 3D images of a volume (VR), for example substantially cylindrical or truncated, as a fraction of the aforementioned volume region (RV) of the subject (S), preferably as a variable fraction of this volume region, the 3D images of this volume (VR) showing either said target (C) and at least one end portion (3') of the needle (3), preferably the entire part of the needle (3) inserted into the subject (S), or at least one end portion (3') of the needle (3) and a volume (VP) preceding this end (3') during the insertion, this visualization of refreshed 3D images of said volume (VR) preferably continuing at least until the target (C) is reached by the needle (3),the two steps f) and g) then being carried out together during the same apnea phase of the subject (S).,

[0036] In terms of information for the practitioner, the method according to the invention provides a scalable 3D visualization. Thus, it may be provided, first of all, to visualize a large volume region (RV) encompassing the target and a surrounding area, the skin (or at least an internal area close to the skin), at least the various reasonable insertion trajectories and all the soft and fragile structures (OVT) located in this region, this during step b) and at least part of a preliminary planning phase; then to visualize a smaller volume (VR, VT) comprising at least the end of the needle and at least one intermediate volume just large enough to be able to know and take into account the presence of the target, organs, vessels and / or tissues or similar fragile structures present in this environment, this just before and during the intervention.

[0037] Alternatively, and depending on the experience and wishes of the practitioner, the visualized volume may, from step b), be limited to a smaller volume than the volume region (RV) explored by the probe (1), for example a truncated cone volume (VR) extending around a median insertion direction deemed reasonable by the practitioner (passing through the center of the target) and between the skin and the target. For example, such a truncated cone would have a base in the form of a disc or ellipse with a diameter at the base of approximately 5 to 10 cm at the skin level) and the apex of which surrounds the center of the target, or even the latter.

[0038] The implementation of a robotic arm carrying the 2D ultrasound probe, subject to learning of the volume to be explored and capable of then carrying out, at the request of the practitioner, semi-automatic or automatic acquisitions with optimized parameters and therefore standardized 3D visualizations, relieves the practitioner and guarantees constant image quality.

[0039] Furthermore, adapting the processing and display of images to the useful volume during the different phases allows in particular faster image refreshment when the provision of real-time information is important (during insertion), as well as easier interpretation when the area viewed is significantly limited to the area of ​​interest at the time of acquisition.

[0040] During step b), the robotic arm (2) can facilitate the movement of the probe (1) by applying a controlled force to it so that the practitioner is not obliged to force during this exploration, while avoiding excessive pressure.

[0041] It should be noted that, during each exploration with the 2D ultrasound probe (1) and during the insertion of the needle (3), the patient is assumed to be in apnea.

[0042] Furthermore, by means of the human-machine interface (6, 7), the practitioner is able to trigger explorations and recovery of 2D images with the probe (1) held by the robotic arm (2), to navigate in the 3D images produced by changing in particular the viewing angle relative to the volume displayed (for example to view this volume with its contents along the axis of a needle insertion trajectory, which is advantageously merged with the median axis of this volume) and to choose the components that he wishes to see displayed (apart from automatic acquisition and visualization operations).

[0043] The method according to the invention therefore provides interactive, almost instantaneous visual assistance, timed to the patient's breathing, non-interfering with the practitioner's other tasks and adaptive according to the progress of the treatment protocol. This assistance also continues continuously throughout the different phases of the protocol implemented: prior planning, rapid re-planning just before insertion and the insertion phase.

[0044] While steps b) and c) correspond to a learning phase of the robotic arm (2) carrying the probe (1), in order to then allow it to carry out explorations automatically or semi-automatically, steps d) and e) are associated with a preparatory phase of planning the intervention resulting, where appropriate by successive repetitions, in obtaining a substantially standardized 3D image of the volume region (VR) deemed optimal, and in its visualization, for example in the form of a planisphere, with integrated indication of the possible trajectories of insertion of the needle (3) determined automatically or by the practitioner, then of a proposal of a possible trajectory of optimized and secure insertion of the needle (3), selected automatically or by the practitioner.

[0045] However, the last repetition of steps d) and e), during step f), corresponds to a replanning phase preceding, during the same apnea phase, imminently the start of the insertion phase and taking into account a probable minor modification of the position of the target (C) and of the soft structures (OVT) between two apneas of the subject (S), this step f) comprising the creation and the visualization of a new substantially standardized 3D image of the volume region (RV) of the same type as that obtained at the end of step e), this with a view to a possible automatic or manual adjustment of the positioning and / or the orientation of the needle holder (8) corresponding to the optimized trajectory selected at the end of step e) and at least one validation of this trajectory, possibly finely readjusted, by the practitioner.Since there is no major difference between the two apneas in terms of the position of the structures (OVT) and the target (C), a fine adjustment is sufficient to re-optimize the planned trajectory, without requiring a more complete replanning.

[0046] In order to facilitate and speed up the choice or validation of a possible optimized insertion trajectory of the needle (3) during step e), it may advantageously be provided that in step b), the trajectories deemed reasonable by the practitioner are defined by the latter via a median insertion direction passing through the target (C) and correspond to rectilinear trajectories included in a conical or truncated volume, preferably with a circular or ellipsoidal base, possibly corresponding to the reduced volume (VR), extending by tapering from the skin (P) to the target (C) at least and with a median axis (AML) passing through the target (C), this median axis corresponding to the aforementioned median direction and being able to be materialized by a corresponding positioning of the needle holder (8).Preferably, at least one possible, or even optimized, insertion trajectory is determined after analysis of the content of said first 3D image and taking into account the specific circumstances linked to the subject (S) and the configuration of the intervention, automatically or semi-automatically or by the practitioner.

[0047] In order to be able to carry out with the probe (1) mounted on the robotic arm (2) a rapid and efficient exploration, the substantially standardized 3D images of step c) and step d) advantageously consist of 3D images produced from 2D images acquired along intersecting planes with respect to the reasonable insertion trajectories of step b) and with sufficient and constant quality, namely with an optimized and reproducible trajectory of the 2D ultrasound probe (1), that is to say in particular by applying a substantially constant and as low as possible force at the level of the 2D probe (1) allowing the acquisition of 2D images with the aforementioned quality and by moving the latter with a controlled and optimized acquisition speed, in particular the highest possible speed nevertheless guaranteeing sufficient density and quality of volume information to produce images of acceptable quality.When the movement of the probe (1) is a simple translation, a substantially constant and high speed (with the aforementioned restriction) is advantageously applied during the exploration, advantageously with variations less than a predetermined threshold value, for example 10%, preferably 5%. However, in the case of rotational or tilting movements, or combined movements, it is necessary to slow down where appropriate. so that the movement of the acquisition plane in the volume at the end of the probe is not too significant in relation to the number of 2D images per second that can be acquired.

[0048] In accordance with advantageous embodiment variants of the invention, it may be provided that in step e), the visualization of the tissues or structures (OVT) in the substantially standardized 3D image of the volume region (VR) is carried out by means of a visualization mode among: i) a successive visualization of sectional views corresponding to 2D interpolated images, ii) a 3D visualization obtained by a direct volume rendering method and iii) a 3D visualization obtained by a surface rendering method, after carrying out a segmentation phase of said substantially standardized 3D image of the volume region (VR).

[0049] According to a possible additional feature of the invention, the method may comprise an additional final step h) consisting of simultaneously displaying, and where appropriate superimposed, at least two 3D images of the volume region (RV) of the subject (S) extending from the skin (P) to the target (C) concerned, and comprising the latter, or alternatively of a more restricted volume region containing the location of the target (C), one of which was taken before the intervention and the treatment by means of the needle (3) and the other of which was taken after this intervention. This optional step makes it possible, after completion of the treatment, by merging before / after images or their display in parallel, to verify that the treatment carried out during the intervention does indeed affect, in its entirety, the area of ​​the target (C).

[0050] As shown in Figure 2, the method may consist, before step a), in positioning the robotic arm (2) carrying the probe (1), relative to the subject (S) lying on a surgical table (9), in such a way that the base (10) of said robotic arm (2) is located on the lateral side of said table (9) which is opposite to that occupied by the practitioner, and where appropriate likely to be subjected to the passage of the 2D ultrasound probe (1). This arrangement makes it possible to avoid any interference between the practitioner and the robotic arm (2) and also to be able to easily apply sufficient pressure during explorations with the probe (1).

[0051] The indication of the possible rectilinear insertion trajectories or of the optimized trajectory of the needle (3) can be carried out by the practitioner, with a view to their display in the 3D images, either virtually by means of the human-machine interface, or preferably by manipulation by the practitioner of the needle holder (8) and validation of the position and orientation of the latter by said practitioner when the desired trajectory is displayed in the 3D image.

[0052] The needle holder (8) is advantageously position-controlled in such a way that it remains permanently oriented so that the direction of insertion of the needle that it carries passes through the center of the target (C).

[0053] Advantageously, the practitioner can indicate a trajectory by suitable positioning of the needle holder (8) for example (its position being known due to the knowledge of the position of the two robotic arms by the medical installation also comprising the display means), in order to visualize this trajectory during the acquisition of the first 3D image produced in an assisted manner during step b), but also during step d). It is thus possible to provide guidance information during the manual or assisted acquisition of the 2D images and to facilitate the visualization of only the relevant volume (to avoid the display of irrelevant information likely to distract the practitioner).

[0054] There are several options for finding a needle insertion path (possible, optimized), which differ in their degree of automation.

[0055] Thus, in accordance with a manual option, it is possible for the practitioner to move the needle holder (8) and interactively visualize a tubular volume (VT) whose median axis (AML) corresponds to the trajectory resulting from the current orientation and positioning of the needle holder (8). In this case, the practitioner moves the needle holder somewhat randomly, and if he notices a possible, or even optimized, trajectory, he stops. This is a user-friendly and efficient approach, since the needle holder always remains directed towards the target.

[0056] A second option is to offer the practitioner a map of possible insertion options (for example on a planisphere of the volume region). In this way, all options are taken into account and the practitioner can make the best choice in an informed manner. Once the optimized trajectory has been chosen, directs the needle holder as close as possible to its choice, advantageously visualizing the position of its trajectory in said map (planisphere for example).

[0057] Finally, according to a third option, it can be considered to create a 3D model of the volume region (VR) integrating the structures (OVT) and the target (C) from the 3D image, then to calculate the best trajectory and to propose it directly to the practitioner.

[0058] The explorations with the probe (1) of step d) can be carried out entirely automatically at the simple request of the practitioner, thus completely freeing the latter from this task.

[0059] Alternatively, and to allow the practitioner to further control these explorations and subsequent image acquisition, for example to limit their amplitude or center them on a reduced volume of interest which may vary, in particular during insertion, the method may consist of carrying out the explorations with the probe (1), which are subsequent to step c) and initiated at the initiative of the practitioner, in a semi-automatic or collaborative manner, the movement of displacement of the probe (1) and the extent of the exploration being determined by manual action of the practitioner, the actual displacement of the probe (1) being nevertheless constrained by the trajectory parameters determined in step c). The exploration movements of the probe (1) may be carried out either bidirectionally or unidirectionally with initial repositioning of the probe (1) after completion of an exploration.In the latter case, it may possibly be possible to envisage an automatic initial relocation of the probe (1) with regard to the target (C), with an adequate position and orientation, by the robotic arm (2) before each new exploratory movement of the latter carried out during the subsequent stages.

[0060] In accordance with an embodiment variant based on more advanced assistance, making it possible to ensure that the target has not moved or, if not, to be able to estimate its movement, in particular between two apneas, the analysis of the 2D images acquired in step a), with a view to identifying and locating the target (C) to be treated, for example a cancerous tumor, is carried out by or with the assistance of an algorithm, for example an artificial intelligence program having previously undergone adequate training, this location of the target (C) being automatically repeated at each subsequent 3D image exploration and display.

[0061] The repetitions referred to in step d) make it possible to ensure the position of the target very frequently. In particular, at least one repetition is necessary after step b), when the practitioner or clinician has planned and positioned the needle holder or needle guide (8) for the first time. Indeed, this phase is relatively long and therefore incompatible with continuous apnea on the part of the subject (S). Repeatedly performing step d) makes it possible to correct, as provided for in step f), an expected error in repositioning the target (C) and the surrounding organs, tissues and structures (OVT) to the target (C) at the time of the apnea (second, third, etc.) during which the needle will actually be inserted (3).

[0062] According to another advantageous characteristic of the invention, the movement by the practitioner of the ultrasound probe (1) in steps a) and b), is controlled by constraint and limitation of movement by the robotic arm (2), in terms of orientation and position of the probe (1) relative to the skin (P) of the subject (S) and speed of movement, the practitioner being able to advantageously adjust the pressure setpoint for applying the probe (1) to the skin (P), controlled by the robotic arm (2) during the movement and / or deactivate the assistance of the robotic arm (2), by means of suitable means forming part of the human-machine interface (6, 7). These arrangements make it possible to avoid harmful movements of the probe (1) potentially involving laborious and time-consuming, or even impossible, reconstruction of the 3D images.Indeed, during each exploration, the successively acquired 2D images must, as far as possible, follow one after the other in a regular and continuous manner, and in particular, overlapping of these images and intersections between these images must be avoided. This is in order to produce a sequence of 2D images whose relative position and acquisition conditions facilitate the calculation and consistency of the resulting 3D image. Deactivation of the assistance of the robotic arm may be desired for finer control by the practitioner, with less effort.

[0063] In addition or alternatively to the preceding characteristic, the invention may provide that the pressure of application of the probe (1) on the skin (P) is, during the movement of the latter at each exploration carried out during the phase b) and / or phase d), automatically controlled by the robotic arm (2) by real-time control of the value of this pressure by means of a suitable sensor or a control loop exploiting the result of an analysis of the quality of the 2D ultrasound image generated in real time, said application pressure being minimized during phases d), e) and f) to a value just allowing the acquisition of 2D images of sufficient quality to produce a standardized 3D image, in order to generate minimal deformation of the tissues visualized.

[0064] The control aims, particularly in the last three phases mentioned above, in particular to optimize the quality of the 2D images produced while limiting as much as possible the pressure exerted on the skin (P) of the patient or subject (S) in order to limit as much as possible the deformations induced at the level of the underlying tissues and structures.

[0065] During the first clinician-assisted exploration in step b), effective pressure control can also be implemented, especially if the robotic arm is in exoskeleton mode, so as not to injure the patient.

[0066] During the phase of calculating an ideal trajectory, from steps d) to f) (planning and replanning phases), there are potentially several passes or explorations, automatically to calibrate the movement in order to determine the lowest possible application pressure, but still allowing the acquisition of quality images.

[0067] Finally, during step g) (during which 3D images are acquired regularly), there is a permanent monitoring of the pressure in order to take into account a possible movement of the patient or a possible problem with the robotic arm.

[0068] Advantageously, the production of substantially standardized 3D images at the level of steps c), d) and f) is carried out automatically, on the basis of the 2D images produced by the probe (1) manipulated by the robotic arm (2), at each command or with the participation of the practitioner, by making the probe (1) follow a determined trajectory, in particular optimized in terms of position and orientation of the probe (1) relative to the skin (P) of the subject (S) and speed of movement of the probe (1), during movement, this trajectory having been determined during or after step b).

[0069] In order to enable the acquisition of good quality images, the method according to the invention may further consist, on request or automatically, in spreading by spraying, by means of a suitable device associated or integrated with the probe (1), ultrasound contact gel in front of and / or behind the probe (1) during one, preferably each exploratory movement. This arrangement advantageously makes it possible to prevent the probe (1) from removing the present and necessary gel, pass after pass, thus certainly compromising the quality of the acquired ultrasound images due to contact, and therefore poor acoustic transmission.

[0070] To mark and understand the potential trajectory of the needle in the patient's body, and in accordance with another advantageous characteristic of the invention, the 3D visualization of the organs, vessels and / or tissues (soft or fragile structures OVT) present between the surface of the skin (P) and the target (C) as provided in step e), comprises the display by overprinting of a collimation or aiming pattern centered on the target (C), making it possible to define and visually simulate the virtual trajectory of insertion of the needle (3) for a given positioning and orientation of the needle holder (8), and advantageously by interaction via the human-machine interface, to explore the possibilities of rectilinear trajectory of insertion of the needle (3) and entry point in the subject (S) and to select one or more proposals.This pattern (e.g. a cross as in Figure 8) is set up and visible when the target is viewed in the displayed 3D image (e.g. of a reduced volume VR as in Figures 7A and 7B) along the axis of the relevant insertion direction.

[0071] The exploration of the possibilities of a rectilinear trajectory for inserting the needle (3) can be carried out by the practitioner by physically manipulating the needle holder (8) carried by the other robotic arm (4), then by recording and requesting the visualization of the trajectory(ies) retained by him in the 3D images (by means of suitable control means of the human-machine interface).

[0072] Alternatively, it may be planned to carry out the exploration of the possibilities of a rectilinear trajectory for inserting the needle (3) by simulating or collaboratively carrying out a constrained movement of the other robotic arm (4) and therefore of the needle holder (8), for example in the form of a constrained ball joint movement, such that the insertion trajectory always passes through the center of the target (C) regardless of the position of said arm (4) and said needle holder (8).

[0073] Preferably, the determination of the positioning and orientation of the needle holder (8) with a view to a possible optimal insertion trajectory by using a 3D visualization of the organs, vessels and / or tissues or structures (OVT) present between the skin (P) and the target (C), as provided in step e), comprises the automatic analysis and display of at least one, preferably several, proposal(s) of rectilinear trajectory of insertion of the needle (3) and entry point (PE) in the subject (S), each associated with an orientation or a plurality of orientations contained in a virtual cone whose apex touches the target (C), for example the center thereof, the practitioner being able to select or validate one or one of the proposal(s) via the human-machine interface. This cone is normally included in the truncated volume (VR) delimiting the displayed volume.When multiple cones can be identified, it may be advantageous to choose the largest cone to determine an effective path for the safest possible insertion.

[0074] In order to limit the resources required for an augmented display (target, structures, trajectory) useful to the practitioner and to be able to carry out this display with real-time refresh, the method may consist, after selecting a proposed needle insertion trajectory (3) and entry point (PE) in the subject (S), in visualizing a limited cylindrical tubular volume (VT) with said trajectory as longitudinal median axis (AML) and making it possible to visualize the organs, vessels and / or tissues or structures (OVT) possibly present in this volume. This reduced display may also already be carried out during the exploratory phase of the possible trajectories. Thus, only the structures close to the insertion trajectory and therefore relevant will be displayed in the visualized limited volume. Such a tubular volume may for example have a diameter of between 1.5 and 3 cm, advantageously of the order of 2 cm.

[0075] According to another possible characteristic of the invention, alternative or complementary to the preceding characteristic, step g) may comprise the prior segmentation of the needle (3) in the images acquired successively as the needle (3) is inserted and in that the volume (VR) visualized consists of a volume elongated along the direction of insertion of the needle. (3), preferably a tubular volume comprising said target (C) and at least one end portion (3') of the needle, as well as the intermediate volume (VP) located between them, with the organs, vessels and / or tissues or structures (OVT) possibly present in this volume. It is thus possible to know quantitatively the distance to the target remaining to be covered by the needle during its insertion and after the acquisition of each 3D image. In addition, this arrangement also makes it possible to check whether the trajectory is still directed towards the target, the latter possibly being able to move during insertion. In addition, it will also be possible to visualize in real time the bending of the needle during its insertion and to carry out a possible slight trajectory correction if necessary.Finally, by limiting the display to such a limited volume and centered on the needle being inserted, possibly coupled with a more limited scope of exploration of the probe, it is possible to facilitate visualization in terms of resources and responsiveness, while limiting the risks of distraction by displaying irrelevant peripheral information.

[0076] In order to save time during acquisition and visualization, and therefore limit the duration of the patient's apnea, while providing the necessary visual information to the practitioner during insertion, it is very favorably provided that, in step g), the delimitation of the reduced volume (VR) intended to be visualized in 3D is controlled by the insertion movement of the needle (3) and subject to the fraction of the trajectory remaining to be covered at the end portion (3') of the needle (3) up to the target (C), this delimitation being carried out either upstream during the acquisition of the 2D images, or downstream during the 3D visualization.

[0077] In the two cases mentioned above, each of the volumes (VT) or (VR) visualized can therefore vary in size during insertion to be limited to a smaller volume comprising only an end portion (3') of the needle (3) and a volume (VP) preceding this end (3') during insertion up to the target (C), as shown for example in figure 4.

[0078] The invention also relates, according to a second aspect, to a collaborative method for inserting a needle (3) into a living subject (S), implementing a medical installation (5) comprising, on the one hand, at least one human-machine interface with a display device (6) and command or control means (7) and, on the other hand, at least the robotic arm (2) carrying said 2D ultrasound probe (1), the position and orientation of which are known in real time and can be used and recorded, concomitantly with the 2D images generated at each exploration, by said installation (5), said installation (5) further comprising another or second robotic arm (4) on which is fixed a needle holder or needle guide (8), at least the relative position of the two arms (2 and 4) between them being known permanently by said installation (5), said method comprising a preparatory planning phase preceding the actual insertion operation, the latter being carried out automatically by the second robotic arm (4) or by a practitioner with guidance by this second robotic arm (4), the needle holder or needle guide (8) being prepositioned and preoriented during the preparatory phase.

[0079] According to the invention, this insertion method is characterized in that it consists of implementing the assisted and interactive method for producing and displaying 3D ultrasound images as described above, and in particular the aforementioned steps a) to e) during a preparatory planning phase, step f) during a replanning phase following the preparatory planning phase and immediately before the start of the insertion of the needle (3) and step g) during the actual operation of the insertion of the needle (3), the replanning phase and the operation of inserting the needle (3) taking place during the same apnea phase of the subject (S).

[0080] This collaborative insertion process therefore closely integrates the assisted and interactive process of producing and displaying 3D ultrasound images and is dependent for its various constituent phases on the respectively associated stages of this latter process.

[0081] It should be noted that the performance of the insertion itself can be collaborative (guided by the needle holder held by the second robotic arm and pushed by the practitioner) or fully automatic by means of the second robotic arm alone (4) inserting the needle through the needle holder.

[0082] Preferably, and taking into account the fact that the target can move during insertion, it may be possible to consider compensating for any movements and displacements of the target occurring during insertion of the needle (3), by modifying the orientation of the needle holder or needle guide (8) by the arm. robotic (4) carrying it (and therefore the orientation of the trajectory of the needle), carried out automatically or by the practitioner, with a limiting constraint. Another reason for compensation may lie in a possible bending of the needle during insertion.

[0083] In accordance with an advantageous embodiment of the invention, the replanning phase, carried out at the start of the apnea during which the insertion must be carried out, consists of creating a new standardized 3D image in accordance with step f), checking the adequacy of the positioning of the needle holder (8) and of the resulting trajectory in relation to the situation visualized in this new 3D image and carrying out, if necessary, automatically or not, and at least after validation by the practitioner, a possible adjustment of the orientation of said needle holder or needle guide (8) necessary to find an optimized insertion trajectory taking into account a possible minor modification of the position of the target (C) and / or of the soft structures (OVT) between two apneas of the subject (S).

[0084] The adjustment will consist at most of slightly modifying the orientation of the needle holder (8) constrained by a passage of the trajectory through the entry point (PE) previously selected.

[0085] Finally, the invention also relates, according to a third aspect, and as is apparent for example from Figures 1 and 2, to a medical installation (5) for implementing the aforementioned image production method, and preferably the needle insertion method (3) as mentioned previously, this installation (5) comprising, on the one hand, at least one human-machine interface with a display device (6) and command or control means (7) and, on the other hand, at least the robotic arm (2) carrying said 2D ultrasound probe (1), the position and orientation of which are known in real time and can be used and recorded, concomitantly with the 2D images generated at each exploration, by said installation (5).

[0086] Of course, the 2D ultrasound probe (1) is connected to an image processing device enabling the production of 3D images and forming part of the medical installation.

[0087] This installation is characterized in that it also comprises, on the one hand, another or second robotic arm (4) on which a needle holder (8) is fixed, less the relative position of the two arms (2 and 4) between them being known permanently by said installation (5), and, on the other hand, means for the controlled acquisition, processing and exploitation of images produced by the 2D ultrasound probe (1) when the latter is moved during an exploration on a living subject (S) by the robotic arm (2) concerned, or by a practitioner with the assistance of said robotic arm (2), these means allowing the implementation of the assisted and interactive method of production and display of 3D ultrasound images described above.

[0088] The knowledge of the relative position of the two robotic arms (2 and 4) by the installation can result from their manufacture (mounted on the same base), from their possible mutual assembly after manufacture (removable mechanical connection between the two arms) or from their installation in situ. This intrinsic knowledge makes it possible in particular to avoid the use of a camera, with the resulting disadvantages, and from any recalibration operation. The two robotic arms (2 and 4) are preferably six-axis robots and together with their control and piloting means constitute a robotic device or robot cooperating interactively with the ultrasound device and the human-machine interface, where appropriate under the control of a centralized control unit of the installation.

[0089] Advantageously and as shown in Figures 1 and 2, the two robotic arms (2 and 4) are mounted on the same and single base (10), being advantageously kinematically and structurally separate and independent from each other. The two arms are not connected to each other, except when mounted on the same base. This base advantageously integrates a control unit for these two arms (2 and 4) and is preferably equipped with rolling means and associated with a control means (7) for each arm, for example a pedal, forming part of the human-machine interface. The two arms thus extend separately and freely from each other from the base, which contains their control and piloting means and physically determines their common reference frame.

[0090] The means for the controlled acquisition, processing and exploitation of images produced by the 2D ultrasound probe (1), in communicative connection with the display means (11), can for example be housed in this base (10).

[0091] Preferably, the human-machine interface comprises a control console (11), possibly mounted on rolling means, comprising computer means for calculating and processing data, in particular image processing algorithms and software, and the display device (6), advantageously in the form of a touch-sensitive display device, said control console (11) interacting with the two robotic arms (2 and 4).

[0092] With the following description, a particular embodiment of the invention is set forth in more detail, incorporating both methods and the installation, as well as their possible variants or options.

[0093] Firstly, a robotic device is provided comprising two robotic arms (2 and 4) mounted side by side, for example on the same base, one of which manipulates a 2D ultrasound probe (1) and the other of which manipulates a needle holder (8). This device is associated with a human-machine interface comprising a control console (11) with interactive screen (6) and control means (7).

[0094] The first arm (2) manipulates, autonomously or semi-autonomously, the 2D ultrasound probe (1) in order to scan a large area including the needle insertion region (3) and the surrounding structures (OVT). This scanning will thus make it possible to create 3D images on demand in near real time.

[0095] The very first time, the scan will be performed manually by the user (practitioner / clinician) who will co-manipulate the probe (1) under the assistance of the robotic arm. This makes it possible to facilitate but also to slightly constrain the practitioner's gesture to get closer to an ideal trajectory.

[0096] From this first recording, a trajectory will be calculated in order to make the movement of the ultrasound probe manipulated by the robot optimal in terms of speed, pressure applied and the number of 2D images acquired, allowing a high-quality 3D image to be obtained autonomously.

[0097] The position of the target (C) is detected automatically and in real time in the successively acquired 3D ultrasound images.

[0098] The structures to be preserved (OVT) are made visible in these 3D images so as to allow the clinician to visualize all the secure insertion options available and thus enable him to make a truly informed choice.

[0099] The speed of image acquisition and the planning rectification tools allow for the inevitable adjustments necessary to take into account the potential and almost inevitable modification of the target position (C) between the planning phase and the execution phase of the gesture (insertion). Indeed, the planning and execution phases are necessarily carried out from two different images since they are associated with two different apneas.

[0100] To do this, the second robotic arm (4) manipulates a needle guide or needle holder (8) which is always oriented towards the target (C), in other words the direction of the trajectory that it imposes on the needle that it receives always passes through the target (orientation control). This second arm (4) is able, after validation by the clinician, to slightly correct its orientation, and therefore that of the needle holder and the longitudinal direction and trajectory of the needle, to compensate for a change in position of the target (C) between the planning phase and the execution phase.

[0101] Throughout the manual insertion of the needle carried out by the practitioner and guided through the needle holder or needle guide (8), thanks to the automatic or semi-automatic, rapid and regular creation and visualization of 3D images, it is possible to visualize both the trajectory of the needle (3), the target (C) and the surrounding structures (OVT), allowing to guarantee the safety and precision of the gesture.

[0102] Comparison made after processing between an image acquired before and a second image acquired after processing makes it easier to assess the quality and extent of processing.

[0103] The 2D images that allow the creation of the first 3D image are acquired manually using the robot's ability to constrain the movements of the clinician's arm but also to assist him because:

[0104] - This allows the practitioner's skill to be used to acquire an ideally placed volume including the possible trajectories of the needle (3) and adjacent structures (OVT) without requiring robotic intelligence which is otherwise difficult to reconcile with regulatory obligations.

[0105] - This helps the practitioner to make the execution of his gesture easier thanks to the assistance of the robot which will be able to apply sufficient but minimal force in order to limit tissue deformation as much as possible.

[0106] - This makes it possible to constrain the movement of the probe in such a way as to acquire a set of coherent 2D images, associated with their positions, while avoiding acquiring overlapping images making the creation of the resulting 3D image difficult or even impossible.

[0107] - This makes it possible to record a trajectory of the robotic arm (2) very close to the final optimized trajectory, thus avoiding the risk of collision between said robotic arm (2) and the patient (S).

[0108] An optimal trajectory is obtained by calculation from the recorded trajectory acquired by the practitioner because:

[0109] - This allows minimal force to be applied to the tissues with the probe (1) while ensuring optimal image quality.

[0110] - This allows the speed of the probe (1) to be adapted to minimize the acquisition time while acquiring a sufficient number of 2D images to guarantee the quality of the resulting 3D image.

[0111] - This subsequently allows the robot to acquire continuous images autonomously or semi-autonomously (under the influence of the practitioner's hand) in order to be able to follow the movement of the needle (3) inside the patient (S) during insertion.

[0112] Automatic detection of the target (C) using for example artificial intelligence is implemented, but after an initial manual validation because:

[0113] - A first manual validation of the target (C) will make the detection more reliable.

[0114] - This will allow the ultrasound probe (1) to recognize the tumor (target) autonomously and thus ensure its position in near real time after each passage of the probe (1) carried by the robotic arm (2).

[0115] In the search for the best possible insertion trajectory, the robotic arm holding the needle holder (8) is constrained so that the direction of the needle trajectory defined by the needle holder always remains directed towards the target (C) because:

[0116] - It saves a lot of time and does not require tedious adjustment, while being easy since the information on the target's position is available.

[0117] - This allows you to explore a large number of positions in a simplified and user-friendly way.

[0118] For each position of the needle holder (8) and in real time, it is proposed to limit the visualization of the 3D image to a cylinder (VT, VR) located around the trajectory because:

[0119] - This facilitates the visualization of the target (C) and the surrounding soft / fragile structures (OVT), in particular vascular structures, thanks to the creation of a 3D negatoscope limited to this volume or through the use of volume rendering.

[0120] - This ensures that on or in a reasonably sized neighborhood around the needle trajectory (3), no fragile structures to be preserved are present.

[0121] A map of all potential options for the passage of the needle is proposed by juxtaposing the projection of the different volumetric representations on a planisphere because:

[0122] - The manual movement of the needle holder (8) constrained by the robotic arm always directed towards the target (C) makes it possible to obtain a visualization of the cylinder in projection in order to understand if fragile structures (OVT) are found in this volume. It is planned to precalculate these projections for all possible reasonable trajectories of the needle and to display the result on a map like a terrestrial planisphere on which land and sea are represented.

[0123] - In this way, it is possible to choose the best trajectory, that is, the one that allows needle insertion (3) while being as far as possible from the fragile structures (OVT) and having a priori knowledge (the practitioner) of all possible trajectories, thus avoiding wasting time in often tedious trial and error.

[0124] It is possible to calculate the position of the target, neighboring organs and their vascularization using a specially trained Artificial Intelligence or similar software, from an intraoperative 3D ultrasound image. This allows this 3D modeling to be used to automatically calculate the optimal trajectory to reach the target.

[0125] It is also possible, if necessary, to calculate the non-rigid registration of a preoperative 3D scanner image with the intraoperative 3D ultrasound image. This allows the use of 3D modeling of this preoperative image to automatically calculate the optimal trajectory to reach the target.

[0126] Once the planning has been carried out from a 3D ultrasound image that has become obsolete and the needle holder (8) has been positioned, a new 3D image is acquired, automatically or semi-automatically, at the start of the insertion phase. This acquisition makes it possible to detect the new position of the tumor (target) and to automatically carry out the very slight, but systematically necessary, modification of the orientation of the needle holder (8) so as to always be perfectly well directed towards the new position of the target (C) because:

[0127] - This allows us to take into account the inevitable modification of the target (C) between the planning phase and the insertion phase associated with images acquired during separate apneas since they are acquired at two different times.

[0128] - In this way, the modification of orientation of the needle holder (8) is restricted to a rotational movement around the entry point (PE) and is extremely limited, avoiding any risk of contact between the robotic arm (4) and the patient (S).

[0129] - This allows the practitioner to re-evaluate the cylinder (VT, VR) around the new trajectory. And therefore to confirm that the new trajectory is indeed valid.

[0130] The practitioner will insert the needle manually while being able to visualize the tissues and structures located on the trajectory of the needle (3) and situated between the tip of the needle (3') and the target (C) thanks to the 3D images acquired automatically or semi-automatically by the robotic arm (2) with the probe (1) because:

[0131] - This allows the position of the target (C) to be automatically detected and therefore ensures that the direction of the needle is always correct.

[0132] - This allows the practitioner, if the alignment is no longer perfect, to slightly modify the curvature of the needle to modify its trajectory.

[0133] - This allows the progression of the needle (3) to be visualized in near real-time in order to actually position the tip of the needle (3') in the center of the target (C). This is possible even if the target moves slightly under the effect of the deformations induced by the needle and therefore even if its position changes in terms of depth.

[0134] It is possibly possible to automatically extract the position of the needle (3) in the 3D ultrasound images using Artificial Intelligence or suitable similar software, in order to automatically calculate its progression in near real-time in the patient (S) so as not to rely solely on visual feedback.

[0135] It is planned to systematically and automatically deposit ultrasound gel on either side of the ultrasound probe (2) because:

[0136] - This will allow correct image quality to be maintained even when the probe (1) has made several back and forth movements and has cleaned the skin (8) of the patient (S).

[0137] - This avoids irritating the skin (8) of the patient (S) due to friction from the probe (1).

[0138] Once the needle treatment is complete, a new 3D image is acquired automatically or semi-automatically and is compared to a 3D image acquired before treatment to facilitate the evaluation of the treated area.

[0139] Indeed, comparing two similar images allows us to better identify their differences, in particular those induced by processing.

[0140] It may be considered to perform non-rigid registration of ultrasound images acquired before and after treatment in order to be able to compare them more easily and therefore better understand their differences, particularly those resulting from the treatment.

[0141] A detailed description of a possible protocol for implementing the invention integrating the two aforementioned methods and comprising nine successive phases is described below.

[0142] Phase 1: Positioning and preparing the patient

[0143] In this first phase, the patient (S) is positioned and prepared for the procedure. This includes preparing the area to be treated and positioning the patient to ensure optimal access for the practitioner.

[0144] Phase 2: Positioning the robotic device or robot

[0145] The robot comprising the two robotic arms (2 and 4) is positioned on the other side of the table (9) relative to the practitioner so as not to limit his movements, ensuring optimal coordination throughout the procedure.

[0146] Phase 3: Searching for the target using the ultrasound probe (1)

[0147] The practitioner manipulates the 2D transabdominal ultrasound probe to locate the target (C) to be treated. An AI algorithm assists by automatically detecting the target tumor, which speeds up and improves the accuracy of the localization process.

[0148] The ultrasound probe (1) is associated with the first arm (2) of the robot. The robot assists the practitioner so that the movement of the probe (1) can be carried out effortlessly, limiting its parasitic movements while applying limited but sufficient pressure on the skin (8) of the patient (S).

[0149] Phase 4: Creation of a 3D ultrasound image of the needle insertion area

[0150] After locating the target (C), the practitioner scans an area (RV volume region) with the ultrasound probe (1) that encompasses the reasonable potential trajectories of the needle. Scanning this area automatically creates a 3D image including the target (C), the possible trajectories and the surrounding structures (OVT).

[0151] Once this first acquisition is completed, the robot is able to perform similar acquisitions autonomously or semi-autonomously. At any time, the practitioner can now view a 3D image updated in near real time.

[0152] This first scan takes place during the patient's first apnea (S).

[0153] Phase 5: Planning the insertion trajectory

[0154] The practitioner chooses an optimal trajectory for the needle (3) by consulting a map that represents all possible trajectories. The map is created taking into account the presence of fragile structures (OVT) along their paths.

[0155] This map is presented concretely on the installation's touch display interface (6) in the form of a planisphere including risk-free zones and risk zones. This provides access to all possible options.

[0156] Phase 6: Positioning the needle holder

[0157] The practitioner positions the needle holder in the optimal position chosen using the trajectory options map. Since the needle holder is associated with the second robotic arm (4), when the practitioner moves the needle holder, the latter always remains oriented towards the target (C). In addition, the position of the needle holder (8) and the resulting trajectory are naturally reported on the real-time map (e.g. planisphere) corresponding to all the options to facilitate this positioning.

[0158] Phase 7: Insertion of the treatment needle

[0159] Since the insertion of the needle (3) takes place during a second apnea or an apnea following the apneas that led to the determination of the optimal trajectory, the position of the target (C) cannot be guaranteed since it comes from the image acquired during the first apnea or previous apneas. The practitioner will therefore use a new image acquired automatically just after this second apnea or subsequent apnea.

[0160] The new position of the target (C) is then detected automatically and the needle holder (8) will automatically make the slight correction to compensate for the target repositioning error between the two apneas.

[0161] After validation of the trajectory by the practitioner, in particular by visualizing an area corresponding to a cylinder (VT, VR) located along the trajectory of the needle, the latter can begin to insert the needle.

[0162] The clinician will insert the needle (3) while regularly asking the installation (5) comprising the robotic arm (2) with the probe (1) to update the visualization of the 3D image limited to a cylinder located around the trajectory of the needle for near real-time control of the precision of the procedure.

[0163] Phase 8: Target treatment by radiofrequency

[0164] Once the needle is correctly positioned, the practitioner can perform the radiofrequency treatment (or other operation involving the needle).

[0165] Phase 9: Evaluation of the extent of treatment

[0166] Once the treatment is carried out, the practitioner asks the installation (5) to acquire a new 3D image to assess the extent of the treatment. This makes it possible to check the effectiveness of the treatment by comparing two images, the first having been acquired before and the second after the treatment.

[0167] The aforementioned protocol ensures optimal precision, efficiency and safety throughout the insertion and treatment process.

[0168] It is noted that the invention can be implemented in its various aspects indifferently with a needle holder or a needle guide, the mention of one being worth mentioning the other as well.

[0169] Of course, the invention is not limited to the embodiment described and shown in the attached drawings. Modifications remain possible, in particular from the point of view of the constitution of the various elements or by substitution of technical equivalents, without departing from the scope of protection of the invention.

Claims

Claims

1. Assisted and interactive method for producing and displaying 3D ultrasound images, by controlled acquisition and processing of 2D images acquired by means of a 2D ultrasound probe (1) which can be moved over a living subject (S) by a robotic arm (2), or by a practitioner with the assistance of a robotic arm (2), in particular before a needle insertion operation (3) using a needle holder or needle guide (8) in said subject (S), during a preparatory planning phase, the insertion operation being carried out for example by means of or with the assistance of another robotic arm (4) equipped with a needle holder or needle guide (8), said method implementing a medical installation (5) comprising, on the one hand, at least one human-machine interface with a display device (6) and command or control means (7) and, on the other hand, at least the robotic arm (2) carrying said 2D ultrasound probe (1),whose position and orientation are known in real time and can be used and recorded, concomitantly with the 2D images generated at each exploration, by said installation (5), method characterized in that it comprises at least the steps consisting of: a) locating one or the target (C) to be treated in the subject (S) by exploratory movement of the probe (1) on the skin (P) of the latter and analysis of the acquired 2D ultrasound images, by a practitioner possibly assisted by suitable detection software; b) producing a first 3D image of a volume region (RV) of the subject (S), extending from the skin (P) to the target (C) concerned and containing the latter and the reasonable possible introduction trajectories of a needle (3) to reach said target (C) estimated by the practitioner, this 3D image being obtained from a set of acquired 2D images, during an apnea phase of the subject (S),by moving the 2D ultrasound probe (1) by the practitioner with the assistance of the robotic arm (2) and potentially by using the location information from step a); c) automatically determining, during or after step b), the parameters and the path of an optimized and reproducible trajectory for the probe, 2D ultrasound (1) allowing the production of substantially standardized and identically framed 3D images of the aforementioned volume region (VR); d) producing one, or successively several, substantially standardized 3D image(s) of this volume region (VR), in quasi-real time and automatically or at the command of the practitioner, during a breath-hold phase of the subject (S) and on the basis of 2D images freshly acquired following a new exploration with the 2D ultrasound probe (1) carried out by applying the parameters and following the path which were determined during step c);e) visualizing, in this or these substantially standardized 3D images of the volume region (RV) of the subject (S), the organs, vessels and / or similar tissues or soft structures (OVT) present between the skin (P) and said target (C), as well as the latter two, in order to determine the positioning and orientation of a needle holder or needle guide (8) for at least one possible optimized insertion trajectory of the needle (3); f) repeating steps d) and e) just before the start of the insertion of the needle (3), which is then positioned in accordance with the optimized insertion trajectory selected by the practitioner and visualized in the current 3D image, and automatically determining and visualizing the position of the target (C) at this instant in order to possible adjust the positioning and / or orientation of the needle holder or needle guide (8), in particular just before the actual insertion of the needle.;

2. Method according to claim 1, characterized in that steps d) and e) are associated with a preparatory phase of planning the intervention resulting, where appropriate by successive repetitions, in obtaining a substantially standardized 3D image of the volume region (RV) deemed optimal, and in its visualization, for example in the form of a planisphere, with integrated indication of the possible trajectories of insertion of the needle (3) determined automatically or by the practitioner, then of a proposal of a possible optimized and secure trajectory of insertion of the needle (3), selected automatically or by the practitioner.

3. Method according to claim 1 or 2, characterized in that step f) corresponds to a replanning phase preceding, during the same apnea phase, imminently the start of the insertion phase and taking into account a probable minor modification of the position of the target (C) and of the soft structures (OVT) between two apneas of the subject (S), this step f) comprising the creation and visualization of a new substantially standardized 3D image of the volume region (RV) of the same type as that obtained at the end of step e), this with a view to a possible automatic or manual adjustment of the positioning and / or the orientation of the needle holder (8) corresponding to the optimized trajectory selected at the end of step e) and at least one validation of this trajectory, possibly finely readjusted, by the practitioner.

4. Method according to any one of claims 1 to 3, characterized in that, in step b), the trajectories considered reasonable by the practitioner are defined by the latter by means of a median insertion direction passing through the target (C) and correspond to rectilinear trajectories included in a conical or truncated volume, preferably with a circular or ellipsoidal base, possibly corresponding to the reduced volume (VR), extending by tapering from the skin (P) to the target (C) at least and with a median axis (AML) passing through the target (C), this median axis corresponding to the aforementioned median direction and being able to be materialized by a corresponding positioning of the needle holder (8), and in that at least one possible, or even optimized, insertion trajectory,is determined after analysis of the content of said first 3D image and taking into account the specific circumstances linked to the subject (S) and the configuration of the intervention, automatically or semi-automatically or by the practitioner.,

5. Method according to any one of claims 1 to 4, characterized in that the substantially standardized 3D images of step c) and step d) consist of 3D images produced from 2D images acquired along intersecting planes with respect to the reasonable insertion trajectories of step b) and with sufficient and constant quality, namely with an optimized and reproducible trajectory of the 2D ultrasound probe (1), that is to say in particular by applying a substantially constant and as low as possible force at the 2D probe (1) allowing the acquisition of 2D images with the aforementioned quality and by moving the latter with a controlled and optimized acquisition speed, in particular the highest possible speed, nevertheless guaranteeing sufficient density and quality of volumetric information to produce images of acceptable quality.

6. Method according to any one of claims 1 to 5, characterized in that, in step e), the visualization of the tissues or structures (OVT) in the substantially standardized 3D image of the volume region (VR) is carried out by means of a visualization mode among: i) a successive visualization of sectional views corresponding to 2D interpolated images, ii) a 3D visualization obtained by a direct volume rendering method and iii) a 3D visualization obtained by a surface rendering method, after carrying out a segmentation phase of said substantially standardized 3D image of the volume region (VR).

7. Method according to any one of claims 1 to 6, characterized in that it also comprises, after step f) and during the insertion of the needle (3), a step g) consisting of: visualizing in quasi-real time the progressive insertion of the needle (3) by repeated creation and visualization of refreshed 3D images of a volume (VR), for example substantially cylindrical or truncated, as a fraction of the aforementioned volume region (RV) of the subject (S), preferably as a variable fraction of this volume region, the 3D images of this volume (VR) showing either said target (C) and at least one end portion (3') of the needle (3), preferably the entire part of the needle (3) inserted into the subject (S), or at least one end portion (3') of the needle (3) and a volume (VP) preceding this end (3') during the insertion,this visualization of refreshed 3D images of said volume (VR) preferably continuing at least until the target (C) is reached by the needle (3), the two steps f) and g) being carried out together during the same apnea phase of the subject (S).,

8. Method according to any one of claims 1 to 7, characterized in that it comprises an additional final step h) consisting of simultaneously displaying, and where appropriate in a superimposed manner, at least two 3D images of the volume region (RV) of the subject (S) extending from the skin (P) up to the target (C) concerned, and including the latter, or alternatively a more restricted volume region enclosing the location of the target (C), one of which was taken before the intervention and treatment using the needle (3) and the other of which was taken after this intervention.

9. Method according to any one of claims 1 to 8, characterized in that it consists of carrying out the explorations with the probe (1), which are subsequent to step c) and initiated at the initiative of the practitioner, in a semi-automatic or collaborative manner, the movement of displacement of the probe (1) and the extent of the exploration being determined by manual action of the practitioner, the actual displacement of the probe (1) being nevertheless constrained by the trajectory parameters determined in step c), the exploration movements being able to be carried out either bidirectionally or unidirectionally with initial repositioning of the probe (1) after completion of an exploration.

10. Method according to any one of claims 1 to 6, characterized in that the analysis of the 2D images acquired in step a), with a view to identifying and locating the target (C) to be treated, for example a cancerous tumor, is carried out by or with the assistance of an algorithm, for example an artificial intelligence program having previously undergone adequate training, this location of the target (C) being repeated automatically at each subsequent exploration and display of the 3D image. [Claim 1 1] Method according to any one of claims 1 to 6 and 10, characterized in that the movement by the practitioner of the ultrasound probe (1) in steps a) and b), is controlled by constraint and limitation of movement by the robotic arm (2), in terms of orientation and position of the probe (1) relative to the skin (P) of the subject (S) and speed of movement, the practitioner being able to advantageously adjust the pressure setpoint for applying the probe (1) to the skin (P), controlled by the robotic arm (2) during the movement and / or deactivate the assistance of the robotic arm (2), by means of suitable means forming part of the human-machine interface (6, 7).

12. Method according to any one of claims 1 to 6, and 9 to 1 1, characterized in that the pressure of application of the probe (1) on the skin (P) is, during the movement of the latter at each exploration carried out during phase b) and / or phase d), automatically controlled by the robotic arm (2) by real-time control of the value of this pressure by means of an appropriate sensor or a control loop exploiting the result of an analysis of the quality of the 2D ultrasound image generated in real time, said application pressure being minimized during phases d), e) and f) to a value just allowing the acquisition of 2D images of sufficient quality to produce a standardized 3D image, in order to generate minimal deformation of the tissues visualized.

13. Method according to any one of claims 1 to 6 and 9 to 12, characterized in that the production of substantially standardized 3D images at steps c), d) and f) is carried out automatically, on the basis of the 2D images produced by the probe (1) manipulated by the robotic arm (2), at each command or with the participation of the practitioner, by making the probe (1) follow a determined trajectory, in particular optimized in terms of position and orientation of the probe (1) relative to the skin (P) of the subject (S) and speed of movement of the probe (1), during movement, this trajectory having been determined during or after step b).

14. Method according to any one of claims 1 to 13, characterized in that it consists, on demand or automatically, in spreading by spraying, by means of a suitable device associated with or integrated into the probe (1), ultrasound contact gel in front of and / or behind the probe (1) during one, preferably each exploratory movement.

15. Method according to any one of claims 1 to 6 and 9 to 13, characterized in that the 3D visualization of the organs, vessels and / or tissues present between the surface of the skin (P) and the target (C) as provided in step e), comprises the display by overprinting of a collimation or aiming pattern centered on the target (C), making it possible to define and visually simulate the virtual trajectory of insertion of the needle (3) for a given positioning and orientation of the needle holder (8), and advantageously by interaction via the human-machine interface, to explore the possibilities of rectilinear trajectory of insertion of the needle (3) and entry point in the subject (S) and to select one or more proposals.

16. Method according to claim 15, characterized in that it consists of carrying out the exploration of the possibilities of rectilinear trajectory of insertion of the needle (3) by simulating or by carrying out collaboratively a constrained displacement of the other robotic arm (4) and therefore of the needle holder (8), for example in the form of a constrained movement of the ball joint, in such a way that the insertion trajectory always passes through the center of the target (C) whatever the position of said arm (4) and said needle holder (8).

17. Method according to any one of claims 1 to 6 and 9 to 15, characterized in that the determination of the positioning and orientation of a needle holder (8) with a view to a possible optimal insertion trajectory by using a 3D visualization of the organs, vessels and / or tissues or structures (OVT) present between the skin (P) and the target (C), as provided in step e), comprises the automatic analysis and display of at least one, preferably several, proposal(s) of rectilinear trajectory of insertion of the needle (3) and entry point (PE) in the subject (S), each associated with an orientation or a plurality of orientations contained in a virtual cone whose apex touches the target (C), for example the center thereof, the practitioner being able to select or validate the or one of the proposal(s) via the human-machine interface.

18. Method according to any one of claims 15 to 17, characterized in that it consists, after selecting a proposed trajectory for insertion of the needle (3) and entry point (PE) in the subject (S), in visualizing a limited cylindrical tubular volume (VT) with said trajectory as longitudinal median axis (AML) and making it possible to visualize the organs, vessels and / or tissues or structures (OVT) possibly present in this volume.

19. Method according to claim 7, characterized in that step g) comprises the prior segmentation of the needle (3) in the images acquired successively as the needle (3) is inserted and in that the volume (VR) visualized consists of a volume elongated in the direction of insertion of the needle (3), preferably a tubular volume comprising said target (C) and at least one end portion (3') of the needle, as well as the intermediate volume (VP) located between them, with the organs, vessels and / or tissues or structures (OVT) possibly present in this volume.

20. Method according to any one of claims 7 and 19, characterized in that, in step g), the delimitation of the reduced volume (VR) intended to be visualized in 3D is controlled by the insertion movement of the needle (3) and subject to the fraction of the trajectory remaining to be covered at the end portion (3') of the needle (3) up to the target (C), this delimitation being carried out either upstream during the acquisition of the 2D images, or downstream during the 3D visualization.

21. Collaborative method for inserting a needle (3) into a living subject (S), implementing a medical installation (5) comprising, on the one hand, at least one human-machine interface with a display device (6) and command or control means (7) and, on the other hand, at least the robotic arm (2) carrying said 2D ultrasound probe (1), the position and orientation of which are known in real time and can be used and recorded, concomitantly with the 2D images generated at each exploration, by said installation (5), said installation (5) further comprising another or second robotic arm (4) on which a needle holder or needle guide (8) is fixed, at least the relative position of the two arms (2 and 4) between them being known permanently by said installation (5), said method comprising a preparatory planning phase preceding the actual insertion operation,the latter being carried out automatically by the second robotic arm (4) or by a practitioner with guidance by this second robotic arm (4), the needle holder or needle guide (8) being prepositioned and preoriented during the preparatory phase, method characterized in that it consists of implementing the assisted and interactive method for producing and displaying 3D ultrasound images according to any one of claims 1 to 20, and in particular steps a) to e) of claim 1 during a preparatory planning phase, step f) during a replanning phase following the preparatory planning phase and immediately before the start of the insertion of the needle (3) and step g) during the actual operation of inserting the needle (3), the phase of, replanning and the needle insertion operation (3) taking place during the same apnea phase of the subject (S).

22. Insertion method according to claim 21, characterized in that it consists of compensating for any movements and displacements of the target occurring during the insertion of the needle (3), by a modification of the orientation of the needle holder (8) by the robotic arm (4) carrying it, carried out automatically or by the practitioner, with a limiting constraint.

23. Insertion method according to claim 21 or 22, characterized in that the replanning phase, carried out at the start of the apnea during which the insertion is to be carried out, consists of creating a new standardized 3D image in accordance with step f), checking the adequacy of the positioning of the needle holder (8) and of the resulting trajectory in relation to the situation visualized in this new 3D image and carrying out, if necessary, automatically or not, and at least after validation by the practitioner, a possible adjustment of the orientation of said needle holder (8) necessary to find an optimized insertion trajectory taking into account a possible minor modification of the position of the target (C) and / or of the soft structures (OVT) between two apneas of the subject (S).

24. Medical installation (5) for implementing the assisted and interactive method for producing and displaying 3D ultrasound images according to any one of claims 1 to 20, and preferably also the needle insertion method (3) according to any one of claims 21 to 23, this installation (5) comprising, on the one hand, at least one human-machine interface with a display device (6) and command or control means (7) and, on the other hand, at least the robotic arm (2) carrying said 2D ultrasound probe (1), the position and orientation of which are known in real time and can be used and recorded, concomitantly with the 2D images generated at each exploration, by said installation (5), installation (5) characterized in that it also comprises, on the one hand, another or second robotic arm (4) on which a needle holder (8) is fixed,at least the relative position of the two arms (2 and 4) between them being known permanently by said installation (5), and, on the other hand, means for, the controlled acquisition, processing and exploitation of images produced by the 2D ultrasound probe (1) when the latter is moved during an exploration on a living subject (S) by the robotic arm (2) concerned, or by a practitioner with the assistance of said robotic arm (2).

25. Medical installation according to claim 24, characterized in that the two robotic arms (2 and 4) are mounted on the same and single base (10), being advantageously kinematically and structurally separate and independent, this base (10) advantageously integrating a control unit for these two arms (2 and 4) and being preferably equipped with rolling means and associated with a control means (7) for each arm, for example a pedal, forming part of the human-machine interface.

26. Medical installation according to claim 24 or 25, characterized in that the human-machine interface comprises a control console (11), possibly mounted on rolling means, comprising computer means for calculating and processing data, in particular image processing algorithms and software, and the display device (6), advantageously in the form of a touch display device, said control console (11) interacting with the two robotic arms (2 and 4).

Images