Method for producing and displaying 3D ultrasound images, needle insertion method using same and installation for their implementation

Abstract

Method for producing and displaying 3D ultrasound images, needle insertion method using it and installation for their implementation The invention relates to an assisted and interactive method for producing and displaying 3D ultrasound images, by controlled acquisition and processing of 2D images acquired by means of a 2D ultrasound probe (1) which can be moved over a living subject (S) by a robotic arm (2), or by a practitioner with the assistance of a robotic arm (2), in particular before and during an operation of inserting a needle (3) into said subject (S), this operation being carried out for example by means of or with the assistance of another robotic arm (4) equipped with a needle holder (8).This method is used during an insertion planning phase, during a replanning phase just before insertion and during insertion, and achieves the display of relevant volumes adapted to the different phases showing the needle trajectory (3), the fragile structures (OVT) and the target (C). Figure to be published with the abstract: Fig. 3.

Description

Title of the invention: Method for producing and displaying 3D ultrasound images, needle insertion method using it and installation for their implementation

[0001] The present invention relates to the fields of medical imaging and assistance in the positioning of needles under ultrasound guidance assisted by a robotic installation, and more particularly the creation and exploitation of 3D ultrasound images from the autonomous or semi-autonomous acquisition of a set of 2D ultrasound images and its advantageous use in relation to procedures for inserting and implanting sampling, injection or localized (target) treatment needles in subjects.

[0002] Its objects are an assisted and interactive method for producing and displaying 3D ultrasound images by controlled acquisition and processing of 2D images, a needle insertion method, automatic or preferably assisted or collaborative, using the aforementioned method, and a robotic medical installation for implementing these two methods.

[0003] A medical technique currently in full development is percutaneous surgery using a needle inserted into the patient, both for the treatment of cancers and for injection or localized sampling.

[0004] In the case of the treatment of cancers located in the abdominal region, it consists of positioning the end of one or more needles in a tumor, passing through the abdominal wall. The destruction of the tumor takes place in particular by generating significant temperature variations (hot or cold) at the end of the needle. It is thus possible, for example, to treat tumors by radiofrequency, cryoablation, microwaves or even by electroporation.

[0005] This approach is gaining popularity because it is inexpensive, minimally invasive, and advances in ablation techniques make it possible to treat targets of increasingly diverse sizes, shapes, and locations. This allows this type of treatment to be performed most often on an outpatient basis, a source of savings for the health system but also of well-being for the patient.

[0006] The main difficulty associated with this curative technique is the precise positioning of the needles, while preserving the surrounding fragile structures. The correct positioning of the needles is a gesture that requires singular expertise.

[0007] In the case of the pancreas, the kidney and the liver, these organs being made up of soft tissues and moreover located in the abdominal cavity, they are subject to displacements as well as to deformations. These displacements and these deformations are notably induced by physiological movements, mainly respiratory movement, and the insertion of the needle itself.

[0008] It is essential to note that if the patient is required to hold his breath during the insertion of the needles, two successive apneas in no way guarantee that the target will be located in a similar manner. The same applies to structures (tissues, organs, vessels) located between the skin and the target.

[0009] Currently, various technical solutions are known for carrying out needle insertion processes (assisted or guided) in patients or subjects, based on various 3D medical imaging technologies (scanner, MRI or 3D ultrasound). They all have more or less significant drawbacks, in terms of costs, complexity of implementing the necessary installations, duration of the procedure, harmful exposure to radiation, compatibility with an environment having a strong magnetic field, comfort of use for the practitioner and / or precision of execution, in particular linked to the certain obsolescence of the imaging data at the time of the intervention since they were acquired well before the intervention during the planning phase.

[0010] Thus, the known methods implementing X-ray imaging suffer from a complex installation configuration (markers, cameras), a high cost of the imaging equipment (CT scanner), the impact of a high dose of X-rays and above all the obsolescence of the imaging data during their actual use, due to the acquisition and processing times of the latter.

[0011] The same defects are identified for solutions implementing an MRI process, except for the absence of X-ray dose, but for MRI, the presence of a high intensity magnetic field significantly limits the availability of compatible processing equipment and has a major impact on its cost.

[0012] Finally, methods using 3D ultrasound scanners to assist or guide the insertion of needles are also already known, but they also have numerous limitations.

[0013] Thus, these 3D ultrasound scanners are expensive, lack precision (poor quality images compared to 2D ultrasound imagers), use a bulky probe that is tedious to handle, also require a spatial location and referencing system (camera or electromagnetic sensor) and are not optimized in terms of data processing (quantity and relevance of the data acquired and processed). In addition, the problem of data obsolescence also affects the images provided by these 3D ultrasound imagers, unlike 2D ultrasound imagers (which are not always suitable for use in the aforementioned context and require controlled movement to provide usable images).

[0014] There are also various solutions for assisting the insertion of needles using robotic devices, most of which are associated with the use of a scanner-type imager and a camera. However, these solutions require a recalibration between the frame of reference of the imager and that of the robot arm, and the implementation of markers in the scanner and on the robot that can be identified by the camera and the scanner.

[0015] Furthermore, these known solutions also do not take into account the highly probable displacement of the target and the soft and fragile structures (affected by the insertion) between two apnea phases and makes blind insertion carried out within the framework of these solutions problematic and risky. In addition, the ergonomic difficulties linked to the necessary use of a specific camera represent a significant challenge, which can impact the efficiency and fluidity of procedures in the operating room. Furthermore, it should be considered that the trajectory chosen by the clinician for the insertion of the needle is not necessarily the best: this limitation raises questions about the flexibility and adaptability of the method according to the anatomical particularities of each patient.Finally, the economic aspect of these solutions cannot be ignored: its high cost, linked both to the qualified personnel and the specific infrastructures necessary for its implementation, represents a major obstacle to its widespread adoption, particularly in a context of limited resources.

[0016] The aim of the present invention is to propose a new solution making it possible to overcome at least the main limitations mentioned above, in particular those linked to the obsolescence of the images used, to the presence of a camera, to the absence of consideration of changes occurring between two apneas and to limited collaboration between the robotic devices and the practitioner, all while exploiting the advantages linked to the use of ultrasound.

[0017] According to a first and main aspect of the invention, the invention aims to provide a relevant, ergonomic and interactive visual aid for the practitioner, at least before and during an intervention involving needle insertion.

[0018] For this purpose, its main object is an assisted and interactive method for producing and displaying 3D ultrasound images, by controlled acquisition and processing of 2D images acquired by means of a 2D ultrasound probe which can be moved over a living subject by a robotic arm, or by a practitioner with the assistance of a robotic arm, in particular before and during an operation of inserting a needle into said subject, this operation being carried out for example by means of or with the assistance of another robotic arm equipped with a needle holder, said method implementing a medical installation comprising, on the one hand, at least one human-machine interface with a display device and command or control means and, on the other hand, at least the robotic arm carrying said 2D ultrasound probe, the position and orientation of which are known in real time and can be used and recorded, concomitantly with the 2D images generated at each exploration, by said installation,

[0019] method characterized in that it comprises at least the steps consisting of:

[0020] a) to locate one or the target to be treated in the subject by exploratory displacement of the probe on the latter's skin and analysis of the acquired 2D ultrasound images, by a practitioner possibly assisted by suitable detection software;

[0021] b) producing a first 3D image of a volume region of the subject, extending from the skin to the target concerned and containing the latter and the reasonable possible trajectories of introduction of a needle to reach said target estimated by the practitioner, this 3D image being obtained from a set of 2D images acquired, during a phase of apnea of ​​the subject, by movement of the 2D ultrasound probe by the practitioner with the assistance of the robotic arm and potentially by exploitation of the location information of step a);

[0022] c) automatically determining, during or after step b), the parameters and the path of an optimized and reproducible trajectory for the 2D ultrasound probe allowing the production of substantially standardized and identically framed 3D images of the aforementioned volume region;

[0023] d) producing one, or successively several, substantially standardized 3D image(s) of this volume region, in quasi-real time and automatically or on command of the practitioner, during a phase of apnea of ​​the subject and on the basis of 2D images freshly acquired following a new exploration with the 2D ultrasound probe carried out by applying the parameters and following the path which were determined during step c);

[0024] e) visualizing, in this or these substantially standardized 3D images of the volume region of the subject, the organs, vessels and / or tissues or similar soft structures present between the skin and said target, as well as the latter two, in order to enable the positioning and orientation of a needle holder to be determined for at least one possible optimized insertion trajectory of the needle;

[0025] f) repeating steps d) and e) just before the start of the insertion of the needle, which is then positioned in accordance with the optimized insertion trajectory selected by the practitioner and visualized in the current 3D image, and automatically determining and visualizing the position of the target at this instant with a view to a possible adjustment of the positioning and / or orientation of the needle holder, in particular just before the actual insertion of the needle;

[0026] g) then to visualize in quasi-real time the progressive insertion of the needle by repeated creation and visualization of refreshed 3D images of a volume, for example substantially cylindrical or truncated cone-shaped, as a fraction of the aforementioned volume region of the subject, preferably as a variable fraction of this volume region, the 3D images of this volume showing either said target and at least one end portion of the needle, preferably the entire part of the needle inserted into the subject, or at least one end portion of the needle and a volume preceding this end during insertion, this visualization of refreshed 3D images of said volume preferably continuing at least until the target is reached by the needle,

[0027] the two steps f) and g) being carried out together during the same apnea phase of the subject.

[0028] It also has as objects, according to two other aspects, a needle insertion method using the aforementioned 3D ultrasound image production and display method, and a medical installation for implementing these two methods.

[0029] The invention will be better understood, thanks to the following description, which relates to a preferred embodiment, given by way of non-limiting example, and explained with reference to the appended schematic drawings, in which:

[0030] [Fig.l] schematically represents a medical installation according to a preferred embodiment, for implementing the imaging and needle insertion methods according to the invention;

[0031] [Fig.2] represents the installation of [Fig.l] during an intervention on a subject, here a human patient;

[0032] [Fig.3] is a schematic representation in detail and in section of a part of the body of the subject just before needle insertion;

[0033] [Fig.4] is a representation similar to that of [Fig.3] during the insertion of the needle, for example as a partial and detailed view (on another scale) of the scene represented [Fig.2];

[0034] [Fig.5] is a representation of a set of acquired 2D ultrasound images, automatically or collaboratively with the practitioner, via the probe carried by the first robotic arm forming part of the installation according to the invention;

[0035] [Fig.6] is a representation of a volumetric region of a patient's abdomen in the form of a planisphere showing the fragile / soft structures and allowing the estimation of the different possible trajectories for inserting a needle in order to reach the target;

[0036] [Fig.7A] and [Fig.7B] are 3D images of a tubular / truncated volume region of the patient's body between the skin and the target and surrounding a needle trajectory, viewed from two different directions;

[0037] [Fig.8] is an image showing the tubular / truncated region of Figures 7A and 7B viewed along the insertion direction, with the entry point marked by a cross, and,

[0038] [Fig.9] is a 3D image showing a needle having reached the target.

[0039] The invention relates, according to a first aspect of the invention, to an assisted and interactive method for producing and displaying 3D ultrasound images, by controlled acquisition and processing of 2D images acquired by means of a 2D ultrasound probe (1) which can be moved over a living subject (S) by a robotic arm (2) carrying it, or by a practitioner with the assistance of a robotic arm (2) moved by the latter and holding said probe (1), in particular before and during an operation of inserting a needle (3) into said subject (S), this latter operation being carried out for example by means of or with the assistance of another robotic arm (4) equipped with a needle holder (8).

[0040] This method implements a medical installation (5) comprising, on the one hand, at least one human-machine interface with a display device (6) and command or control means (7) and, on the other hand, at least the robotic arm (2) carrying said 2D ultrasound probe (1), the position and orientation of which are known in real time and can be used and recorded, concomitantly with the 2D images generated at each exploration, by said installation (5).

[0041] This method is characterized in that it comprises at least the steps consisting of:

[0042] a) locating one or the target (C) to be treated in the subject (S) by exploratory movement of the probe (1) on the skin (P) of the latter and analysis of the acquired 2D ultrasound images, by a practitioner possibly assisted by suitable detection software;

[0043] b) producing a first 3D image of a volume region (RV) of the subject (S), extending from the skin (P) to the target (C) concerned and containing the latter and the reasonable possible trajectories of introduction of a needle (3) to reach said target (C) estimated by the practitioner, this 3D image being obtained from a set of 2D images acquired, during an apnea phase of the subject (S), by movement of the 2D ultrasound probe (1) by the practitioner with the assistance of the robotic arm (2) and potentially by exploitation of the location information from step a);

[0044] c) automatically determining, during or after step b), the parameters and the path of an optimized and reproducible trajectory for the 2D ultrasound probe (1) allowing the production of substantially standardized and identically framed 3D images of the aforementioned volume region (VR);

[0045] d) to produce one, or successively several, substantially standardized 3D image(s) of this volume region (VR), in quasi-real time and automatically or on command of the practitioner, during a phase of apnea of ​​the subject (S) and on the basis of freshly acquired 2D images following a new exploration with the 2D ultrasound probe (1) carried out by applying the parameters and following the path which were determined during step c);

[0046] e) visualizing, in this or these substantially standardized 3D images of the volume region (RV) of the subject (S), the organs, vessels and / or tissues or similar soft structures (OVT) present between the skin (P) and said target (C), as well as the latter two, in order to enable the positioning and orientation of a needle holder (8) to be determined for at least one possible optimized insertion trajectory of the needle (3);

[0047] f) repeating steps d) and e) just before the start of the insertion of the needle (3), which is then positioned in accordance with the optimized insertion trajectory selected by the practitioner and visualized in the current 3D image, and automatically determining and visualizing the position of the target (C) at this instant with a view to a possible adjustment of the positioning and / or orientation of the needle holder (8), in particular just before the actual insertion of the needle;

[0048] g) then visualizing in quasi-real time the progressive insertion of the needle (3) by repeated creation and visualization of refreshed 3D images of a volume (VR), for example substantially cylindrical or truncated, as a fraction of the aforementioned volume region (RV) of the subject (S), preferably as a variable fraction of this volume region, the 3D images of this volume (VR) showing either said target (C) and at least one end portion (3') of the needle (3), preferably the entire part of the needle (3) inserted into the subject (S), or at least one end portion (3') of the needle (3) and a volume (VP) preceding this end (3') during the insertion, this visualization of refreshed 3D images of said volume (VR) preferably continuing at least until the target (C) is reached by the needle (3),

[0049] the two steps f) and g) being carried out together during the same apnea phase of the subject (S).

[0050] Thanks to the aforementioned provisions of the invention, it is possible to propose a solution making it possible to achieve the set goal. Indeed, the method in question makes it possible to provide continuous, collaborative and interactive assistance to the practitioner during the preparatory phase of planning the intervention, during the fine adjustment phase just before the start of the insertion of the needle and throughout the insertion time until the target is reached. This method exploits the advantages of 2D ultrasound, while allowing 3D visualization, this by means of particular provisions. The method also takes advantage of preliminary targeted learning phases, provided by the practitioner for the robotic arm (2) carrying the probe (1) and the software means forming part of the installation (5), to subsequently provide effective assistance said practitioner by automation or semi-automation of ultrasound explorations and automation of target recognition and tracking.

[0051] In terms of information intended for the practitioner, the method according to the invention provides an evolving 3D visualization. Thus, it may be provided, first of all, to visualize a large volume region (RV) encompassing the target and a surrounding area, the skin (or at least one internal area close to the skin), at least the various reasonable insertion trajectories and all the soft and fragile structures (OVT) located in this region, this during step b) and at least part of a preliminary planning phase; then to visualize a smaller volume (VR, VT) comprising at least the end of the needle and at least one intermediate volume just large enough to be able to know and take into account the presence of the target, organs, vessels and / or tissues or similar fragile structures present in this environment, this just before and during the intervention.

[0052] Alternatively, and depending on the experience and wishes of the practitioner, the visualized volume may, from step b), be limited to a smaller volume than the volume region (RV) explored by the probe (1), for example a truncated cone volume (VR) extending around a median insertion direction deemed reasonable by the practitioner (passing through the center of the target) and between the skin and the target. For example, such a truncated cone would have a base in the form of a disc or ellipse with a diameter at the base of approximately 5 to 10 cm at the skin level) and the apex of which surrounds the center of the target, or even the latter.

[0053] The implementation of a robotic arm carrying the 2D ultrasound probe, subject to learning of the volume to be explored and capable of then carrying out, at the request of the practitioner, semi-automatic or automatic acquisitions with optimized parameters and therefore standardized 3D visualizations, relieves the practitioner and guarantees constant image quality.

[0054] Furthermore, adapting the processing and display of the images to the useful volume during the different phases allows in particular faster refreshing of the images when the provision of real-time information is important (during insertion), as well as easier interpretation when the area viewed is limited substantially to the area of ​​interest at the time of acquisition.

[0055] During step b), the robotic arm (2) can facilitate the movement of the probe (1) by applying a controlled force to it so that the practitioner is not obliged to force during this exploration, while avoiding excessive pressure.

[0056] It should be noted that, during each exploration with the 2D ultrasound probe (1) and during the insertion of the needle (3), the patient is assumed to be in apnea.

[0057] Furthermore, by means of the human-machine interface (6, 7), the practitioner is able to trigger explorations and recovery of 2D images with the probe (1) held by the robotic arm (2), to navigate in the 3D images produced by changing in particular the viewing angle in relation to the displayed volume (for example to view this volume with its contents along the axis of a needle insertion trajectory, which is advantageously merged with the median axis of this volume) and to choose the components that it wishes to see displayed (apart from the automatic acquisition and viewing operations).

[0058] The method according to the invention therefore provides interactive, almost instantaneous visual assistance, timed to the patient's breathing, non-interfering with the practitioner's other tasks and adaptive according to the progress of the treatment protocol. This assistance also continues continuously throughout the different phases of the protocol implemented: prior planning, rapid replanning just before insertion and the insertion phase.

[0059] While steps b) and c) correspond to a learning phase of the robotic arm (2) carrying the probe (1), in order to then allow it to carry out explorations automatically or semi-automatically, steps d) and e) are associated with a preparatory phase of planning the intervention resulting, where appropriate by successive repetitions, in obtaining a substantially standardized 3D image of the volume region (VR) deemed optimal, and in its visualization, for example in the form of a planisphere, with integrated indication of the possible trajectories for insertion of the needle (3) determined automatically or by the practitioner, then a proposal for a possible optimized and secure insertion trajectory of the needle (3), selected automatically or by the practitioner.

[0060] However, the last repetition of steps d) and e), during step f), corresponds to a replanning phase preceding, during the same apnea phase, imminently the start of the insertion phase and taking into account a probable minor modification of the position of the target (C) and of the soft structures (OVT) between two apneas of the subject (S), this step f) comprising the creation and the visualization of a new substantially standardized 3D image of the volume region (RV) of the same type as that obtained at the end of step e), this with a view to a possible automatic or manual adjustment of the positioning and / or the orientation of the needle holder (8) corresponding to the optimized trajectory selected at the end of step e) and at least one validation of this trajectory, possibly finely readjusted, by the practitioner.Since there is no major difference between the two apneas in terms of the position of the structures (OVT) and the target (C), a fine adjustment is sufficient to re-optimize the planned trajectory, without requiring a more complete re-planning.

[0061] In order to facilitate and make faster the choice or validation of a possible optimized insertion trajectory of the needle (3) during step e), it may be advantageously provided that in step b), the trajectories considered reasonable by the practitioner are defined by the latter via a median insertion direction passing through the target (C) and correspond to rectilinear trajectories included in a conical or truncated volume, preferably with a circular or ellipsoidal base, possibly corresponding to the reduced volume (VR), extending by tapering from the skin (P) to the target (C) at least and with a median axis (AML) passing through the target (C), this median axis corresponding to the aforementioned median direction and being able to be materialized by a corresponding positioning of the needle holder (8).Preferably, at least one possible, or even optimized, insertion trajectory is determined after analysis of the content of said first 3D image and taking into account the specific circumstances linked to the subject (S) and the configuration of the intervention, automatically or semi-automatically or by the practitioner.

[0062] In order to be able to carry out with the probe (1) mounted on the robotic arm (2) a rapid and efficient exploration, the substantially standardized 3D images of step c) and step d) advantageously consist of 3D images produced from 2D images acquired along intersecting planes with respect to the reasonable insertion trajectories of step b) and with sufficient and constant quality, namely with an optimized and reproducible trajectory of the 2D ultrasound probe (1), that is to say in particular by applying a substantially constant and as low as possible force at the level of the 2D probe (1) allowing the acquisition of 2D images with the aforementioned quality and by moving the latter with a controlled and optimized acquisition speed, in particular the highest possible speed nevertheless guaranteeing sufficient density and quality of volume information to produce images of acceptable quality.When the movement of the probe (1) is a simple translation, a substantially constant and high speed (with the aforementioned restriction) is advantageously applied during the exploration, advantageously with variations less than a predetermined threshold value, for example 10%, preferably 5%. However, in the case of rotational or tilting movements, or combined movements, it is necessary to slow down if necessary so that the movement of the acquisition plane in the volume at the end of the probe is not too great in relation to the number of 2D images per second that it is possible to acquire.

[0063] In accordance with advantageous embodiment variants of the invention, it may be provided that in step e), the visualization of the tissues or structures (OVT) in the substantially standardized 3D image of the volume region (VR) is carried out by means of a visualization mode among: i) a successive visualization of sectional views corresponding to 2D interpolated images, ii) a 3D visualization obtained by a direct volume rendering method and iii) a visualization 3D obtained by a surface rendering method, after carrying out a segmentation phase of said substantially standardized 3D image of the volume region (VR).

[0064] According to a possible additional feature of the invention, the method may comprise an additional final step h) consisting of simultaneously displaying, and where appropriate superimposed, at least two 3D images of the volume region (RV) of the subject (S) extending from the skin (P) to the target (C) concerned, and comprising the latter, or alternatively of a more restricted volume region containing the location of the target (C), one of which was taken before the intervention and the treatment by means of the needle (3) and the other of which was taken after this intervention. This optional step makes it possible, after completion of the treatment, by merging before / after images or their display in parallel, to verify that the treatment carried out during the intervention does indeed affect, in its entirety, the area of ​​the target (C).

[0065] As shown in [Fig.2], the method may consist, before step a), in positioning the robotic arm (2) carrying the probe (1), relative to the subject (S) lying on a surgical table (9), in such a way that the base (10) of said robotic arm (2) is located on the lateral side of said table (9) which is opposite to that occupied by the practitioner, and where appropriate likely to be subjected to the passage of the 2D ultrasound probe (1). This arrangement makes it possible to avoid any interference between the practitioner and the robotic arm (2) and also to be able to easily apply sufficient pressure during explorations with the probe (1).

[0066] The indication of the possible rectilinear insertion trajectories or of the optimized trajectory of the needle (3) can be carried out by the practitioner, with a view to their display in the 3D images, either virtually by means of the human-machine interface, or preferably by manipulation by the practitioner of the needle holder (8) and validation of the position and orientation of the latter by said practitioner when the desired trajectory is displayed in the 3D image.

[0067] The needle holder (8) is advantageously position-controlled in such a way that it remains permanently oriented so that the direction of insertion of the needle that it would carry passes through the center of the target (C).

[0068] Advantageously, the practitioner can indicate a trajectory by suitable positioning of the needle holder (8) for example (its position being known due to the knowledge of the position of the two robotic arms by the medical installation also comprising the display means), in order to visualize this trajectory during the acquisition of the first 3D image produced in an assisted manner during step b), but also during step d). It is thus possible to provide guidance information during the manual or assisted acquisition of the 2D images and to facilitate the display of only the relevant volume (to avoid displaying irrelevant information that could distract the practitioner).

[0069] Several options are possible for finding a needle insertion trajectory (possible, optimized), which are distinguished by their degree of automation.

[0070] Thus, in accordance with a manual option, it is possible for the practitioner to move the needle holder (8) and interactively visualize a tubular volume (VT) whose median axis (AML) corresponds to the trajectory resulting from the current orientation and positioning of the needle holder (8). In this case, the practitioner moves the needle holder somewhat randomly, and if he notices a possible, or even optimized, trajectory, he stops. This is a user-friendly and efficient approach, since the needle holder always remains directed towards the target.

[0071] A second option consists of offering the practitioner a map of possible insertion options (for example on a planisphere of the volume region). Thus, all the options are taken into account and the practitioner can make the best choice in an informed manner. When his choice of optimized trajectory is made, he directs the needle holder as close as possible to his choice by advantageously visualizing the position of his trajectory in said map (planisphere for example).

[0072] Finally, according to a third option, it may be envisaged to create a 3D model of the volume region (RV) integrating the structures (OVT) and the target (C) from the 3D image, then to calculate the best trajectory and to propose it directly to the practitioner.

[0073] The explorations with the probe (1) of step d) can be carried out entirely automatically at the simple request of the practitioner, thus completely freeing the latter from this task.

[0074] Alternatively, and to allow the practitioner to further control these explorations and subsequent image acquisition, for example to limit their amplitude or center them on a reduced volume of interest which may vary, in particular during insertion, the method may consist of carrying out the explorations with the probe (1), which are subsequent to step c) and initiated at the initiative of the practitioner, in a semi-automatic or collaborative manner, the movement of displacement of the probe (1) and the extent of the exploration being determined by manual action of the practitioner, the actual displacement of the probe (1) being nevertheless constrained by the trajectory parameters determined in step c). The exploration movements of the probe (1) may be carried out either bidirectionally or unidirectionally with initial repositioning of the probe (1) after completion of an exploration.In the latter case, it may be possible to envisage an automatic initial relocation of the probe (1) with respect to the target (C), with a position and an orientation. adequate, by the robotic arm (2) before each new exploratory movement of the latter carried out during the subsequent stages.

[0075] In accordance with an alternative embodiment based on more advanced assistance, making it possible to ensure that the target has not moved or, if not, to be able to estimate its movement, in particular between two apneas, the analysis of the 2D images acquired in step a), with a view to identifying and locating the target (C) to be treated, for example a cancerous tumor, is carried out by or with the assistance of an algorithm, for example an artificial intelligence program having previously undergone adequate training, this location of the target (C) being repeated automatically at each subsequent exploration and display of the 3D image.

[0076] The repetitions referred to in step d) make it possible to ensure the position of the target very frequently. In particular, at least one repetition is necessary after step b), when the practitioner or clinician has planned and positioned the needle holder or needle guide (8) for the first time. Indeed, this phase is relatively long and therefore incompatible with continuous apnea on the part of the subject (S). Repeatedly carrying out step d) makes it possible to correct, as provided for in step f), an expected error in repositioning the target (C) and the organs, tissues and structures (OVT) surrounding the target (C) at the time of the apnea (second, third, etc.) during which the insertion of the needle (3) will actually take place.

[0077] According to another advantageous characteristic of the invention, the movement by the practitioner of the ultrasound probe (1) in steps a) and b), is controlled by constraint and limitation of movement by the robotic arm (2), in terms of orientation and position of the probe (1) relative to the skin (P) of the subject (S) and speed of movement, the practitioner being able to advantageously adjust the pressure setpoint for applying the probe (1) to the skin (P), controlled by the robotic arm (2) during the movement and / or deactivate the assistance of the robotic arm (2), by means of suitable means forming part of the human-machine interface (6, 7). These arrangements make it possible to avoid harmful movements of the probe (1) potentially involving laborious and time-consuming, or even impossible, reconstruction of the 3D images.Indeed, during each exploration, the successively acquired 2D images must, as far as possible, follow one after the other in a regular and continuous manner, and in particular, overlapping of these images and intersections between these images must be avoided. This is in order to produce a sequence of 2D images whose relative position and acquisition conditions facilitate the calculation and consistency of the resulting 3D image. Deactivation of the assistance of the robotic arm may be desired for finer control by the practitioner, with less effort.

[0078] In addition or alternatively to the preceding characteristic, the invention may provide that the pressure of application of the probe (1) on the skin (P) is, during the displacement of the latter at each exploration carried out during phase b) and / or phase d), automatically controlled by the robotic arm (2) by real-time control of the value of this pressure by means of an appropriate sensor or a control loop exploiting the result of an analysis of the quality of the 2D ultrasound image generated in real time, said application pressure being minimized during phases d), e) and f) to a value just allowing the acquisition of 2D images of sufficient quality to produce a standardized 3D image, in order to generate minimal deformation of the tissues visualized.

[0079] The control aims, in particular in the last three aforementioned phases, in particular to optimize the quality of the 2D images produced while limiting as much as possible the pressure exerted on the skin (P) of the patient or subject (S) in order to limit as much as possible the deformations induced at the level of the underlying tissues and structures.

[0080] During the first clinician-assisted exploration in step b), effective pressure control can also be implemented, especially if the robotic arm is in exoskeleton mode, so as not to injure the patient.

[0081] During the phase of calculating an ideal trajectory, from steps d) to f) (planning and replanning phases), there are potentially several passes or explorations, automatically to calibrate the movement so as to determine the lowest possible application pressure, but still allowing quality images to be acquired.

[0082] Finally, during step g) (during which 3D images are acquired regularly), there is a permanent monitoring of the pressure so as to take into account a possible movement of the patient or a possible problem with the robotic arm.

[0083] Advantageously, the production of substantially standardized 3D images at the level of steps c), d) and f) is carried out automatically, on the basis of the 2D images produced by the probe (1) manipulated by the robotic arm (2), at each command or with the participation of the practitioner, by making the probe (1) follow a determined trajectory, in particular optimized in terms of position and orientation of the probe (1) relative to the skin (P) of the subject (S) and speed of movement of the probe (1), during movement, this trajectory having been determined during or after step b).

[0084] In order to enable the acquisition of good quality images, the method according to the invention may further consist, on request or automatically, in spreading by spraying, by means of a suitable device associated with or integrated into the probe (1), ultrasound contact gel in front of and / or behind the probe (1) during one, preferably each exploratory movement. This arrangement advantageously makes it possible to prevent the probe (1) from making the present and necessary gel disappear, pass after pass, thus definitely compromising the quality ultrasound images acquired due to poor acoustic contact, and therefore transmission.

[0085] To mark and understand the potential trajectory of the needle in the patient's body, and in accordance with another advantageous characteristic of the invention, the 3D visualization of the organs, vessels and / or tissues (soft or fragile structures OVT) present between the surface of the skin (P) and the target (C) as provided in step e), comprises the display by overprinting of a collimation or aiming pattern centered on the target (C), making it possible to define and visually simulate the virtual trajectory of insertion of the needle (3) for a given positioning and orientation of the needle holder (8), and advantageously by interaction via the human-machine interface, to explore the possibilities of rectilinear trajectory of insertion of the needle (3) and entry point in the subject (S) and to select one or more proposals. This pattern (for example a cross as in [Fig.8]) is set up and visible when the target is seen in the displayed 3D image (e.g. of a reduced VR volume like that of Figures 7A and 7B) along the axis of the relevant insertion direction.

[0086] The exploration of the possibilities of rectilinear trajectory of insertion of the needle (3) can be carried out by the practitioner by physically manipulating the needle holder (8) carried by the other robotic arm (4), then by recording and requesting the visualization of the trajectory(ies) retained by him in the 3D images (by means of suitable control means of the human-machine interface).

[0087] As a variant, it may be provided to carry out the exploration of the possibilities of rectilinear trajectory of insertion of the needle (3) by simulating or by collaboratively carrying out a constrained movement of the other robotic arm (4) and therefore of the needle holder (8), for example in the form of a constrained ball joint movement, in such a way that the insertion trajectory always passes through the center of the target (C) whatever the position of said arm (4) and said needle holder (8).

[0088] Preferably, the determination of the positioning and orientation of the needle holder (8) with a view to a possible optimal insertion trajectory by using a 3D visualization of the organs, vessels and / or tissues or structures (OVT) present between the skin (P) and the target (C), as provided in step e), comprises the automatic analysis and display of at least one, preferably several, proposal(s) of rectilinear trajectory of insertion of the needle (3) and entry point (PE) in the subject (S), each associated with an orientation or a plurality of orientations contained in a virtual cone whose apex touches the target (C), for example the center thereof, the practitioner being able to select or validate one or one of the proposal(s) via the human-machine interface. This cone is normally included in the truncated volume (VR) delimiting the displayed volume.When multiple cones can be identified, it may be advantageous to . choose the largest cone to determine the safest effective insertion path possible.

[0089] In order to limit the resources required for an augmented display (target, structures, trajectory) useful to the practitioner and to be able to carry out this display with a real-time refresh, the method may consist, after selecting a proposed trajectory for insertion of the needle (3) and entry point (PE) in the subject (S), in visualizing a limited cylindrical tubular volume (VT) with said trajectory as longitudinal median axis (AML) and making it possible to visualize the organs, vessels and / or tissues or structures (OVT) possibly present in this volume. This reduced display may also already be carried out during the exploratory phase of the possible trajectories. Thus, only the structures close to the insertion trajectory and therefore relevant will be displayed in the visualized limited volume. Such a tubular volume may for example have a diameter of between 1.5 and 3 cm, advantageously of the order of 2 cm.

[0090] According to another possible characteristic of the invention, alternative or complementary to the preceding characteristic, step g) may comprise the prior segmentation of the needle (3) in the images acquired successively as the needle (3) is inserted and in that the volume (VR) visualized consists of a volume elongated along the direction of insertion of the needle (3), preferably a tubular volume comprising said target (C) and at least one end portion (3') of the needle, as well as the intermediate volume (VP) located between them, with the organs, vessels and / or tissues or structures (OVT) possibly present in this volume. It is thus possible to know quantitatively the distance to the target remaining to be covered by the needle during its insertion and after the acquisition of each 3D image.In addition, this arrangement also makes it possible to check whether the trajectory is still directed towards the target, the latter possibly moving during insertion. In addition, it will also be possible to visualize in real time the bending of the needle during its insertion and to make a possible slight trajectory correction if necessary. Finally, by limiting the display to such a limited volume and centered on the needle being inserted, possibly coupled with a more limited exploration scope of the probe, it is possible to facilitate visualization in terms of resources and reactivity, while limiting the risks of distraction by displaying irrelevant peripheral information.

[0091] In order to save time during acquisition and visualization, and therefore limit the duration of the patient's apnea, while providing the necessary visual information to the practitioner during insertion, it is very favorably provided that, in step g), the delimitation of the reduced volume (VR) intended to be visualized in 3D is controlled by the insertion movement of the needle (3) and subject to the fraction of trajectory remaining to be traveled at the end portion (3') of the needle (3) to the target (C), this delimitation being carried out either upstream during the acquisition of the 2D images, or downstream during the 3D visualization.

[0092] In the two cases mentioned above, each of the volumes (VT) or (VR) displayed can therefore vary in size during insertion to be limited to a smaller volume comprising only an end portion (3') of the needle (3) and a volume (VP) preceding this end (3') during insertion up to the target (C), as shown for example in [Fig.4].

[0093] The invention also relates, according to a second aspect, to a collaborative method for inserting a needle (3) into a living subject (S), implementing a medical installation (5) comprising, on the one hand, at least one human-machine interface with a display device (6) and command or control means (7) and, on the other hand, at least the robotic arm (2) carrying said 2D ultrasound probe (1), the position and orientation of which are known in real time and can be used and recorded, concomitantly with the 2D images generated at each exploration, by said installation (5), said installation (5) further comprising another or second robotic arm (4) on which a needle holder (8) is fixed, at least the relative position of the two arms (2 and 4) between them being known permanently by said installation (5), said method comprising a preparatory planning phase preceding the actual insertion operation,the latter being carried out automatically by the second robotic arm (4) or by a practitioner with guidance by this second robotic arm (4), the needle holder (8) being prepositioned and preoriented during the preparatory phase.

[0094] According to the invention, this insertion method is characterized in that it consists of implementing the assisted and interactive method for producing and displaying 3D ultrasound images as described above, and in particular the aforementioned steps a) to e) during a preparatory planning phase, step f) during a replanning phase following the preparatory planning phase and immediately before the start of the insertion of the needle (3) and step g) during the actual operation of the insertion of the needle (3), the replanning phase and the operation of inserting the needle (3) taking place during the same apnea phase of the subject (S).

[0095] This collaborative insertion process therefore closely integrates the assisted and interactive process for producing and displaying 3D ultrasound images and is dependent for its various constituent phases on the associated steps of the latter process.

[0096] It should be noted that the performance of the insertion itself can be collaborative (guided by the needle holder held by the second robotic arm and pushed by the practitioner) or fully automatic by means of the second robotic arm alone (4).

[0097] Preferably, and taking into account the fact that the target can move during insertion, it may be envisaged to compensate for any movements and displacements of the target occurring during insertion of the needle (3), by a modification of the orientation of the needle holder (8) by the robotic arm (4) carrying it, carried out automatically or by the practitioner, with a limiting constraint. Another reason for compensation may lie in any bending of the needle during insertion.

[0098] In accordance with an advantageous embodiment of the invention, the replanning phase, carried out at the start of the apnea during which the insertion must be carried out, consists of creating a new standardized 3D image in accordance with step f), checking the adequacy of the positioning of the needle holder (8) and of the resulting trajectory in relation to the situation visualized in this new 3D image and carrying out, if necessary, automatically or not, and at least after validation by the practitioner, a possible adjustment of the orientation of said needle holder (8) necessary to find an optimized insertion trajectory taking into account a possible minor modification of the position of the target (C) and / or of the soft structures (OVT) between two apneas of the subject (S).

[0099] The adjustment will consist at most of slightly modifying the orientation of the needle holder (8) constrained by a passage of the trajectory through the entry point (PE) selected previously.

[0100] Finally, the invention also relates, according to a third aspect, and as is apparent for example from Figures 1 and 2, to a medical installation (5) for implementing the needle insertion method (3) as mentioned above, comprising, on the one hand, at least one human-machine interface with a display device (6) and command or control means (7) and, on the other hand, at least the robotic arm (2) carrying said 2D ultrasound probe (1), the position and orientation of which are known in real time and can be used and recorded, concomitantly with the 2D images generated at each exploration, by said installation (5).

[0101] Of course, the 2D ultrasound probe (1) is connected to an image processing device allowing the production of 3D images and forming part of the medical installation.

[0102] This installation is characterized in that it also comprises, on the one hand, another or second robotic arm (4) on which a needle holder (8) is fixed, at least the relative position of the two arms (2 and 4) between them being known permanently by said installation (5), and, on the other hand, means for the controlled acquisition, processing and exploitation of images produced by the 2D ultrasound probe (1) when the latter is moved during an exploration on a living subject (S) by the robotic arm (2) concerned, or by a practitioner with the assistance of said robotic arm (2), these means allowing the implementation of the assisted and interactive process for producing and displaying 3D ultrasound images described above.

[0103] Knowledge of the relative position of the two robotic arms (2 and 4) by the installation may result from their manufacture (mounted on the same base), from their possible mutual assembly after manufacture (removable mechanical connection between the two arms) or from their installation in situ. This intrinsic knowledge makes it possible in particular to avoid the use of a camera, with the resulting disadvantages, and from any recalibration operation. The two robotic arms (2 and 4) are preferably six-axis robots and together with their control and piloting means constitute a robotic device or robot cooperating interactively with the ultrasound device and the human-machine interface, where appropriate under the control of a centralized control unit of the installation.

[0104] Advantageously and as shown in Figures 1 and 2, the two robotic arms (2 and 4) are mounted on the same single base (10), which advantageously integrates a control unit for these two arms (2 and 4), said base (10) being preferably equipped with rolling means and associated with a control means (7) for each arm, for example a pedal, forming part of the human-machine interface.

[0105] The means for the controlled acquisition, processing and exploitation of images produced by the 2D ultrasound probe (1), in communicative connection with the display means (11), can for example be housed in this base (10).

[0106] Preferably, the human-machine interface comprises a control console (11), possibly mounted on rolling means, comprising computer means for calculating and processing data, in particular image processing algorithms and software, and the display device (6), advantageously in the form of a touch-sensitive display device, said control console (11) interacting with the two robotic arms (2 and 4).

[0107] With the following description, a particular embodiment of the invention is explained in more detail, incorporating the two methods and the installation, as well as their possible variants or possible options.

[0108] Firstly, a robotic device is provided comprising two robotic arms (2 and 4) mounted side by side, for example on the same base, one of which manipulates a 2D ultrasound probe (1) and the other of which manipulates a needle holder (8). This device is associated with a human-machine interface comprising a control console (11) with interactive screen (6) and control means (7).

[0109] The first arm (2) manipulates, autonomously or semi-autonomously, the 2D ultrasound probe (1) in order to scan a large area including the insertion region. of the needle (3) and the surrounding structures (OVT). This scanning will thus make it possible to create 3D images on demand in near real time.

[0110] The very first time, the scan will be carried out manually by the user (practitioner / clinician) who will co-manipulate the probe (1) under the assistance of the robotic arm. This makes it possible to facilitate but also to slightly constrain the practitioner's gesture to get closer to an ideal trajectory.

[0111] From this first recording, a trajectory will be calculated in order to make the movement of the ultrasound probe manipulated by the robot optimal in terms of speed, pressure applied and the number of 2D images acquired, making it possible to obtain a high-quality 3D image autonomously.

[0112] The position of the target (C) is detected automatically and in real time in the successively acquired 3D ultrasound images.

[0113] The structures to be preserved (OVT) are made visible in these 3D images so as to allow the clinician to visualize all of the secure insertion options available and thus enable him to make a truly informed choice.

[0114] The speed of image acquisition and the planning rectification tools make it possible to make the inevitable adjustments necessary to take into account the potential and almost inevitable modification of the position of the target (C) between the planning phase and the execution phase of the gesture (insertion). Indeed, the planning and execution phases are necessarily carried out from two different images since they are associated with two different apneas.

[0115] To do this, the second robotic arm (4) manipulates a needle guide or needle holder (8) which is always oriented towards the target (C). This second arm is able, after validation by the clinician, to slightly correct its orientation to compensate for a change in position of the target (C) between the planning phase and the execution phase.

[0116] Throughout the manual insertion of the needle carried out by the practitioner and guided through the needle holder (8), thanks to the automatic or semi-automatic, rapid and regular creation and visualization of 3D images, it is possible to visualize both the trajectory of the needle (3), the target (C) and the surrounding structures (OVT), making it possible to guarantee the safety and precision of the gesture.

[0117] The comparison made after processing between an image acquired before and a second image acquired after processing makes it easier to evaluate the quality and extent of processing.

[0118] The 2D images which make it possible to create the first 3D image are acquired manually using the robot's capacity to constrain the movements of the clinician's arm but also to assist him because:

[0119] - This allows the practitioner's skill to be used to acquire a volume ideally well placed including the possible trajectories of the needle (3) and adjacent structures (OVT) without requiring robotic intelligence which is otherwise difficult to reconcile with regulatory obligations.

[0120] - This helps to assist the practitioner in order to make the execution of his gesture easier. thanks to the assistance of the robot which will be able to apply sufficient but minimal force in order to limit tissue deformation as much as possible.

[0121] - This makes it possible to constrain the movement of the probe so as to acquire a set of coherent 2D images, associated with their positions, while avoiding acquiring overlapping images making the creation of the resulting 3D image difficult or even impossible.

[0122] - This makes it possible to record a trajectory of the robotic arm (2) very close to the optimized final trajectory thus avoiding the risks of collision between said robotic arm (2) and the patient (S).

[0123] An optimal trajectory is obtained by calculation from the recorded trajectory acquired by the practitioner because:

[0124] - This allows minimal force to be applied to the tissues with the probe (1) while ensuring optimal image quality.

[0125] - This allows the speed of the probe (1) to be adapted to minimize the duration of acquisition while acquiring a sufficient number of 2D images to ensure the quality of the resulting 3D image.

[0126] - This subsequently allows the robot to acquire autonomously or in a semi-autonomous (under the impulse of the practitioner's hand) continuous images in order to be able to follow the movement of the needle (3) inside the patient (S) during insertion.

[0127] Automatic detection of the target (C) using for example artificial intelligence is implemented, but after an initial manual validation because:

[0128] - A first manual validation of the target (C) will make the detection more reliable.

[0129] - This will allow the ultrasound probe (1) to recognize the tumor (target) of autonomously and thus ensure in quasi-real time its position after each passage of the probe (1) carried by the robotic arm (2).

[0130] In the context of the search for the best possible insertion trajectory, the robotic arm which holds the needle holder (8) is constrained, so that it always remains directed towards the target (C) because:

[0131] - This is a significant time saver and does not require tedious adjustment, while being easy since the information of the target's position is available.

[0132] - This allows a large number of positions to be explored in a simplified manner and friendly.

[0133] For each position of the needle holder (8) and in real time, it is proposed to limit the visualization of the 3D image to a cylinder (VT, VR) located around the trajectory because:

[0134] - This facilitates visualization of the target (C) and soft / fragile structures surrounding (OVT), in particular vascular, thanks to the creation of a 3D negatoscope limited to this volume or through the use of volume rendering.

[0135] - This ensures that on or in a reasonably sized neighborhood around of the needle trajectory (3), no fragile structure to be preserved is present.

[0136] A map of all the potential options for the passage of the needle is proposed by juxtaposing the projection of the different volumetric representations on a planisphere because:

[0137] - The manual movement of the needle holder (8) constrained by the robotic arm always directed towards the target (C) allows to obtain a visualization of the cylinder in projection in order to understand if fragile structures (OVT) are found in this volume. It is planned to precalculate these projections for all possible reasonable trajectories of the needle and to display the result on a map like a terrestrial planisphere on which land and sea are represented.

[0138] - In this way, it is possible to choose the best trajectory, that is to say the one which allows needle insertion (3) while being as far as possible from fragile structures (OVT) and having a priori knowledge (the practitioner) of all possible trajectories, thus avoiding wasting time in often tedious trial and error.

[0139] It is possible to calculate the position of the target, the neighboring organs and their vascularizations using an Artificial Intelligence that has undergone suitable training or similar software, from a 3D intraoperative ultrasound image. This makes it possible to use this 3D modeling to automatically calculate the optimal trajectory for reaching the target.

[0140] It is also possible, if necessary, to calculate the non-rigid registration of a preoperative 3D scanner image with the intraoperative 3D ultrasound image. This makes it possible to use 3D modeling of this preoperative image to automatically calculate the optimal trajectory for reaching the target.

[0141] Once the planning has been carried out from a 3D ultrasound image that has become obsolete and the needle holder (8) has been positioned, a new 3D image is acquired, automatically or semi-automatically, at the start of the insertion phase. This acquisition makes it possible to detect the new position of the tumor (target) and to automatically carry out the very slight, but systematically necessary, modification of the orientation of the needle holder (8) so as to always be perfectly well directed towards the new position of the target (C) because:

[0142] - This allows to take into account the inevitable modification of the target (C) between the planning phase and the insertion phase associated with images acquired during separate apneas since acquired at two different times.

[0143] - In this way, the modification of orientation of the needle holder (8) is restricted to a rotational movement around the entry point (PE) and is extremely limited, avoiding any risk of contact between the robotic arm (4) and the patient (S).

[0144] - This allows the practitioner to re-evaluate the cylinder (VT, VR) around the new trajectory. And therefore to confirm that the new trajectory is indeed valid.

[0145] The practitioner will insert the needle manually while being able to visualize the tissues and structures located on the trajectory of the needle (3) and situated between the tip of the needle (3') and the target (C) thanks to the 3D images acquired automatically or semi-automatically by the robotic arm (2) with the probe (1) because:

[0146] - This allows the position of the target (C) to be automatically detected and therefore ensure that the needle direction is still correct.

[0147] - This allows the practitioner, if the alignment is no longer perfect, to slightly modify the curvature of the needle to change its trajectory.

[0148] - This allows the progression of the needle (3) to be visualized in near real-time in order to actually position the tip of the needle (3') in the center of the target (C). This is possible even if the target moves slightly under the effect of the deformations induced by the needle and therefore even if its position changes in terms of depth.

[0149] It is possibly possible to automatically extract the position of the needle (3) in the 3D ultrasound images using Artificial Intelligence or suitable similar software, in order to automatically calculate its progression in near real-time in the patient (S) so as not to rely solely on visual feedback.

[0150] It is planned to systematically and automatically deposit ultrasound gel on either side of the ultrasound probe (2) because:

[0151] - This will allow correct image quality to be maintained even when the probe (1) will have made several trips and would have cleaned the skin (8) of the patient (S).

[0152] - This avoids irritating the skin (8) of the patient (S) due to friction from the probe (1).

[0153] Once the needle treatment is complete, a new 3D image is acquired automatically or semi-automatically and is compared to a 3D image acquired before treatment to facilitate the evaluation of the treated area.

[0154] Indeed, comparing two similar images allows us to better identify their differences, in particular those induced by the processing.

[0155] It may be envisaged to carry out non-rigid registration of the ultrasound images acquired before and after the treatment in order to be able to compare them more easily and by therefore better understand their differences, particularly those resulting from treatment.

[0156] A detailed description of a possible protocol for implementing the invention integrating the two aforementioned methods and comprising nine successive phases is described below.

[0157] Phase 1: Positioning and preparation of the patient

[0158] In this first phase, the patient (S) is positioned and prepared for the procedure. This includes preparing the area to be treated and positioning the patient in such a way as to ensure optimal access for the practitioner.

[0159] Phase 2: Positioning of the robotic device or robot

[0160] The robot comprising the two robotic arms (2 and 4) is positioned on the other side of the table (9) relative to the practitioner so as not to limit his movements, ensuring optimal coordination throughout the procedure.

[0161] Phase 3: Search for the target using the ultrasound probe (1)

[0162] The practitioner manipulates the 2D transabdominal ultrasound probe to locate the target (C) to be treated. An AI algorithm assists him by automatically detecting the target tumor, which accelerates and improves the accuracy of the localization process.

[0163] The ultrasound probe (1) is associated with the first arm (2) of the robot. The robot assists the practitioner so that the movement of the probe (1) can be carried out without effort, limiting its parasitic movements while applying limited but sufficient pressure on the skin (8) of the patient (S).

[0164] Phase 4: Creation of a 3D ultrasound image of the needle insertion area

[0165] After locating the target (C), the practitioner scans using the probe ultrasound (1) an area (RV volume region) that encompasses the reasonable potential trajectories of the needle. Scanning this area automatically creates a 3D image including the target (C), the possible trajectories as well as the surrounding structures (OVT).

[0166] Once this first acquisition is completed, the robot is capable of performing similar acquisitions autonomously or semi-autonomously. At any time, the practitioner can now view a 3D image updated in near real time.

[0167] This first scan takes place during a first apnea of ​​the patient (S).

[0168] Phase 5: Planning the insertion trajectory

[0169] The practitioner chooses an optimal trajectory for the needle (3) by consulting a map that represents all possible trajectories. The map is created by taking into account the presence of fragile structures (OVT) along their paths.

[0170] This map is presented concretely on the touch display interface (6) of the installation in the form of a planisphere comprising risk-free zones and risk zones. This allows access to all possible options.

[0171] Phase 6: Positioning the needle holder

[0172] The practitioner positions the needle holder in the optimal position chosen using the trajectory options map. The needle holder being associated with the second robotic arm (4), when the practitioner moves the needle holder, the latter always remains oriented towards the target (C). In addition, the position of the needle holder (8) and the resulting trajectory are naturally reported on the real-time map (for example, planisphere) corresponding to all the options to facilitate this positioning.

[0173] Phase 7: Insertion of the treatment needle

[0174] Since the insertion of the needle (3) takes place during a second apnea or an apnea following the apneas that led to the determination of the optimal trajectory, the position of the target (C) cannot be guaranteed since it comes from the image acquired during the first apnea or previous apneas. The practitioner will therefore use a new image acquired automatically just after this second apnea or subsequent apnea.

[0175] The new position of the target (C) is then detected automatically and the needle holder (8) will automatically make the slight correction to compensate for the error in repositioning the target between the two apneas.

[0176] After validation of the trajectory by the practitioner, in particular by visualizing an area corresponding to a cylinder (VT, VR) located along the trajectory of the needle, the latter can begin to insert the needle.

[0177] The clinician will insert the needle (3) while regularly asking the installation (5) comprising the robotic arm (2) with the probe (1) to update the visualization of the 3D image limited to a cylinder located around the trajectory of the needle for near real-time control of the precision of the procedure.

[0178] Phase 8: Target treatment by radiofrequency

[0179] Once the needle is correctly positioned, the practitioner can perform the radiofrequency treatment (or other operation involving the needle).

[0180] Phase 9: Evaluation of the extent of treatment

[0181] Once the treatment has been carried out, the practitioner asks the installation (5) to acquire a new 3D image to evaluate the extent of the treatment. This makes it possible to verify the effectiveness of the treatment by comparing two images, the first having been acquired before and the second after the treatment.

[0182] The aforementioned protocol ensures optimal precision, efficiency and safety throughout the insertion and treatment process.

[0183] Of course, the invention is not limited to the embodiment described and shown in the attached drawings. Modifications remain possible, in particular from the point of view of the constitution of the various elements or by substitution of technical equivalents, without departing from the scope of protection of the invention.

Claims

1. Claims Assisted and interactive method for producing and displaying 3D ultrasound images, by controlled acquisition and processing of 2D images acquired by means of a 2D ultrasound probe (1) which can be moved on a living subject (S) by a robotic arm (2), or by a practitioner with the assistance of a robotic arm (2), said method implementing a medical installation (5) comprising, on the one hand, at least one human-machine interface with a display device (6) and command or control means (7) and, on the other hand, at least the robotic arm (2) carrying said 2D ultrasound probe (1), the position and orientation of which are known in real time and can be used and recorded, concomitantly with the 2D images generated at each exploration, by said installation (5), method characterized in that it comprises at least the steps consisting of: a) to locate one or the target (C) to be treated in the subject (S) by exploratory movement of the probe (1) on the skin (P) of the latter and analysis of the acquired 2D ultrasound images, by a practitioner possibly assisted by suitable detection software; b) producing a first 3D image of a volume region (RV) of the subject (S), extending from the skin (P) to the target (C) concerned and containing the latter and the reasonable possible introduction trajectories of a needle (3) to reach said target (C) estimated by the practitioner, this 3D image being obtained from a set of 2D images acquired, during an apnea phase of the subject (S), by movement of the 2D ultrasound probe (1) by the practitioner with the assistance of the robotic arm (2) and potentially by exploitation of the location information from step a); c) automatically determining, during or after step b), the parameters and the path of an optimized and reproducible trajectory for the 2D ultrasound probe (1) allowing the production of substantially standardized and identically framed 3D images of the aforementioned volume region (VR); d) to produce one, or successively several, substantially standardized 3D image(s) of this volume region (VR), in quasi-real time and automatically or at the command of the practitioner,

2. during an apnea phase of the subject (S) and on the basis of 2D images freshly acquired following a new exploration with the 2D ultrasound probe (1) carried out by applying the parameters and following the route which were determined during step c); e) visualizing, in this or these substantially standardized 3D images of the volume region (RV) of the subject (S), the organs, vessels and / or similar soft tissues or structures (OVT) present between the skin (P) and said target (C), as well as the latter two, in order to enable the positioning and orientation of a needle holder (8) to be determined for at least one possible optimized insertion trajectory of the needle (3); f) repeating steps d) and e) just before the start of the insertion of the needle (3), which is then positioned in accordance with the optimized insertion trajectory selected by the practitioner and visualized in the current 3D image, and automatically determining and visualizing the position of the target (C) at this instant with a view to a possible adjustment of the positioning and / or orientation of the needle holder (8), in particular just before the actual insertion of the needle; g) then visualizing in quasi-real time the progressive insertion of the needle (3) by repeatedly creating and visualizing refreshed 3D images of a volume (VR), for example substantially cylindrical or truncated, as a fraction of the aforementioned volume region (RV) of the subject (S), preferably as a variable fraction of this volume region, the 3D images of this volume (VR) showing either said target (C) and at least one end portion (3') of the needle (3), preferably the entire part of the needle (3) inserted into the subject (S), or at least one end portion (3') of the needle (3) and a volume (VP) preceding this end (3') during the insertion, this visualization of refreshed 3D images of said volume (VR) preferably continuing at least until the target (C) is reached by the needle (3), the two steps f) and g) being carried out together during the same phase apnea of ​​the subject (S). Method according to claim 1, characterized in that steps d) and e) are associated with a preparatory phase of planning the intervention resulting, where appropriate by successive repetitions, in obtaining a substantially standardized 3D image of the region volumetric (RV) deemed optimal, and its visualization, for example in the form of a planisphere, with integrated indication of the possible trajectories for insertion of the needle (3) determined automatically or by the practitioner, then a proposal for a possible optimized and secure insertion trajectory of the needle (3), selected automatically or by the practitioner.

3. Method according to claim 1 or 2, characterized in that step f) corresponds to a replanning phase preceding, during the same apnea phase, imminently the start of the insertion phase and taking into account a probable minor modification of the position of the target (C) and of the soft structures (OVT) between two apneas of the subject (S), this step f) comprising the creation and the visualization of a new substantially standardized 3D image of the volume region (RV) of the same type as that obtained at the end of step e), this with a view to a possible automatic or manual adjustment of the positioning and / or the orientation of the needle holder (8) corresponding to the optimized trajectory selected at the end of step e) and at least one validation of this trajectory, possibly finely readjusted, by the practitioner.

4. Method according to any one of claims 1 to 3, characterized in that, in step b), the trajectories considered reasonable by the practitioner are defined by the latter by means of a median insertion direction passing through the target (C) and correspond to rectilinear trajectories included in a conical or truncated volume, preferably with a circular or ellipsoidal base, possibly corresponding to the reduced volume (VR), extending by tapering from the skin (P) to the target (C) at least and with a median axis (AML) passing through the target (C), this median axis corresponding to the aforementioned median direction and being able to be materialized by a corresponding positioning of the needle holder (8), and in that at least one possible, or even optimized, insertion trajectory,is determined after analysis of the content of said first 3D image and taking into account the specific circumstances linked to the subject (S) and the configuration of the intervention, automatically or semi-automatically or by the practitioner.

5. Method according to any one of claims 1 to 4, characterized in that the substantially standardized 3D images of step c) and step d) consist of 3D images produced from images 2D images acquired along intersecting planes relative to the reasonable insertion trajectories of step b) and with sufficient and constant quality, namely with an optimized and reproducible trajectory of the 2D ultrasound probe (1), that is to say in particular by applying a substantially constant and as low as possible force at the level of the 2D probe (1) allowing the acquisition of 2D images with the aforementioned quality and by moving the latter with a controlled and optimized acquisition speed, in particular the highest possible speed nevertheless guaranteeing sufficient density and quality of volumetric information to produce images of acceptable quality.

6. Method according to any one of claims 1 to 5, characterized in that, in step e), the visualization of the tissues or structures (OVT) in the substantially standardized 3D image of the volume region (VR) is carried out by means of a visualization mode among: i) a successive visualization of sectional views corresponding to 2D interpolated images, ii) a 3D visualization obtained by a direct volume rendering method and iii) a 3D visualization obtained by a surface rendering method, after carrying out a segmentation phase of said substantially standardized 3D image of the volume region (VR).

7. Method according to any one of claims 1 to 6, characterized in that it comprises an additional final step h) consisting in displaying simultaneously, and where appropriate in a superimposed manner, at least two 3D images of the volume region (RV) of the subject (S) extending from the skin (P) to the target (C) concerned, and comprising the latter, or alternatively of a more restricted volume region containing the location of the target (C), one of which was taken before the intervention and the treatment by means of the needle (3) and the other of which was taken after this intervention.

8. Method according to any one of claims 1 to 7, characterized in that it consists of carrying out the explorations with the probe (1), which are subsequent to step c) and initiated at the initiative of the practitioner, in a semi-automatic or collaborative manner, the movement of displacement of the probe (1) and the extent of the exploration being determined by manual action of the practitioner, the effective displacement of the probe (1) being nevertheless constrained by the trajectory parameters determined in step c), the displacements exploration that can be carried out either bidirectionally or unidirectionally with initial repositioning of the probe (1) after completion of an exploration.

9. Method according to any one of claims 1 to 8, characterized in that the analysis of the 2D images acquired in step a), with a view to identifying and locating the target (C) to be treated, for example a cancerous tumor, is carried out by or with the assistance of an algorithm, for example an artificial intelligence program having previously undergone adequate training, this location of the target (C) being repeated automatically at each subsequent exploration and display of the 3D image.

10. Method according to any one of claims 1 to 9, characterized in that the movement by the practitioner of the ultrasound probe (1) in steps a) and b), is controlled by constraint and limitation of movement by the robotic arm (2), in terms of orientation and position of the probe (1) relative to the skin (P) of the subject (S) and speed of movement, the practitioner being able to advantageously adjust the pressure setpoint for applying the probe (1) to the skin (P), controlled by the robotic arm (2) during the movement and / or deactivate the assistance of the robotic arm (2), by means of suitable means forming part of the human-machine interface (6, 7).

11. Method according to any one of claims 1 to 10, characterized in that the pressure of application of the probe (1) on the skin (P) is, during the movement of the latter at each exploration carried out during phase b) and / or phase d), automatically controlled by the robotic arm (2) by real-time control of the value of this pressure by means of an appropriate sensor or a control loop exploiting the result of an analysis of the quality of the 2D ultrasound image generated in real time, said application pressure being minimized during phases d), e) and f) to a value just allowing the acquisition of 2D images of sufficient quality to produce a standardized 3D image, in order to generate minimal deformation of the tissues visualized.

12. Method according to any one of claims 1 to 11, characterized in that the production of substantially standardized 3D images at steps c), d) and f) is carried out automatically, on the basis of the 2D images produced by the probe (1) manipulated by the robotic arm (2), at each command or with the participation of the practitioner, by making the probe (1) follow a determined trajectory, in particular optimized in terms of position and orientation of the probe (1) relative to the skin (P) of the subject (S) and speed of movement of the probe (1), during movement, this trajectory having been determined during or after step b).

13. Method according to any one of claims 1 to 12, characterized in that it consists, on demand or automatically, in spreading by spraying, by means of a suitable device associated with or integrated into the probe (1), ultrasound contact gel in front of and / or behind the probe (1) during one, preferably each exploratory movement.

14. Method according to any one of claims 1 to 13, characterized in that the 3D visualization of the organs, vessels and / or tissues present between the surface of the skin (P) and the target (C) as provided in step e), comprises the display by overprinting of a collimation or aiming pattern centered on the target (C), making it possible to define and visually simulate the virtual trajectory of insertion of the needle (3) for a given positioning and orientation of the needle holder (8), and advantageously by interaction via the human-machine interface, to explore the possibilities of rectilinear trajectory of insertion of the needle (3) and entry point in the subject (S) and to select one or more proposals.

15. Method according to claim 14, characterized in that it consists of carrying out the exploration of the possibilities of rectilinear trajectory of insertion of the needle (3) by simulating or by carrying out collaboratively a constrained displacement of the other robotic arm (4) and therefore of the needle holder (8), for example in the form of a constrained movement of the ball joint, in such a way that the insertion trajectory always passes through the center of the target (C) whatever the position of said arm (4) and said needle holder (8).

16. Method according to any one of claims 1 to 15, characterized in that the determination of the positioning and orientation of a needle holder (8) with a view to a possible optimal insertion trajectory by using a 3D visualization of the organs, vessels and / or tissues or structures (OVT) present between the skin (P) and the target (C), as provided in step e), comprises the automatic analysis and display of at least one, preferably several, rectilinear insertion trajectory proposals of the needle (3) and entry point (PE) in the subject (S), each associated with an orientation or a plurality of orientations contained in a virtual cone whose summit touches the target (C), for example the center of the latter, the practitioner being able to select or validate the or one of the proposition(s) via the human-machine interface.

17. Method according to any one of claims 14 to 16, characterized in that it consists, after selecting a proposed trajectory for insertion of the needle (3) and entry point (PE) in the subject (S), in visualizing a limited cylindrical tubular volume (VT) with said trajectory as longitudinal median axis (AML) and making it possible to visualize the organs, vessels and / or tissues or structures (OVT) possibly present in this volume.

18. Method according to any one of claims 1 to 17, characterized in that step g) comprises the prior segmentation of the needle (3) in the images acquired successively as the needle (3) is inserted and in that the volume (VR) visualized consists of a volume elongated along the direction of insertion of the needle (3), preferably a tubular volume comprising said target (C) and at least one end portion (3') of the needle, as well as the intermediate volume (VP) located between them, with the organs, vessels and / or tissues or structures (OVT) possibly present in this volume.

19. Method according to any one of claims 1 to 18, characterized in that, in step g), the delimitation of the reduced volume (VR) intended to be viewed in 3D is controlled by the insertion movement of the needle (3) and subject to the fraction of the trajectory remaining to be covered at the end portion (3') of the needle (3) up to the target (C), this delimitation being carried out either upstream during the acquisition of the 2D images, or downstream during the 3D viewing.

20. Medical installation (5) for implementing a needle insertion method (3), said installation (5) comprising, on the one hand, at least one human-machine interface with a display device (6) and command or control means (7) and, on the other hand, at least the robotic arm (2) carrying said 2D ultrasound probe (1), the position and orientation of which are known in real time and can be used and recorded, concomitantly with the 2D images generated at each exploration, by said installation (5), installation (5) characterized in that it also comprises, on the one hand, another or second robotic arm (4) on which a needle holder (8) is fixed, at least the relative position of the two arms (2 and 4) between them being known permanently by said installation (5), and, on the other hand, means for the controlled acquisition, processing and exploitation of images produced by the 2D ultrasound probe (1) when the latter is moved during an exploration on a living subject (S) by the robotic arm (2) concerned, or by a practitioner with the assistance of said robotic arm (2), these means allowing the implementation of the assisted and interactive method of production and display of 3D ultrasound images according to any one of claims 1 to 19, within the framework of the method of the needle insertion method (3) mentioned above,which includes a preparatory planning phase preceding the actual insertion operation, the latter being carried out automatically by the second robotic arm (4) or by a practitioner with guidance by this second robotic arm (4), the needle holder (8) being prepositioned and preoriented during the preparatory phase.

21. Medical installation according to claim 20, characterized in that the two robotic arms (2 and 4) are mounted on the same and single base (10), which advantageously integrates a control unit for these two arms (2 and 4), said base (10) being preferably equipped with rolling means and associated with a control means (7) for each arm, for example a pedal, forming part of the human-machine interface.

22. Medical installation according to claim 20 or 21, characterized in that the human-machine interface comprises a control console (11), possibly mounted on rolling means, comprising computer means for calculating and processing data, in particular image processing algorithms and software, and the display device (6), advantageously in the form of a touch display device, said control console (11) interacting with the two robotic arms (2 and 4).

Images