System for computer-assisted navigation surgery

Abstract

The present invention relates to a method and system for computer-assisted navigation in orthopaedic surgery, including converting an action planned in a virtual environment relative to a virtual reference R P into a physical action performed using a surgical tool (20) on a patient in a real operating room environment for orthopaedic surgery.

Description

Technical Field

[0001] The present invention relates to computer-assisted surgery. This field includes a set of computer and / or physical systems designed to assist surgeons in planning and / or performing surgical interventions. It includes the following three series of systems: navigation systems, robotic systems, and augmented and / or mixed reality guidance systems.

[0002] The main elements of current computer-assisted surgery systems are as follows.

[0003] First, a digital model of the patient is created in the following ways: either from patient-specific preoperative data obtained from medical imaging (CT scans, MRI, X-rays, etc.), or by combining a general database (such as an anatomical atlas, a statistical shape model) and data generated during surgery through computer processing of intraoperative images (fluoroscopy, ultrasound, etc.), or using patient anatomical data obtained by a three-dimensional locator and a palpation system.

[0004] Second, a surgical plan is generated from the computer processing of the digital patient model. If the patient model is derived from preoperative data, this task is performed before the surgery, or if the patient model is derived from intraoperative data, this task is performed at the beginning of the surgery.

[0005] Third, surgical navigation based on a method of matching preoperative data with intraoperative data. Surgical navigation allows the surgeon to visualize the current positioning of the surgical instruments relative to the patient's anatomy and to track the progress of the surgery according to the surgical plan. In a standard navigation system, this information is displayed on a screen. In a mixed reality navigation system, surgical navigation is displayed in an augmented or virtual reality manner through a helmet or goggles.

[0006] Fourth, a system for assisting in achieving the surgical posture: capable of generating patient-specific instruments (such as customized cutting or milling guides) from the preoperative plan. Alternatively, providing visual decision support information to the surgeon. If an electromechanical system assists the surgeon in performing the surgical procedure, it is a robotic system.

[0007] The present invention belongs to the field of computer-assisted surgery for robotic system navigation, and single-compartment and knee arthroplasty are the main but not the only applications.

[0008] Computer-assisted robotic joint replacement uses active or collaborative systems (synergistic systems that combine the skills of the surgeon and the robot to form a performance-enhancing partnership) and passive navigation systems. Background Art

[0009] Generally, two main solutions are adopted in the prior art.

[0010] The first solution involves using markers placed on the bone and surgical instruments, in combination with an optical tracking system, to ensure the matching of real and digital reference points and to determine the exact position of the bone and surgical instruments in space. An example of this solution is presented in US Patent US10,136,950 B2, which uses surgical markers to match the patient's anatomy configured to be tracked by a navigation system. To utilize surgical navigation in a surgical procedure to accomplish accurate planning, tracking, and navigation of surgical instruments, tools, and / or medical devices, a surgeon typically couples a "tracking array" to the surgical marker. These tracking arrays allow the surgeon to track the physical positions of these surgical components as well as the patient's bone during the surgery. By determining the physical position of the tracking array, the software associated with the tracking system can accurately calculate the position of the tracked components relative to the surgical planning image. However, using markers fixed to the bone poses a risk of fracture at the anchoring point and prolongs the surgical time. Additionally, the visibility (line of sight) of the markers must be ensured throughout the surgery, so these solutions are not robust to occlusion and limit the positioning of the surgical personnel relative to the surgical area. They increase the number of steps and tools (markers, probes) and require mechanical palpation of the anatomy to match the digital model. These limitations increase the surgical time due to the cumbersome work and require the surgeon and surgical personnel to learn how to place the markers and how to operate. Finally, the associated optical tracking system is both bulky and expensive.

[0011] In addition, there is no pure optical tracking that allows for a real-time control loop. To achieve this level of performance, mechanical tracking is required.

[0012] The second solution involves intraoperative visualization solutions using ionizing radiation, such as fluoroscopy. During orthopedic and trauma surgeries, a fluoroscopy-based navigation system allows for real-time tracking of surgical instruments and superimposes their contours onto fluoroscopy images. For fluoroscopic navigation, the instruments used during the surgical intervention are equipped with markers coupled to a positioning system.

[0013] However, the second solution also has the same drawbacks as the first solution: lack of robustness to occlusion, limitation of the positioning of the surgical personnel relative to the surgical area, increase in the number of required instruments, and the addition of bulky equipment (C-arm device) to the operating room. This second solution also implies the use of ionizing radiation, which is harmful to both the patient and the medical staff operating in the vicinity. Summary of the Invention

[0014] The present invention is a method that allows for the alignment of at least one planned surgical action defined in a virtual environment with a physical surgical action in a real environment, more specifically in an operating room for orthopedic surgery.

[0015] Embodiments of the present invention relate to the situation where first, a trajectory, position, displacement, or other movement is determined through digital simulation in a virtual reference (such as surgical planning simulation). To be usable in a real environment, it is necessary to convert this trajectory, position, displacement, or other movement into a real reference of the patient and apply these movements thereto. For this purpose, a plurality of transformations must be determined to match the virtual reference with the real reference of the patient.

[0016] Preferably, the present invention relates to the automatic control of a bracket for a bone processing tool to conform to surgical planning.

[0017] Another example of the application of the present invention relates to augmented reality for controlling the position of an image superimposed on the field of view area of an operator (such as a surgeon) through augmented vision goggles or a projector.

[0018] The present invention relates to a system for computer-assisted surgery, which includes converting an action planned in a virtual environment relative to a virtual reference R P into a physical action performed in a real operating room environment for orthopedic surgery of a patient using a surgical tool, the surgical tool being fixed to a kinematic chain that includes a sensor unit having at least one sensor configured to follow in real time the spatial configuration of the kinematic chain; the system includes:

[0019] - A receiving module configured to receive at least one 3D image acquired from at least one 3D imaging sensor; the 3D image includes at least a part of the target anatomical structure of the patient;

[0020] - A calculation module configured to:

[0021] ο Calculate a virtual reference R by registering a digital model of the target with at least a part of the target included in the 3D image P with a target reference R C the transformation between C T P and the acquisition reference of the 3D imaging sensor R A and the transformation between the target reference R C ; C T A ;

[0022] ο Apply the transformation C T P so as to register the digital model of the target in the target reference R C such that each point included in the digital model of the target has a known localization in the target reference R C ;

[0023] ο Calculate the reference R of the surgical tool O and the target reference RC Transformation between C T O ;

[0024] ο Apply the transformation C T O to the reference R of the surgical tool O , so as to determine the position and spatial orientation of the surgical tool in the target reference R C ;

[0025] Thereby determining the position and spatial orientation of the surgical tool in the virtual reference R P and the target reference R C , so as to reproduce in the target reference R C the actions planned in the virtual reference R P .

[0026] According to one embodiment, the calculation module is further configured to calculate the transformation C T A :

[0027] - Define a region of interest in the 3D image including the target;

[0028] - Register the region of interest including the target to the digital model of the target to determine C T A .

[0029] According to one embodiment, defining the region of interest includes automatically detecting the region of interest by means of a segmentation algorithm.

[0030] According to one embodiment, the kinematic chain includes at least one mechanical reference rigidly fixed to the target anatomical structure, and at least one 3D image includes at least a part of the mechanical reference; the calculation module is further configured to:

[0031] - Use the data obtained from the sensor unit included in the kinematic chain to calculate the reference R of the surgical tool O and the reference R of the mechanical reference M transformation between O T M ;

[0032] - Calculate the reference R of the mechanical reference by matching the digital model of the mechanical reference with at least a part of the mechanical reference included in the 3D image M and the acquisition reference R A transformation between M T A ;

[0033] such that C T O the transformation is from the acquisition reference RA 、The mechanical reference R M and the target reference R C between the transformation O T M 、 M T A and C T A obtained in the combination of.

[0034] According to one embodiment, in which the target anatomical structure is fixed to at least one mechanical reference, the system further comprises a correction module configured to use a sensor unit of the kinematic chain to track the movement of the target relative to the surgical tool, and thus whenever a deviation in the position and / or spatial orientation of the target is detected, correct the conversion of the planned action from the virtual environment to the real environment for the said deviation.

[0035] This embodiment advantageously allows avoiding the need for visual tracking or a new registration procedure when the patient moves.

[0036] According to one embodiment, the at least one 3D imaging sensor is fixed to the kinematic chain, and the calculation module is further configured to calculate the reference R of the surgical tool according to the data obtained from the sensor unit of the kinematic chain O and the acquisition reference R of the 3D imaging sensor A between the transformation A T O , such that C T O The transformation is from the acquisition reference R A and the target reference R C between the transformation A T O and the transformation C T A obtained in the combination of.

[0037] According to one embodiment, the acquisition of the 3D image received by the receiving module is performed using at least two sensors and projectors to perform the acquisition by stereoscopic vision or structured light.

[0038] According to one embodiment, the 3D imaging sensor fixed to the kinematic chain moves along a known trajectory, and a plurality of 3D images are acquired along the trajectory, and the calculation module is further configured to jointly process the plurality of 3D images acquired along the trajectory to register with the target digital model using the plurality of 3D images.

[0039] According to one embodiment, the receiving module is further configured to receive thermal images, ultrasound images, multispectral images, images at the microscopic scale, and / or monocular color images.

[0040] According to one embodiment, when the 3D imaging sensor is fixed to the kinematic chain, the system further includes a correction module configured to use the 3D imaging sensor and a visual tracking algorithm to track the movement of the target relative to the surgical tool, such that whenever a deviation in the position and / or spatial orientation of the target is detected, the conversion of the planned action from the virtual environment to the real environment is corrected for the deviation.

[0041] The present invention relates to a method for computer-assisted surgery, which includes converting the planned action in a virtual environment relative to a virtual reference R P into a physical action performed in a real operating room environment for an orthopedic surgery of a patient using a surgical tool, the surgical tool being fixed to a kinematic chain including a sensor unit having at least one sensor configured to follow in real time the spatial configuration of the kinematic chain; the method includes the following steps:

[0042] - receiving at least one 3D image acquired from at least one 3D imaging sensor; the 3D image including at least a part of the target anatomical structure of the patient;

[0043] - calculating a virtual reference R by registering a digital model of the target with at least a part of the target included in the 3D image P between the target reference R C and the transformation C T P as well as the acquisition reference of the 3D imaging sensor R A and the target reference R C between the transformation C T A ;

[0044] - applying the transformation C T P so as to register the digital model of the target in the target reference R C such that each point included in the digital model of the target has a known localization in the target reference R C ;

[0045] - calculating the transformation O between the reference R of the surgical tool and the target reference R C ; C T O ;

[0046] - applying the transformation C T O to the reference R of the surgical tool O to determine the position and spatial orientation of the surgical tool in the target reference R C ;

[0047] Thereby determining the position and spatial orientation of the surgical tool in the virtual reference R P and the target reference R C such that the actions planned in the virtual reference R C are reproduced in the target reference R P .

[0048] According to one embodiment, calculating the transformation C T A comprises the following steps:

[0049] - Defining a region of interest in a 3D image including the target;

[0050] - Registering the region of interest including the target to a digital model of the target to determine C T A .

[0051] According to one embodiment, the step of defining the region of interest comprises automatically detecting the region of interest by means of a segmentation algorithm.

[0052] According to one embodiment, at least one 3D imaging sensor is fixed to a kinematic chain, and the method further comprises the following steps: calculating the transformation O between the reference R A of the surgical tool and the acquisition reference R A T O of the 3D imaging sensor based on data obtained from the sensor unit of the kinematic chain, such that C T O the transformation is obtained from the combination of the transformation A T O and the transformation A between the acquisition reference R C and the target reference R C T A .

[0053] According to one embodiment, the kinematic chain includes at least one mechanical reference rigidly fixed to the target anatomical structure, and at least one 3D image includes at least a portion of the mechanical reference; the method further comprises the following steps:

[0054] - Using data obtained from the sensor unit included in the kinematic chain to calculate the transformation O between the reference R M of the surgical tool and the reference R O T M of the mechanical reference;

[0055] - Calculating the reference R M of the mechanical reference and the acquisition reference RA Transformation between M T A ;

[0056] such that C T O the transformation is from the acquisition reference R A the mechanical reference designation R M and the target reference R C Transformation between O T M , M T A and C T A and obtained from a combination of

[0057] According to one embodiment, wherein the mechanical reference is rigidly fixed to the target anatomical structure, the movement of the target relative to the surgical tool is tracked by the sensor unit of the kinematic chain, and thus whenever a deviation in the position and / or spatial orientation of the target is detected, the transformation of the planned action from the virtual environment to the real environment is corrected for the deviation.

[0058] According to one embodiment, the kinematic chain consists of a deformable structure including a plurality of rigid elements connected by joints.

[0059] According to one embodiment, the kinematic chain further includes sensors for measuring the forces applied to its elements.

[0060] According to one embodiment, the acquisition of the 3D image collected by the receiving module is performed using at least two sensors and a projector to perform the acquisition by stereovision or structured light.

[0061] According to one embodiment, the 3D imaging sensor fixed on the kinematic chain moves along a known trajectory and acquires a plurality of 3D images along the trajectory, and the method further includes the following steps: jointly processing the plurality of 3D images acquired along the trajectory to register the plurality of 3D images with the target digital model.

[0062] According to one embodiment, the method further includes receiving thermal images, ultrasound images, multispectral images, images at the microscopic scale, and / or monocular color images.

[0063] According to one embodiment, wherein the 3D imaging sensor is fixed on the kinematic chain, the movement of the target relative to the surgical tool is tracked by the 3D imaging sensor and a visual tracking algorithm, such that whenever a deviation in the position and / or spatial orientation of the target is detected, the transformation of the planned action from the virtual environment to the real environment is corrected for the deviation.

[0064] According to one embodiment, the three-dimensional digital model of the target is generated based on computed tomography images or MRI images.

[0065] According to one embodiment, a three-dimensional digital model of a target is generated using 2D X-ray photographs, the 2D X-ray photographs including the target, a statistical shape model of the target, and / or 3D images acquired during surgery by a 3D imaging sensor.

[0066] According to one embodiment, the three-dimensional digital model of the target is digitally modified to simulate measurement noise or the presence of cartilage, the modification being calculated based on training data or biomechanical simulation data.

[0067] According to one embodiment, the action of matching the digital model of the target with at least a portion of the target included in the 3D image is a non-rigid transformation.

[0068] The invention also relates to a computer program product comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method according to any one of the embodiments described herein.

[0069] The invention also relates to a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method according to any one of the embodiments described herein.

[0070] As described above, one embodiment of the invention relates to a method for computer-assisted surgery using a kinematic chain that includes at least one mechanical reference rigidly fixed to a target anatomical structure. The following paragraphs relate to this particular embodiment.

[0071] The method enables a mechanical reference that is kinematically linked to a surgical tool via the kinematic chain: during the entire surgery, the position and orientation of the surgical tool relative to the mechanical reference are known, thanks to position and / or displacement sensors of the components of the kinematic chain. On the other hand, the mechanical reference is rigidly linked to the bone element to be processed (i.e., the target). During surgery, the invention uses depth data (also referred to as 3D images) to register the bone element with planning data obtained preoperatively or intraoperatively (at the start of the surgery, after soft tissue incision but before bone resection), the planning data including a digital model of the target and the planned surgical actions to be performed during the orthopedic surgery. In this way, the position and orientation of the mechanical reference relative to the target are determined. Knowledge of the registration process and the kinematic chain geometry makes it possible to determine the trajectory of the surgical tool in the target reference frame. The position and orientation of the surgical tool relative to the bone element are always known, and its path can be corrected, for example, in the event of patient movement, to conform to a previously established surgical plan.

[0072] The present invention overcomes the drawbacks of the prior art as it does not require ionizing imaging or multiple optical markers to control the surgical tool holder to perform the actions included in the surgical planning. Considering the potential hazards and time-consuming nature, the present invention shows direct health benefits for the medical team and the patient. It also provides advantages to the surgeon in terms of surgical length and ease of use.

[0073] The present invention relates to a method comprising the following steps.

[0074] A first step of acquiring at least one 3D image from a 3D image sensor of a scene, the first step comprising:

[0075] - at least a part of the target (i.e., the bone element to be processed);

[0076] - at least a part of the mechanical reference linked to the tool movement.

[0077] A second step of mapping the 3D image to a digital model of the target by applying an image processing algorithm to determine the transformation A T C defining the positioning and spatial orientation of the acquisition reference R C with respect to the reference R A of the target.

[0078] A third step of mapping the 3D image to a digital model of the mechanical reference such that the transformation M T A can be determined, which transformation M T A defines the position and orientation of the acquisition reference R A of the 3D image sensor with respect to the reference R M of the mechanical reference.

[0079] A fourth step of using C T A and M T A transformation matrices to calculate the transformation matrix between the target reference R C and the mechanical reference reference R M .

[0080] A fifth step of transforming the initial plan into the target reference R C .

[0081] In parallel, the method comprises the following steps: real-time acquisition of the position, spatial orientation and / or displacement of the elements of the kinematic chain that links the mechanical reference to the surgical tool during the physical action of using the surgical tool.

[0082] Advantageously, the method does not rely on real-time optical monitoring of the scene. Thus, it avoids problems associated with standard surgical navigation systems, such as the diversity of markers and loss of tracking in the case of occlusion of the surgical area. In fact, once the planning data is known in the target reference R C the sensors of the kinematic chain can provide at any time knowledge of the position and orientation of the surgical tool relative to the target to be calculated.

[0083] According to a variant embodiment, the second and third mapping steps are replaced by the following steps.

[0084] The region of interest extraction step includes extracting a first region of interest corresponding to the target and a second region of interest corresponding to the mechanical reference in order to determine:

[0085] - a first subset of the 3D digital image associated with the target;

[0086] - a second subset of the 3D digital image associated with the mechanical reference.

[0087] The extraction of the regions of interest can be performed automatically by a segmentation algorithm.

[0088] The step of mapping the first subset associated with the target to the digital model of the target to determine the transformation C T A step.

[0089] The step of mapping the subset associated with the mechanical reference to the digital model of the mechanical reference to determine the transformation M T A step.

[0090] According to a variant of this embodiment, carried out individually or in a technically practical combination, the invention also relates to the following additional features:

[0091] - The kinematic chain consists of a rigid deformable structure including sensors measuring the relative positions of its components;

[0092] - The first step of acquiring the 3D image is carried out by acquiring a textured 3D image by stereovision using at least two cameras and a projector;

[0093] - The first step of acquiring the 3D image is carried out by acquiring a textured 3D image by structured light using at least two sensors and a projector;

[0094] - The first acquisition step further includes acquiring an RGB-D image;

[0095] - The first acquisition step further includes acquiring a thermal image;

[0096] - The first acquisition step further includes acquiring an ultrasonic image;

[0097] - The first acquisition step further includes acquiring a multispectral image;

[0098] - The first acquisition step further includes acquiring an image at the microscale;

[0099] - The first acquisition step further includes acquiring a monocular color image;

[0100] - The movement of the surgical tool relative to the target is tracked by means of position and / or displacement sensors of elements of the kinematic chain between the target and the surgical tool, such that the conversion of the planned action from the virtual environment to the real environment can be corrected;

[0101] - The three-dimensional digital model of the target is generated from a scanner or MRI images;

[0102] - The three-dimensional digital model of the target has been digitally modified to simulate measurement noise or the presence of cartilage, and the modification is calculated based on training data or biomechanical simulation data;

[0103] Advantageously, it enables the determination of the transformation A relative to the target reference R C that defines the position and orientation of the acquisition reference R C T A The digital reset process of the second step is non-rigid.

[0104] Define

[0105] In the present invention, the following terms have the following meanings:

[0106] - "3D sensor" or "3D camera" or "depth camera" or "3D scanner" means that a 3D sensor is a system for acquiring 3D topological data of a real scene. These topological data are recorded in the form of a point cloud and / or a depth map.

[0107] Multiple acquisition techniques allow obtaining these topological data, for example:

[0108] · Techniques based on wave propagation time measurement, such as ultrasonic waves or light (LiDAR LIDAR, time of flight);

[0109] · Stereo cameras or sensors, which are cameras with two or more lenses, each lens having a separate image sensor or film frame. This allows the camera to simulate human binocular vision and thus be able to capture three-dimensional images;

[0110] · Techniques based on light deformation, such as structured light 3D scanners, which project a pattern of light onto an object and observe the deformation of the pattern on the object. The advantages of structured light 3D scanners are speed and accuracy. Instead of scanning one point at a time, structured light scanners scan multiple points or the entire field of view at once. Scanning the entire field of view in a fraction of a second can reduce or eliminate motion distortion problems;

[0111] · Techniques based on laser scanning of samples or using laser technology to scan surfaces, such as hand-held lasers or time-of-flight 3D laser scanners;

[0112] These terms can also refer to RGB-D, color, multi-spectral, or thermal imagers.

[0113] - "Reference" refers to a coordinate system that uses one or more numbers or coordinates to uniquely determine the position of a point or other geometric element on a manifold (e.g., Euclidean space).

[0114] - "Tracking", in computer vision, refers to the act of tracking the position and spatial orientation of an object between successive image frames. Marker-based tracking relies on using a positioning device associated with a marker attached to the object of interest. Markerless tracking relies on extracting visual features from the object of interest itself and matching them on a frame-by-frame basis.

[0115] - "Registration" or "matching" or "pose estimation" refers to the process of transforming different data sets into one coordinate system.

[0116] - "Three-dimensional digital model" refers to a three-dimensional digital (or virtual) model that is a three-dimensional virtual object. The position and orientation of the model are known in the associated digital reference.

[0117] - "Planning", in the context of surgery, refers to a sequence of actions to be performed during different surgical stages. This surgical planning can be obtained with the aid of a simulation program performed before the surgery, which uses a three-dimensional digital model of the patient's skeleton as the surgical target. For example, in the case of knee replacement surgery, the preoperative planning will consist of defining the machining plane and the drilling axes related to the three-dimensional models of the femur and tibia.

[0118] - "Preoperative data" refers to the images (or slices) of the patient obtained through medical imaging (CT, MRI, PET, etc.). The three-dimensional model of the patient is obtained by performing segmentation processing on each image and then interpolating between the images.

[0119] - "Intraoperative data" refers to the data collected during the surgery. This can include medical imaging (fluoroscopy, ultrasound, etc.), three-dimensional data, color and temperature information, information from proprioceptive sensors, force feedback related to surgical tools, etc.

[0120] - "Machining" refers to the mechanical process of cutting or other material removal. The purpose of machining is to modify the dimensions, accuracy, geometry, and surface condition of all surfaces of the finished component so as to move from the original initial state to the final state according to a predefined model. Detailed implementation

[0121] The following detailed description will be better understood when read in conjunction with the accompanying drawings. For illustrative purposes, the steps implemented by the system and the method are shown in the preferred embodiment. However, it should be understood that the present invention is not limited to the exact arrangements, structures, features, embodiments, and aspects shown. The drawings are not drawn to scale and are not intended to limit the scope of the claims to the depicted embodiments. Therefore, it should be understood that where features are referred to in the appended claims followed by reference numerals, these reference numerals are included only to enhance the understandability of the claims and do not limit the scope of the claims in any way.

[0122] The features and advantages of the present invention will become apparent from the following description of system embodiments, which is given by way of example only and with reference to the drawings, in which:

[0123] Figure 1 and 1B A schematic diagram showing different reference systems defined in the present invention according to an embodiment in which the kinematic chain includes at least one mechanical reference.

[0124] Figure 2 A schematic diagram of a process step in which the target reference R C and the virtual reference R P are aligned to determine the acquisition reference R A relative to R C for positioning and orientation C T A transformation. C T A is a homogeneous transformation matrix composed of C R A rotation matrix and C T A translation vector. It defines the position of the target reference R C relative to the acquisition reference R A in position.

[0125] Figure 3 A fourth step according to an embodiment of the present invention is shown, in which the mechanical reference R M relative to the acquisition reference R A is determined for position and spatial orientation (transformation matrix M T A ). M T Ais a homogeneous transformation matrix, consisting of a rotation matrix MR A and a translation vector M T A It defines the reference R of the mechanical reference M relative to the acquisition reference R A in terms of position and spatial orientation.

[0126] Figure 4 Indicates the C TM transformation between the mechanical reference and the target is calculated through A T C (the inverse C T A matrix) and M T A matrix combination.

[0127] Figure 5 Corresponds to the sixth step of this process. The transformation O T M determined by the kinematic chain linking the mechanical reference and the surgical tool C is combined with the transformation C T O (the position of the surgical tool in the target and the planning reference).

[0128] Figure 6 and 7 show two examples of how to construct the system and the kinematic chain according to an embodiment, where the mechanical reference is kinematically linked to the surgical tool. The human-machine interface represented by the screen 60 here provides visual feedback on each process step.

[0129] Figure 8 Shows some machining planes P1, P2, P3, P4, P5, P6 of the femoral knee implant I, the femur F, and the tibia T implanted in a knee replacement surgery.

[0130] Figure 9 Shows an example of the system of the present invention and a kinematic chain including a surgical tool at one end, where the 3D imaging sensor has a known position relative to the patient.

[0131] Figure 10 Schematically represents the step of calculating the transformation P between the virtual reference R C and the target reference R C T P and the transformation A between the acquisition reference R C of the 3D imaging sensor and the target reference R C T A of.

[0132] Figure 11 Schematically represents the reference R of the surgical tool O and the target reference R C between the transformation C T O steps.

[0133] Figure 12 Is a schematic diagram of an embodiment in which at least one 3D imaging sensor is fixed on the kinematic chain. The reference R of the surgical tool is also illustrated in this figure O and the acquisition reference R of the 3D imaging sensor A between the transformation A T O and the acquisition reference R A and the target reference R C between the transformation C T A , the transformation A T O and the transformation C T A for calculating the transformation C T O .

[0134] Although various embodiments have been described and illustrated, the detailed description should not be construed as limited thereto. Without departing from the true spirit and scope of the present disclosure as defined by the claims, those skilled in the art can make various modifications to the embodiments.

[0135] The object of the present invention is to match the preoperative planning data with the actual surgery occurring in the operating room in an uninterrupted and real-time manner.

[0136] Preoperative surgical planning includes a 3D digital model of at least a part of the patient, especially including the target bone element, and a set of ordered geometric equations characterizing the machining actions on the target bone element of the patient. In a preferred embodiment, the preoperative surgical planning includes a machining plan corresponding to each surgical action. The term preoperative surgical planning should be understood as defining the surgical actions of the surgical planning using the planning data obtained preoperatively (preoperative) or intraoperatively during the first stage of the surgery when the surgical action on the target bone element on the target bone is about to start.

[0137] The digital model of the target can be generated from medical images acquired before the surgery. It can be modified to take into account elements that are not visible in the medical images, such as cartilage that is not visible in CT scan images. In this case, the modification is generated from training data or biomechanical simulation data. The digital model can also be generated by a statistical model or an abacus as well as patient data that is related or not related to the preoperative medical images. In addition, the digital model can be adjusted by considering the data acquired during the operation.

[0138] The planning data for determining a preoperative surgical plan is acquired at the location, time, and position of the patient relative to an imaging device that is completely independent of the imaging device for the surgery. However, the digital data of the digital planning model must be closer to the physical reality to allow the movement of a real machining tool to be controlled based on the digital planning data.

[0139] For the remainder of the description, the following references will be considered:

[0140] - The digital planning image stored in a computer memory and the plan of the surgical actions of the surgical plan or the virtual reference R P . The bone element to be machined (the femur 10 in the example described) in the digital model of the target has a known positioning and spatial orientation in this virtual reference R P . The surgical planning data and the surgical actions (such as the position of the cutting plane 11 or the drilling axis) are known in the same reference R P ;

[0141] - The target reference R C corresponds to the physical coordinate system of the bone element to be machined (i.e., the target) (in this case the surface of the femoral head);

[0142] - The surgical tool reference R O corresponds to the physical coordinate system of the surgical tool 20;

[0143] - The acquisition reference R A corresponds to the coordinate system of the 3D imaging sensor 30, where the data acquired during the operation is represented;

[0144] - The mechanical reference R M corresponds to the physical coordinate system of the mechanical reference 40, which is kinematically linked to the surgical tool 20 through the elements of the kinematic chain. The position and spatial orientation of the surgical tool 20 relative to the mechanical reference 40 are known by the sensors of the kinematic chain, thus providing signals representing the angular displacement and / or linear displacement of the surgical tool 20 relative to the mechanical reference 40.

[0145] The positioning of the bone target 10, the kinematic chain 70, the surgical tool 20, the 3D imaging sensor 30, and their respective references is as Figure 9 shown.

[0146] The method aims to precisely guide the surgical tool 20 that is movably fixed to the kinematic chain 70. The surgical tool 20 can be, for example, a machining tool.

[0147] In the present invention, the term kinematic chain refers to a set of rigid elements that are joined by joints to restrict or provide movement in a desired manner.

[0148] According to one embodiment, the kinematic chain consists of a deformable structure including a plurality of rigid elements connected by joints.

[0149] According to the present invention, the kinematic chain includes a sensor unit having at least one sensor configured to follow in real time the spatial configuration of the kinematic chain.

[0150] The sensors of the sensor unit can be encoders or inertial units including accelerometers and / or gyroscopes.

[0151] According to one embodiment, the kinematic chain further includes sensors for measuring the forces applied to its elements.

[0152] In one embodiment, the first step of the method includes receiving at least one 3D image acquired from at least one 3D imaging sensor 30, wherein the 3D image is acquired to include at least a part of the target anatomical structure 10 of the patient.

[0153] The 3D image obtained from the 3D imaging sensor 30 of the present invention includes distance information between each point of the scene acquired in the image and the 3D imaging sensor 30. Thus, the original 3D image obtained by the 3D imaging sensor 30 is a so-called depth map or depth image, which can be presented in the form of a two-dimensional array representing a grayscale image or an RGB image, where the size of the array depends on the camera type and the sensor size.

[0154] According to one embodiment, the acquisition of the 3D image is performed using at least two cameras and a projector to perform the acquisition by stereovision or structured light.

[0155] The use of the 3D imaging sensor advantageously allows obtaining information about the surface morphology of the bone in a simple and fast manner, since one image allows capturing all the surgical areas without contacting the patient (as in the case of palpation techniques).

[0156] According to one embodiment, the method further includes a preprocessing step of performing a noise reduction algorithm.

[0157] According to one embodiment, at least one 3D imaging sensor 30 has a fixed position relative to the target 10 in the operating room. In this embodiment, the 3D imaging sensor 30 is independent of the kinematic chain (i.e., the 3D imaging sensor is not fixed to the kinematic chain). In one example, the 3D imaging sensor 30 is fixed to the wall of the operating room or positioned on a tripod or fixed by means of an articulated arm. In the case where the 3D imaging sensor 30 is displaced to capture multiple 3D images, an inertial measurement unit (IMU) fixed to the 3D imaging sensor 30 can measure its relative motion and determine the motion trajectory.

[0158] According to an alternative embodiment, as Figure 12As shown, at least one 3D imaging sensor 30 is fixed to the kinematic chain. This advantageously allows for a constant proximity during the surgery to the relative position of the surgical tool 20 with respect to the 3D imaging sensor 30.

[0159] When at least one 3D imaging sensor 30 is fixed to the kinematic chain 70, the 3D imaging sensor moves along a known trajectory along which the 3D imaging sensor 30 can acquire a plurality of 3D images.

[0160] According to one embodiment, the first step further includes receiving thermal images, ultrasound images, multispectral images, images at the microscopic scale, and / or color images.

[0161] According to one embodiment, the method includes retrieving a digital model of the target bone to be treated with the surgical tool 20 during the surgery from a computer-readable storage medium, a server, etc. The digital model is a three-dimensional virtual representation of the target bone 10.

[0162] In one embodiment, imaging data acquired using a computed tomography or MRI system is used to generate a three-dimensional digital model of the target. Other imaging techniques, such as X-rays, fluoroscopy, ultrasound, or other imaging means, can also be used. In this case, the three-dimensional digital model is obtained before the surgery.

[0163] In one embodiment, the three-dimensional digital model of the target is generated using 2D X-ray photographs, which include the target, a statistical shape model of the target, and / or 3D images acquired intraoperatively by the 3D imaging sensor 30. This embodiment also advantageously allows for the generation of a three-dimensional model even when 3D imaging data (i.e., computed tomography or MRI) is not available.

[0164] In one embodiment, the three-dimensional digital model of the target is modified to simulate measurement noise or the presence of cartilage. The modification can be calculated from training data or biomechanical simulation data.

[0165] According Figure 2 and Figure 10 to one embodiment shown, another step of the method includes: calculating a virtual reference R by registering the digital model of the target with at least a portion of the target 10 included in the 3D image P and the transformation C between the target reference R C T P and the acquisition reference of the 3D imaging sensor R A and the transformation C between the target reference R C T A .

[0166] 3D registration consists of finding the transformation between two 3D models of the same object such that their overlapping regions match as closely as possible. This can be performed by an algorithm that iteratively aligns the two models by alternately associating each point in the intraoperative image with its nearest neighbor in the preoperative model and alternately transforming the intraoperative model to best fit the estimated matches in a transformation estimation step. This process is repeated until the distance between each point in the intraoperative and preoperative models is minimized below a threshold.

[0167] This step advantageously allows determination of the transformation that will align the virtual reference R P with the target reference R C in the operating room. Additionally, using a digital model of the skeleton and its registration with at least one 3D image of the target in the surgical field allows determination of the transformation between the virtual reference R P and the target reference R C independently of any external markers attached to the patient.

[0168] In one embodiment, registering the digital model of the target with at least a portion of the target included in the 3D image is obtained from a rigid transformation.

[0169] Alternatively, registering the digital model of the target with at least a portion of the target included in the 3D image can be a non-rigid transformation. Advantageously, this embodiment allows the shape of the digital model of the target obtained from pre-acquired images to be adapted to the 3D image acquired during surgery.

[0170] According to one embodiment, the step of calculating the transformation C T A includes a first step of defining a region of interest in the 3D image that includes at least a portion of the target.

[0171] According to one embodiment, the step of defining the region of interest includes automatically detecting the region of interest by means of a segmentation algorithm.

[0172] Alternatively, the operator can provide information in the form of a manual delineation of the contour of the region of interest that includes the target as an input to the method.

[0173] The region of interest that includes the target is then registered to the digital model of the target to determine C T A .

[0174] In one embodiment, the method further includes the step of applying the transformation C T P so as to register the digital model of the target in the target reference R C such that each point included in the digital model of the target is in the target reference R Chas a known positioning. This step advantageously allows the virtual reference associated with the digital model of the target and the actions of the surgical planning to be aligned with the target reference R in the operating room C Align.

[0175] In Figure 11 In one embodiment shown, the method includes calculating the transformation O between the reference R of the surgical tool and the target reference R C T C T O .

[0176] The method can then implement applying the transformation C T O to the reference R of the surgical tool to determine the position and spatial orientation of the surgical tool 20 in the target reference R O . This last step allows determining the positioning and spatial orientation of the surgical tool 20 in the virtual reference R C and the target reference R P so as to reproduce in the target reference R C the actions planned in the virtual reference R C . P

[0177] When the positioning and spatial orientation of the surgical tool 20 in the virtual reference R P and the target reference R C are known, the kinematic chain carrying the surgical tool 20 can be used to guide the execution of the actions planned in the initial surgical planning. However, during the execution of these actions, the spatial orientation and position of the target may change, for example, by a medical staff member moving the target.

[0178] This will result in a mismatch between the virtual reference R P and the target reference R C , and thus, the surgical tool 20 executing the planned actions with reference to the virtual reference R P will be mispositioned.

[0179] To prevent this undesirable situation, the movement of the target 10 relative to the surgical tool 20 can be tracked such that whenever a deviation in the positioning and spatial orientation of the target is detected, the registration of the planned actions from the virtual reference R P and the target reference R C is immediately corrected for the deviation.

[0180] According to one embodiment in which the 3D imaging sensor 30 is fixed to the kinematic chain 70 as shown in Figure 12 , the method is configured to calculate the reference R of the surgical tool based on data obtained from the sensor unit of the kinematic chain O ​The reference of [object] and the acquisition reference R of the 3D imaging sensor A The transformation between A T O and by combining the said transformation A T O into the transformation C T A so as to obtain the transformation between the surgical tool R O and the target reference R C between C T O transformation.

[0181] As Figure 12 shown, in the case where the 3D imaging sensor 30 is fixed to the kinematic chain 70, according to one embodiment, the movement of the target 10 relative to the surgical tool 20 is calculated by a visual tracking algorithm that uses the real-time 3D images captured by the 3D imaging sensor 30 for the continuous pose (i.e., spatial orientation and positioning) estimation of the target 10 relative to the 3D imaging sensor 30 as input. The relative movement between the 3D imaging sensor 30 and the target 10 is calculated by performing frame-to-frame registration, i.e., by registering the current 3D image (time step i) with the previous 3D image (time step i-1). Since the 3D imaging sensor 30 is mechanically linked to the surgical tool 20, the relative movement estimated from visual tracking should match the relative movement calculated from the sensor unit of the kinematic chain 70. If this is not the case, it means that the target 10 has moved and the registration of the planning actions from the virtual reference R P and the target reference R C needs to be performed again to correct the deviation.

[0182] According to an alternative embodiment where the kinematic chain 70 is independent of the 3D imaging sensor 30, the kinematic chain 70 includes at least one mechanical reference 40 rigidly fixed to the target anatomical structure 10. According to Figure 1 and 1B shown in this embodiment, at least one 3D image must include at least a part of the mechanical reference.

[0183] According to Figure 3 shown in this embodiment, the method further includes the following steps:

[0184] - Using the data obtained from the sensor unit included in the kinematic chain 70 to calculate the transformation O between the reference R of the surgical tool M and the reference R of the mechanical reference O T M ;

[0185] - Calculating a reference R of the mechanical reference by matching a digital model of the mechanical reference with at least a part of the mechanical reference included in at least one 3D image M With the acquisition reference R A The transformation between M T A ;

[0186] Such that C T O The transformation is from the acquisition reference R A The mechanical reference R M And the target reference R C The transformation between O T M 、 M T A And C T A Obtained from the combination of

[0187] As Figure 1 、 1B And as shown in 7, when the mechanical reference 40 is fixed to the target 10, according to one embodiment, the movement of the target 10 relative to the surgical tool 20 is tracked by the sensor unit of the kinematic chain 70, such that whenever a deviation in the position and spatial orientation of the target 10 is detected, the registration of the planned actions from the virtual reference R P And the target reference R C Is corrected immediately for the deviation. In fact, since the mechanical reference 40 is rigidly fixed to the target anatomical structure 10 and is simultaneously part of the kinematic chain 70, the information collected from the sensor unit can be used to detect all movements of the target 10 relative to the surgical tool 20. This correction may include calculating a correction transformation C Between the new target reference R P And the virtual reference R Cnew TP, such that each point included in the digital model of the target has a known position in the new target reference R of the mechanical reference C

[0188] A specific embodiment of the steps of the method to be implemented when using a kinematic chain 70 including at least one mechanical reference 40 is described in detail in the following paragraphs

[0189] ​As an example, the surgical tool 20 is associated with a tool holder (such as a clamp or a screw) connected to a mechanical reference, which is mechanically fixed to the bone to be processed. The mechanical connection between the tool holder and the mechanical reference is provided by a kinematic chain having a deformable structure, which includes at least two elements, for example, a hinged assembly including a plurality of hinged elements and a sensor unit having at least one sensor, and the sensor unit provides a signal in real time according to the relative spatial orientation and position of the hinged elements. The sensor unit thus provides digital information, enabling the position of the active end of the surgical tool in space to be determined in real time relative to the fixed point of the element fixed to the bone.

[0190] In addition, intraoperative data acquisition and its matching with digital surgical planning allow the position of the mechanical reference fixed to the bone relative to the surface of the bone element to be processed to be known. The known trajectory can be transformed into the real world through simulation in a virtual reference, for example, to ensure tool guidance during surgical planning, control of its movement, or position control relative to a predetermined position.

[0191] The first step

[0192] For this purpose, a 3D image is acquired by a 3D imaging sensor 30 (such as a camera), the field of view of which includes a part of the surgical area, including at least a part of the surface to be processed 10 (such as the femur) and a part of the mechanical reference 40. The result of the acquisition can be displayed on the screen 60.

[0193] The acquisition can be performed by a 3D camera, a pair of cameras acquiring images with active stereovision, a 3D scanner, or lidar to provide a three-dimensional image or depth map of type (x, y, z; a), a point cloud, specified parameters (such as color or intensity).

[0194] 3D image acquisition of the surgical scene

[0195] The texture 3D digitization solution uses two or more calibrated cameras and projectors to perform stereovision acquisition and phase-shift structured light acquisition to achieve precise 3D reconstruction of the surgical area. The proposed solution integrates a spatio-temporal super-resolution scheme with non-rigid 3D registration to correct 3D information and complete the scanned view.

[0196] Structured light is encoded by time multiplexing. Two sine patterns with opposite phases and a third white pattern are successively projected onto the surgical scene. First, 2D sampling is applied to each camera separately to locate the stripe intersections. Then, through the stereo matching between the obtained primitives and optical triangulation, a non-dense 3D model of the scene is estimated for each pair of cameras. The spatial resolution of this model depends on the number of stripes forming the patterns used.

[0197] Then, a dense 3D model is obtained for each camera-projector pair used by estimating the phase information of the points located within the stripes. Traditional phase-shift structured light-based methods require offline calibration of the camera with respect to the projector and a phase unwrapping step.

[0198] Spatial and temporal super-resolution enables the completion and correction of the 3D model of the observed scene, since 3D scanning is capable of generating distortions and artifacts mainly caused by occlusions, position variations, and even light reflections on the acquisition surface. Thus, on the one hand, the different 3D models provided by all the camera pairs are utilized, and on the other hand, the 3D frame calculated at time t-1 is utilized to obtain a high-resolution 3D model corrected at time t. Spatiotemporal super-resolution is first provided by a first 3D matching step and then by a merging and denoising step.

[0199] Non-rigid 3D matching methods allow for the handling of possible distortions or deformations of non-rigid observed regions. The meshing and texturing plating of the obtained 3D point cloud allows for the completion of the textured 3D frame at instant t.

[0200] The result of this first step is recorded in the memory of a computer in the form of a point cloud in a 3D image of the region containing the visible part of the skeleton 50 and the visible part of the reference 40, each point cloud being defined by the luminous intensity, color, and coordinates (x, y, z) in the acquisition reference R A .

[0201] Additional imaging modality

[0202] A particular solution is to acquire additional images of different natures in the same acquisition reference R A or in a reference calibrated using R A to obtain additional information. Calibration of the additional modality requires the use of a geometric test pattern visible from different viewpoints by both the 3D imaging sensor 30 and the additional modality. The resulting image pairs are processed and rescaled to derive a calibration matrix.

[0203] The additional image can be, for example, a thermal image produced by a thermal camera that captures the surgical area at an orientation and distance close to those of the 3D imaging sensor 30. This image advantageously makes it easier to distinguish patient tissue from surgical tools.

[0204] It can also be the acquisition of a color camera, an ultrasound probe, or a multispectral sensor.

[0205] Step 2: Extract the region of interest

[0206] The following processing steps include isolating the part of the image corresponding to the target 10 (femur) and the part of the image corresponding to the mechanical reference 40 by using at least one 3D image recorded during the acquisition step.

[0207] To this end, the 3D digital image of the entire scene and the images obtained from the additional images (if any) are processed by an algorithm for characterizing subsets of depth maps or point clouds.

[0208] The result of this processing will be segmentation or classification:

[0209] - Using a first indicator (label) associated with a mechanical reference corresponding to a first subset of points of the 3D image,

[0210] - Using a second indicator (label) associated with a target (femur) corresponding to a second subset of points of the 3D image,

[0211] - Using a third indicator (label) for the background (non-relevant subset of the image).

[0212] This processing step will be performed by successive contours, colors and alternatively using a trained classifier, or by artificial intelligence, or by geometric priors taking into account the positioning of the mechanical reference and the target relative to the acquisition system.

[0213] If the matching algorithms applied in the third and fourth steps are robust enough to outliers, then the step of extracting the region of interest is optional.

[0214] The third step: Match the physical reference system linked to the target with the acquisition reference system.

[0215] The third step consists of registering the target reference R associated with the physical target C with the acquisition reference R A to perform a match:

[0216] - A subset of the 3D digital image associated with the target determined in the previous step, and

[0217] - A 3D digital model of the target recorded together with the planning data.

[0218] This processing consists of determining C T A the transformation, giving the position and orientation of the acquisition reference R A relative to the target reference R C as Figure 2 shown.

[0219] This processing uses registration techniques to find the potential deformations common to two geometric structures of the same nature, enabling them to be linked, i.e., attempting to describe the second structure as obtained from the first by applying a spatial transformation.

[0220] Match techniques based on prior extraction of feature points, inducing deformations from feature points, or using geometric structures derived from the original images are known to those skilled in the art: parts of points, curves, or surfaces obtained by segmentation, as long as these capture the basic information of the image, whether geometric - points or lines of strong curvature - or anatomical.

[0221] A suitable technique is based on point - to - point registration, by the process of estimating the best transformation between two sets of data such that their overlapping regions match as closely as possible. This can be carried out by an algorithm that iteratively aligns the two models through a matching step that alternately associates each point of the intraoperative image with its nearest neighbor in the preoperative model and a transformation estimation step that transforms the intraoperative cloud to the best - fit estimated match. This process is repeated until the distance between each point of the intraoperative and preoperative models is minimized and below a threshold.

[0222] If the geometric transformation includes rotation and translation, the registration is said to be rigid. If the geometric transformation is of higher order (polynomial, spline,...) or if the transformation is not parameterized, the registration is said to be non - rigid.

[0223] In the context of the present invention, rigid registration is generally sufficient to calculate the transformation matrix from the target reference to the virtual reference R P of.

[0224] The fourth step: Map the physical reference system linked to the mechanical reference to the acquisition system.

[0225] The fourth step includes registering and processing the mechanical reference R associated with the mechanical fiducial M with the acquisition reference R A by matching:

[0226] - the 3D image of the mechanical fiducial, and

[0227] - the three - dimensional digital model of the mechanical fiducial.

[0228] Applying the same type of rigid registration process using a subset of the 3D image points corresponding to the mechanical fiducial and their digital representation in the computer memory.

[0229] Figure 3 The result of the shown process is used to determine M T A the transformation, giving the position and orientation of the mechanical reference R M relative to the acquisition reference R A of.

[0230] Fifth step: Target reference R C and mechanical reference R M transformation between

[0231] Figure 4 The fifth step of the shown process includes calculating the target reference R CTransformation with respect to the mechanical reference R M Due to the two previous steps, it is known that C T A and M T A matrices (as shown in Figure 2 and Figure 3 ), the transformation matrix MT C representing the relationship between the target reference R M and the mechanical reference R C can be derived.

[0232] The sixth step: Shift from the initial planning to the physical planning

[0233] Figure 5 The sixth step represented in C involves converting the planned actions of the initial surgical plan into physical actions performed in the target reference R O T M transformation obtained using data determined from sensors of the kinematic chain including the mechanical reference and the machining tool is combined with MT C to calculate the transformation C T O , thereby allowing the calculation of the machining tool positioning in the target reference and thus transposing the planning data into the real environment.

[0234] Then, the knowledge of the transformation C T O can be used to correct the machining tool trajectory based on the movement of the target during the intervention.

[0235] The surgical planning data transposed onto the intraoperative image can be displayed on the screen 60 in the operating room. This provides the surgeon with visual feedback on the surgical progress related to the schedule.

[0236] According to one embodiment, steps 2 and 3 are implemented by a single processing using an algorithm robust to outliers, such as Random Sample Consensus (RANSAC), without going through the subset extraction step.

[0237] Figure 8 An example of providing the determination of the anatomical structure A, which is a typical anatomical structure that is well-known to often require surgery, namely, the knee joint. As is well-known, the knee joint includes three bones, the femur F, the tibia T, and the patella. (We intentionally exclude the patella as it has no explanatory value). Thus, the example described in this specification relates to the field of orthopedic surgery and more specifically to the preparation of the femur F and the tibia T for implanting a femoral knee implant I.

[0238] The preparation according to this example comprises a series of well-known steps, each step being carried out according to a given pre-calculated machining plan P1, P2, P3, P4, P5, P6 included in the surgical plan (see Figure 8 ) for machining one of the bones F or T (traditionally, a saw is used to cut the bone). These machining steps are themselves well-known and they are generally carried out in the same order, depending on the strategy adopted by the operator (surgeon). In Figure 8 , each machining plan P1, P2, P3, P4, P5, P6 is numbered in a recognized chronological order. These machining plans P1, P2, P3, P4, P5, P6 are generally determined by a preoperative surgical plan.

[0239] The preoperative surgical plan applies only to a specific type of implant (size, design, brand, etc.) for a specific surgery on a specific patient. Each patient (and each surgery) is given a personalized preoperative surgical plan. Thus, the machining plans P1, P2, P3, P4, P5, P6 for each surgery vary slightly. The first step in a typical preoperative surgical plan is to establish a 3D digital model of the target bones F, T. One way to obtain a 3D digital bone F, T model is to use medical imaging such as computed tomography, X-ray, MRI, fluoroscopy, ultrasound or other imaging means. X-ray or scanner, or even MRI, acquisitions are usually carried out with the patient fully weight-bearing, usually a frontal (also called coronal or anteroposterior) view, a lateral (or side) view with the knee fully extended and / or flexed 20° - 30°, a long leg view including the lower limb from the femoral head to the ankle joint, and finally a view of the knee at 30° flexion, also called the skyline view. Based on these images, a digital model of the bones F, T to be machined during the surgery can be constructed. Then a specific knee implant I is selected based on the analysis of the 3D digital bone F, T model.

[0240] The present invention aims to precisely and safely machine the bones F, T by means of a surgical device comprising a kinematic chain 70 and a surgical tool 20 as shown in Figure 6 , 7 or 9.

[0241] Once the 3D digital model of the F, T bones is established, it can be stored in the memory of the control unit of the surgical device.

[0242] In one example, the surgical device may include a 3D imaging sensor 30, the position of which within the surgical device is well known. The 3D imaging sensor 30 allows the operator to cooperate with the bone F, T models stored in the memory of the control unit. Once the 3D digital model of the F, T bones of a given patient is determined and stored in the memory of the control unit (surgical device), it can be used for surgery. Once the patient is correctly positioned, the anatomical structure A to be viewed and the surgical device are correctly placed relative to the patient, at least one 3D image of the anatomical structure A is taken. This 3D image is taken using the 3D imaging sensor 30. The control unit of the surgical device may be configured to perform the steps of the method of the present invention. This enables the control unit to position the anatomical structure A relative to the 3D imaging sensor 30 and thus relative to the surgical device. Then, this enables the control unit to set precise machining plans P1, P2, P3, P4, P5, P6 for the specific surgery within the target reference R A for that particular surgery within the target reference R.

[0243] The free surfaces of the bones F, T to be machined are limited, and thus only a few areas allow the surgical tool 20 to contact the bones F, T. This contact must be as minimally invasive as possible so as not to damage either the bones F, T or the surrounding soft tissues, while ensuring the precise relative positioning of the surgical tool 20 relative to the bones F, T.

[0244] As Figure 6 、 7 shown in FIG. 9, the surgical device is intended to machine the anatomical structure A (in this case the knee) of a patient positioned on the operating table. The patient is typically anesthetized and maintained on the operating table by means of specific and well-known fixing devices. In addition, in one embodiment shown in Figure 6 and Figure 7 the entire limb of the patient is fixed to the kinematic chain 70 of the surgical device.

[0245] For example, in addition to the kinematic chain 70 and the surgical tool 20, the surgical device may include a base unit designed to be fixed to the operating table and a mechanical reference 40 designed to fix the anatomical structure A. The surgical tool 20 may be configured to be displaced by the operator.

[0246] In one embodiment, the system for computer-assisted surgery corresponds to the control unit of the surgical device. The control unit 80 can be a computer, for example. The control unit 80 may include a memory, a real-time computing element, a power supply, a power converter, a fuse, and / or an actuator. The control unit 80 may also include an operator interface 60 that allows interaction between the control unit 80 and the operator.

[0247] The operator interface 60 may be configured to

[0248] - Display the images acquired by the 3D sensor and the outputs of Steps 1 to 3.

[0249] - Display real-time information, such as the position of the surgical tool 20 relative to the anatomical structure A.

[0250] - Display the planned implant position and surgical plan to assist the operator in selecting the optimal implant and its position.

[0251] - Configure the position of the machining target of the tool holder.

[0252] The system for computer-assisted surgery of the present invention can be integrated into the surgical device as the control unit described above, or can be a processor configured to execute the steps of the method of the present invention and communicate with the surgical device by wired connection or wirelessly.

[0253] The present invention also includes a computer program product for computer-assisted surgery, which includes instructions that, when executed by a computer, cause the computer to implement the steps of the method according to any one of the embodiments described above.

[0254] The computer program product for performing the above method can be written as a computer program, code segment, instruction, or any combination thereof, for individually or jointly instructing or configuring a processor or computer to operate as a machine or a special-purpose computer to perform the operations executed by the hardware components. In one example, the computer program product includes machine code directly executed by a processor or a computer, such as the machine code generated by a compiler. In another example, the computer program product includes higher-level code executed by a processor or a computer using an interpreter. A person of ordinary skill in the art can easily write instructions or software according to the block diagrams and flowcharts shown in the drawings and the corresponding descriptions in the specification, and the specification discloses algorithms for performing the operations of the above method.

[0255] The present invention also includes a computer-readable storage medium including instructions that, when executed by a computer, cause the computer to implement the steps of the method according to any one of the embodiments described above.

[0256] According to one embodiment, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0257] The computer program for implementing the method of the present invention can generally be distributed to users on distributed computer-readable storage media such as, but not limited to, SD cards, external storage devices, microchips, flash memory devices, portable hard disks, and software websites. The computer program can be copied from the distribution medium to a hard disk or a similar intermediate storage medium.

[0258] A computer program can be run by loading computer instructions from its distribution medium or its intermediate storage medium into the execution memory of a computer, and configuring the computer to operate in accordance with the method of the present invention. All of these operations are well known to those skilled in the art of computer systems.

[0259] Instructions or software for controlling a processor or computer to implement the hardware components and execute the above method, as well as any associated data, data files, and data structures, are recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include read-only memory (R O M), random access memory (RAM), flash memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-RLTH, BD-RE, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, and any device known to those of ordinary skill in the art that can store instructions or software and any associated data, data files, and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed across network-coupled computer systems so that the instructions and software and any associated data, data files, and data structures can be stored, accessed, and executed by the processor or computer in a distributed manner.

Claims

1. A system for computer-assisted guided surgery, comprising performing a conversion of a planned action relative to a virtual reference R in a virtual environment into a physical action to be performed in a real operating room environment for an orthopedic surgery on a patient using a surgical tool (20), the surgical tool (20) being fixed to a kinematic chain (70), the kinematic chain including a sensor unit having at least one sensor configured to follow in real time the spatial configuration of the kinematic chain (70); the system P further comprising a control unit (60) configured to control the kinematic chain (70) based on the spatial configuration of the kinematic chain (70) detected by the at least one sensor, such that the surgical tool (20) performs the physical action corresponding to the planned action in the real operating room environment. Comprising: - A receiving module configured to receive at least one 3D image obtained from at least one 3D imaging sensor (30); the 3D image comprising at least a part of a target anatomical structure (10) of a patient; - A calculation module configured to: O Calculating a virtual reference R by registering a digital model of a target anatomical structure with at least a portion of the target anatomical structure (10) included in a 3D image P A transformation between the target reference R C and C T P and a capture reference R of a 3D imaging sensor A A transformation between the capture reference R C and C T A wherein the target reference R C corresponds to a physical coordinate system of the target anatomical structure (10), and the capture reference R A corresponds to a coordinate system of the 3D imaging sensor (30); O Apply a transformation C T P so as to register the digital model of the target anatomical structure in a target reference R C such that each point included in the digital model of the target anatomical structure has a known location in the target reference R C and has a known location; O Calculate the reference R of the surgical tool O and the target reference R C between the transformation C T O , the reference R of the surgical tool O corresponds to the physical coordinate system of the surgical tool (20), wherein the transformation C T O is calculated based on data obtained from the sensor unit of the kinematic chain (70) and the transformation C T A ; O Apply the transformation C T O to the reference R of the surgical tool (20) O in order to determine the position and spatial orientation of the surgical tool in the target reference R C ; Thereby determining the position and spatial orientation of the surgical tool (20) in the virtual reference R P and the target reference R C so as to reproduce, in the target reference R C the actions planned in the virtual reference R P ​ 2. The system according to claim 1, wherein, The computing module is further configured to calculate the transformation by the following C T A : - Define a region of interest in the 3D image comprising the target anatomical structure (10); - Register the region of interest including the target anatomical structure (10) to a digital model of the target anatomical structure to determine C T A 。 3. The system according to claim 2, wherein, Defining the region of interest comprises automatically detecting the region of interest by means of a segmentation algorithm.

4. The system according to claim 1, wherein, When the motion chain (70) comprises at least one mechanical reference (40) rigidly fixed to the target anatomical structure (10) and the at least one 3D image comprises at least a part of the mechanical reference (40), the calculation module is further configured to: - Receive, in real time, a data representation of the spatial configuration of the motion chain (70) from a sensor unit of the motion chain (70); - Use the received data to calculate a reference R for the surgical tool O Reference R with respect to the mechanical reference M Transformation between O T M ; - Calculating a reference R of the mechanical reference by matching a digital model of the mechanical reference with at least a part of the mechanical reference included in the 3D image M And acquiring the reference R A The transformation between M T A ; such that C T O the transformation is from the acquisition reference R A , the mechanical datum reference R M and the target reference R C between the transformations O T M , M T A and C T A is obtained from the combination of 5. The system according to claim 4, further comprising a correction module configured to use a sensor unit of the motion chain (70) to track the movement of the target anatomical structure (10) rigidly fixed to at least one mechanical reference (40) relative to a surgical tool (20), such that whenever a deviation in the position and / or spatial orientation of the target anatomical structure is detected, the conversion of a planned action from a virtual environment to a real environment is corrected for the deviation.

6. The system according to claim 1, wherein, The at least one 3D imaging sensor (30) is fixed to a kinematic chain (70), and the calculation module is further configured to calculate a reference R of the surgical tool based on data obtained from a sensor unit of the kinematic chain (70). O The acquisition reference R of the 3D imaging sensor A The transformation between A T O such that C T O The transformation is from the acquisition reference R A The transformation between the acquisition reference R and the target reference R C The transformation between C T A and the transformation A T O is obtained from the combination of.

7. The system according to claim 1, wherein, The motion chain (70) is constituted by a deformable structure comprising a plurality of rigid elements connected by joints.

8. The system according to claim 1, wherein, The motion chain (70) further comprises sensors for measuring the forces applied to its elements.

9. The system according to claim 1, wherein, Receiving the acquired 3D image by the receiving module is performed using at least two sensors and a projector to perform the acquisition by stereovision or structured light.

10. The system according to claim 6, wherein, The 3D imaging sensor (30) fixed to the motion chain (70) moves along a known trajectory and a plurality of 3D images are acquired along the trajectory, and the calculation module is further configured to jointly process the plurality of 3D images acquired along the trajectory to register the plurality of 3D images with a digital model of the target anatomical structure.

11. The system according to claim 6, further comprising a correction module configured to use the 3D imaging sensor (30) and a visual tracking algorithm to track the movement of the target anatomical structure (10) relative to the surgical tool (20), such that whenever a deviation in the position and / or spatial orientation of the target anatomical structure is detected, the conversion of a planned action from a virtual environment to a real environment is corrected for the deviation.

12. The system according to claim 1, wherein, The receiving module is further configured to receive thermal images, ultrasound images, multispectral images, images at the microscale and / or monocular color images.

13. The system according to claim 1, wherein, The three-dimensional digital model of the target anatomical structure is generated based on computed tomography images or MRI images.

14. The system according to claim 1, wherein, the three-dimensional digital model of the target anatomical structure is generated using 2D X-ray photographs, the 2D X-ray photographs including the target anatomical structure, a statistical shape model of the target anatomical structure, and / or 3D images acquired during surgery by a 3D imaging sensor.

15. The system according to claim 1, wherein, the three-dimensional digital model of the target anatomical structure is digitally modified to simulate measurement noise or the presence of cartilage, the modification being calculated based on training data or biomechanical simulation data.

16. The system according to claim 1, wherein, the registration of the digital model of the target anatomical structure with at least a portion of the target anatomical structure (10) included in the 3D image is a non-rigid transformation.

17. A computer-implemented method for guiding a surgical tool (20) in a real operating room environment suitable for orthopedic surgery on a patient, the surgical tool (20) being fixed to a kinematic chain (70), the kinematic chain including a sensor unit having at least one sensor configured to follow in real time the spatial configuration of the kinematic chain (70); the method comprises the following steps: - receiving at least one 3D image acquired from at least one 3D imaging sensor (30); the 3D image including at least a portion of a target anatomical structure of the patient; - Calculating a virtual reference R by registering a digital model of a target anatomical structure with at least a portion of the target anatomical structure included in a 3D image P with the target reference R C a transformation between C T P and a capture reference R of a 3D imaging sensor A with the target reference R C a transformation between C T A wherein the target reference R C corresponds to the physical coordinate system of the target anatomical structure (10), and the capture reference R A corresponds to the coordinate system of the 3D imaging sensor (30); - Apply a transformation C T P so as to register the digital model of the target anatomical structure in a target reference R C such that each point included in the digital model of the target anatomical structure has a known location in the target reference R C and has a known location; -Calculate the reference R of the surgical tool O and the target reference R C between the transformation C T O , the reference R of the surgical tool O corresponds to the physical coordinate system of the surgical tool (20), wherein the transformation C T O is calculated based on the data obtained from the sensor unit of the kinematic chain (70) and the transformation C T A ; - Apply the transformation C T O to the reference R of the surgical tool O to determine the position and spatial orientation of the surgical tool (20) in the target reference R C ; Thereby determining the position and spatial orientation of the surgical tool (20) in the virtual reference R P and the target reference R C so as to reproduce in the target reference R C the actions planned in the virtual reference R P ​ 18. The method according to claim 17, wherein, The said transformation C T A is calculated through the following steps: - defining a region of interest in the 3D image including the target anatomical structure; - Register the region of interest including the target anatomical structure to the digital model of the target anatomical structure to determine C T A 。 19. The method according to claim 18, wherein, the step of defining the region of interest includes automatically detecting the region of interest by means of a segmentation algorithm.

20. The method according to claim 17, wherein, the kinematic chain (70) includes at least one mechanical reference (40) rigidly fixed to the target anatomical structure (10), and the at least one 3D image includes at least a portion of the mechanical reference (40); the method further comprises the following steps: - receiving in real time from the sensor unit of the kinematic chain (70) a data representation of the spatial configuration of the kinematic chain (70); - Use the received data to calculate a reference R for the surgical tool O Reference R with respect to the mechanical reference M Transformation between O T M ; - Calculating a reference R of the mechanical reference by matching a digital model of the mechanical reference with at least a part of the mechanical reference included in the 3D image M With the acquisition reference R A The transformation between M T A ; such that C T O the transformation is from the acquisition reference R A , the mechanical datum reference R M and the target reference R C between the transformations O T M , M T A and C T A is obtained from the combination of 21. The method according to claim 20, wherein, the movement of the target anatomical structure (10) relative to the surgical tool (20) is tracked by the sensor unit of the kinematic chain (70) such that whenever a deviation in the position and / or spatial orientation of the target anatomical structure is detected, the transformation of the planned action from the virtual environment to the real environment is corrected for the deviation.

22. The method according to claim 17, wherein, The at least one 3D imaging sensor (30) is fixed to a kinematic chain (70), and the method further comprises the step of calculating a reference R of the surgical tool based on data obtained from a sensor unit of the kinematic chain (70). O The acquisition reference R of the 3D imaging sensor A The transformation between A T O such that C T O The transformation is from the acquisition reference R A To the target reference R C The transformation between C T A And the transformation A T O Is obtained from the combination of 23. The method according to claim 17, wherein, the kinematic chain (70) is constituted by a deformable structure including a plurality of rigid elements connected by joints.

24. The method according to claim 17, wherein, the kinematic chain (70) further includes a sensor for measuring the forces applied to its elements.

25. The method according to claim 17, wherein, Receiving the acquired 3D image by the receiving module is performed using at least two sensors and a projector to perform the acquisition by stereovision or structured light.

26. The method according to claim 22, wherein, the 3D imaging sensor (30) fixed on the kinematic chain (70) moves along a known trajectory and acquires a plurality of 3D images along the trajectory, and the method further comprises the steps of: jointly processing the plurality of 3D images acquired along the trajectory to register with the digital model of the target anatomical structure using the plurality of 3D images.

27. The method according to claim 22, wherein, the movement of the target anatomical structure (10) relative to the surgical tool (20) is tracked by the 3D imaging sensor (30) and a visual tracking algorithm, so that whenever a deviation in the position and / or spatial orientation of the target anatomical structure is detected, the transformation of the planned action from the virtual environment to the real environment is corrected for the deviation.

28. The method according to claim 17, further comprising receiving a thermal image, an ultrasonic image, a multi-spectral image, an image at a microscopic scale, and / or a monocular color image.

29. The method according to claim 17, wherein, the three-dimensional digital model of the target anatomical structure is generated based on a computed tomography image or an MRI image.

30. The method according to claim 17, wherein, the three-dimensional digital model of the target anatomical structure is generated using a 2D X-ray photograph, and the 2D X-ray photograph includes the target anatomical structure, a statistical shape model of the target anatomical structure, and / or a 3D image acquired by the 3D imaging sensor during the surgery.

31. The method according to claim 17, wherein, the three-dimensional digital model of the target anatomical structure is digitally modified to simulate measurement noise or the presence of cartilage, and the modification is calculated based on training data or biomechanical simulation data.

32. The method according to claim 17, wherein, the matching of the digital model of the target anatomical structure with at least a part of the target anatomical structure included in the 3D image is a non-rigid transformation.

33. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method according to claim 17.

34. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method according to claim 17.

Images