COMPUTER-ASSISTED METHOD FOR MODELING A PATIENT'S JOINT

Abstract

COMPUTER-AIDED METHOD FOR MODELING A PATIENT'S JOINT The invention relates to a computer-based method for modeling a patient's joint to provide real-time assistance in creating at least one bone tunnel by arthroscopy in said joint. Figure from the abstract: Fig. 2

Description

Title of the invention: COMPUTER-IMPLEMENTED METHOD FOR MODELING A PATIENT'S JOINT FIELD OF THE INVENTION

[0001] The present invention relates to assistance in arthroscopy surgery. More particularly, the invention relates to a computer-implemented method for modeling a patient's joint. In particular, the method described herein is of interest in assisting in the creation of bone tunnels in a patient's joint during an intervention. STATE OF THE ART

[0002] Digital surgical assistance can be useful and advantageous during surgical procedures such as arthroscopic surgery. Indeed, when a surgeon has to operate on a part of an individual's body, he does not have complete visibility of the interior of this part of the body. Thus, a surgeon's actions could be more precise if the latter could visualize in real time the positioning of his tools in the vicinity and inside the part of the body on which he has to operate.

[0003] Knee ligament reconstruction surgeries are procedures that consist of replacing one or more ligaments of the knee joint, connecting the femur to the tibia with a strip of tendon tissue, which will constitute the neoligament, taken from the patient (autograft), or more rarely, taken post mortem from donors (allograft). The surgeon will then create a bone tunnel in the femur and a bone tunnel in the tibia and position these tunnels in the insertions of the damaged ligament, then pass the graft through these tunnels before fixing it.

[0004] Some solutions have been developed for clinical research purposes, but none have yet entered routine use in the operating room.

[0005] The invention falls within this context. SUMMARY

[0006] The invention relates to a computer-implemented method for modeling a patient's joint for real-time assistance in creating at least one bone tunnel by arthroscopy in said joint, comprising: - a step of obtaining, using an imaging device, a stream of monocular intraoperative two-dimensional images, each of the monocular intraoperative two-dimensional images being obtained at a current time t and comprising at least one portion of interest of the joint; - a first step of calculation, for each monocular intraoperative two-dimensional image, of an associated depth map, by implementing a previously trained learning model; - a second step of calculating a current partial three-dimensional model of the patient's joint at the current time t, on the basis of at least one monocular intraoperative two-dimensional image obtained at and / or before the current time t, at least one depth map calculated at and / or before the time t, information relating to a current position and orientation of the imaging device, and a complete preoperative three-dimensional model of the patient's joint.

[0007] The method thus makes it possible to guide a healthcare professional during an intervention by presenting a model of the patient's anatomy, allowing him to access a calculated partial three-dimensional model of real images. This model will allow the healthcare professional to guide him in the positioning and drilling of bone tunnels. The combination of at least one monocular intraoperative two-dimensional image, at least one associated depth map and information relating to a current position and orientation of the imaging device, associated with a reconstruction algorithm as described in the present application, makes it possible to generate a current partial three-dimensional model of the patient's joint at a time t.

[0008] One of the particularities of the invention is that it does not use a depth sensor, such as Lidar, nor an RGB-D camera capable of directly acquiring a depth map, nor a stereoscopic acquisition system. Here, the two-dimensional images used are monocular images. The invention makes it possible to predict the joint depth observed with an imaging device during an intervention in a reliable and precise manner, in accordance with the actual geometry of the patient.

[0009] Advantageously, the previously trained learning model is an artificial neural network.

[0010] Advantageously, the second calculation step is carried out with a SLAM (simultaneous localization and mapping) type algorithm. The algorithm advantageously uses the stream of monocular intraoperative two-dimensional images delivered by the imaging device as well as the depth maps calculated with the previously trained learning model to calculate a three-dimensional model in real time whose position and orientation relative to the imaging device are known.

[0011] Advantageously, the second calculation step is carried out using a reinforcement learning model. Such a method makes it possible to obtain a precise partial three-dimensional model quickly.

[0012] Advantageously, the learning model has been previously trained with a set of monocular intraoperative two-dimensional training images and a set of associated ground truths, said ground truths being distances, being distances between a virtual imaging device and at least one object included in the set of monocular intraoperative two-dimensional training images and said training images being associated with virtual images from a preoperative three-dimensional model of the patient's joint. To the inventors' knowledge, there is no training database composed of images similar to arthroscopic images. Thus, the learning model is trained specifically on images close to the intraoperative images, and is thus more accurate.

[0013] Advantageously, the preoperative three-dimensional model was obtained from two-dimensional medical images.

[0014] Advantageously, the learning model has been previously trained with a set of monocular intraoperative two-dimensional training images and a set of associated ground truths, said ground truths being distances obtained using pairs of images acquired at different times using an imaging device called a training imaging device and information relating to positions and orientations of said training imaging device. Thus, the learning model is trained specifically on images close to the intraoperative images, and is thus more accurate.

[0015] It is possible to chain the model training phases with training data sets very close in content and quality to the data that the latter will process, and thus increase the accuracy of the model.

[0016] Advantageously, the learning model has been previously trained with a set of monocular intraoperative two-dimensional training images and a set of associated ground truths, said ground truths being distances measured on cadaveric joints, and said training images being two-dimensional images of said cadaveric joints. Thus, the learning model is trained specifically on images close to the intraoperative images, and is thus more accurate.

[0017] Advantageously, the method comprises, between the first calculation step and the second calculation step, an intermediate step of filtering outlier data in each calculated depth map. This step makes it possible to improve the accuracy of the calculation of the current partial three-dimensional model.

[0018] Another aspect of the invention relates to a computing device comprising a circuit configured to implement a method as described above.

[0019] Another aspect of the invention relates to a computer program product comprising instructions for implementing steps of the method described above.

[0020] Yet another aspect of the invention relates to a non-transitory computer-readable recording medium comprising instructions which when executed by a computer, cause the computer to carry out steps of the method described above. DESCRIPTION OF FIGURES

[0021] [Fig.l] depicts a plurality of elements used when performing a method for modeling a patient's joint according to one or more embodiments.

[0022] [Fig.2] represents a flowchart of an example of steps for the realization of a process for modeling a patient's joint.

[0023] [Fig.3] represents a flowchart of an example of a succession of steps for training a learning model used to calculate depth maps.

[0024] [Fig.4] illustrates a device for implementing methods for modeling a articulation of a patient according to one or more embodiments. DEFINITION

[0025] The term "processor" should not be interpreted as being limited to hardware capable of executing software, but generally refers to a processing device, which may, for example, include a computer, a microprocessor, an integrated circuit, or a programmable logic device (PLD). The processor may also include one or more graphics processing units (GPUs), whether used for computer graphics and image processing or for other functions. In addition, the instructions and / or data for executing the associated and / or resulting functionalities may be stored on any medium readable by the processor, such as, for example, an integrated circuit, a hard disk drive, a CD (Compact Disc), an optical disk such as a DVD (Digital Versatile Disc), a RAM (Random-Access Memory), or a ROM (Read-Only Memory).Instructions may be stored in hardware, software, firmware, or any combination thereof. DETAILED DESCRIPTION

[0026] The present invention relates to a computer-implemented method 100, illustrated in [Fig. 2], for modeling a joint of a patient. In the present description, the joint is one of the patient's knees.

[0027] We are here in the situation where the patient must undergo a surgical intervention, by arthroscopy, for ligament reconstruction of his knee.

[0028] Advantageously, the method 100 makes it possible to obtain an intraoperative three-dimensional model of the joint, which will be partially visible during the intervention. surgical. Thanks to this intraoperative three-dimensional model, it is possible to guide the surgeon performing the surgical intervention, for example, when creating bone tunnels.

[0029] [Fig.l] illustrates a plurality of elements used in one or more embodiments of the invention.

[0030] During an arthroscopic intervention on the patient's joint, an imaging device 1 acquires a preferably continuous stream of two-dimensional images of all or part of the patient's joint.

[0031] For example, the imaging device 1 is an arthroscope. An arthroscope is a thin tube equipped with a miniaturized lens connected to a camera for viewing the inside of a joint. The arthroscope is inserted into the joint after incision. The diameter of the arthroscope is of the order of a few millimeters.

[0032] Furthermore, during an arthroscopic intervention to create one or more bone tunnels, a surgeon uses a surgical drilling instrument 5 such as a surgical motor, comprising a drilling end. The drilling end makes it possible to create tunnels in the bone structures.

[0033] During the intervention, a tracking device 3 dedicated to tracking different elements used during the intervention is used. The tracking device 3 comprises for example one or more cameras making it possible to locate and track in real time the position in space of the different instruments used during the intervention, such as the imaging device 1 and the surgical drilling instrument 5. On the imaging device 1 and the surgical drilling instrument 5 are fixed one or more markers configured to emit, receive, or reflect electromagnetic radiation, so as to be located by the tracking device 3. Advantageously, one or more passive markers can be fixed on the imaging device 1 and the surgical drilling instrument 5, so as to be visible on images acquired by the tracking device. For example, the passive markers are QR codes, AR tags or 3D markers.

[0034] The position in space of the imaging device 1 and the drilling instrument 5 relative to the tracking device 3 can then be known by triangulation.

[0035] Advantageously, if the geometry of each tracked instrument such as the imaging device 1 and the surgical drilling instrument 5 is known, the tracking device 3 directly recognizes them without providing them with markers and their respective position and orientation relative to the tracking device is known.

[0036] For example, the tracking device 3 is a Hololens™ type system for locating, using cameras, among other things, the imaging device 1 and the surgical drilling instrument 5 in an operating room. Thus, as soon as the imaging device 1 or the surgical drilling instrument 5 are in the field of vision of the cameras, these are located and their positions are calculated and recorded.

[0037] For example, during the intervention, a programmable device 6 receives in real time a signal from the tracking device 3, on the basis of which it calculates the position and orientation in real time of the imaging device 2 relative to a fixed reference frame R. The position and orientation of the imaging device 2 comprise for example three coordinates and three angles.

[0038] Also, still during the intervention, the programmable device 6 receives in real time a signal from the tracking device 3, on the basis of which it calculates the position and orientation in real time of the piercing end of the surgical piercing instrument 5. The position and orientation of the piercing end of the surgical piercing instrument 5 comprise for example three coordinates and three angles.

[0039] As will be explained, the method 100 advantageously uses the flow of two-dimensional images acquired by the imaging device 1 during the surgical intervention.

[0040] [Fig.2] is an example of a flowchart representing steps of a method for modeling a joint of a patient according to one or more embodiments.

[0041] In a step E1, a stream of monocular intraoperative two-dimensional images is obtained in real time from the imaging device 1. The monocular intraoperative two-dimensional images comprise a portion of interest of the joint.

[0042] Optionally, the monocular intraoperative two-dimensional image stream is preprocessed before a first calculation step E2. In the case where the imaging device is an arthroscope, it typically has several video stream qualities (HD, Full-HD, 4K, 8K) clocked at 60Hz. The optional preprocessing of the monocular intraoperative two-dimensional image stream then consists of resizing the latter into an HD stream so as to adapt the resolution of the stream to the learning model used. Furthermore, temporal subsampling of the images can be carried out in order to allow processing of the video stream in real time. For example, sampling between one image in 3 and one image in 10 can be used.

[0043] During the first calculation step E2, for each monocular intraoperative two-dimensional image obtained at a current time t, a depth map associated with the monocular intraoperative two-dimensional image is calculated, by implementing a previously trained learning model. By depth map associated with an image comprising a plurality of pixels, it is meant an image comprising the same number of pixels as the plurality of pixels, where each pixel has the value distance of the object contained in the corresponding pixel in the image associated with the imaging device.

[0044] In a second calculation step E3, a current partial three-dimensional model 8 of the patient's joint is calculated at time t. The calculation uses at least one monocular intraoperative two-dimensional image obtained at and / or before the current time t, at least one depth map calculated at and / or before time t, information relating to a current position and orientation of the imaging device 1, and a complete preoperative three-dimensional model of the patient's joint.

[0045] Once the current partial three-dimensional model 8 has been calculated, it can be displayed on a viewing device 12 and viewed by the healthcare professional performing the surgical intervention.

[0046] The first calculation step E2 will now be described in more detail.

[0047] It should be noted that images of joints captured using an arthroscope have the particularity of presenting few characteristic points that can facilitate the operation of learning models such as artificial neural networks. Training an artificial neural network so that the latter produces a depth map associated with an arthroscopic image can prove difficult if the choice of training data is not adequate. Arthroscopic images can indeed be blurred, overexposed, or present a limited texture, or even present obstructions. Furthermore, the imaging distances during navigation of an arthroscope can vary over large ranges of values. All these conditions make the training of a learning model very difficult.

[0048] One of the advantages of the invention lies in the use of training data very close in content and quality to the data processed by the learning model, which will make the learning model more accurate after training when it calculates depth maps. The accuracy of the depth maps is important because the latter will be used in combination with the monocular intraoperative two-dimensional images in order to calculate the partial three-dimensional model at each current time.

[0049] In one or more embodiments, the pre-trained learning model used to calculate the depth maps associated with the monocular intraoperative two-dimensional images is an artificial neural network.

[0050] For example, an artificial neural network with a monodepth2 architecture, or of the Unet, PyDNet, DSNet, FastDepth, DeepMatchVO, Ganvo, MiDas type can be used.

[0051] Prior to the implementation of the method 100, the artificial neural network will be trained from training data, during a training phase 200 illustrated in [Fig.3].

[0052] The training phase 200 comprises three training sub-phases E201, E202 and E203. These three sub-phases can be executed in any order. Alternatively, the training phase 200 can comprise only one or two training sub-phases among the sub-phases E201, E202 and E203.

[0053] During a first training sub-phase E201, the artificial neural network is trained on a database Bi containing a first set of monocular intraoperative two-dimensional training images, including the patient's joint, and a set of associated ground truths. A complete three-dimensional model of the patient's joint, described below, obtained preoperatively from MRI images including the joint.

[0054] For example, the preoperative three-dimensional model is obtained from data obtained by magnetic resonance imaging (MRI), prior to the intervention. The data comprises different slices of the patient's joint. By "slice" is meant a two-dimensional image. The different slices are slices in planes parallel respectively to the sagittal, axial and coronal planes as defined in the field of magnetic resonance imaging. The MRI technique can be two-dimensional, i.e. based on sequential acquisition triplets, or three-dimensional, i.e. based on volume acquisitions. Alternatively, other techniques allowing the acquisition of several slices of areas of interest can be used, such as for example computational axial tomography.

[0055] The magnetic resonance imaging technique has the advantage of providing data relating to anatomical structures not accessible by techniques such as radiography. Thus, the preoperative three-dimensional model may comprise, for example, in addition to a representation of the bony contours of the femur, the tibia, the patella and the head of the fibula, a representation of the cartilaginous contours of the femur, the tibia, a complete or partial representation of the anterior cruciate ligament, the posterior cruciate ligament, the external lateral ligament, the internal lateral ligament, the patellofemoral ligament.

[0056] The Bi database is generated using 3D imaging software. For example, Blender software can be used. The preoperative three-dimensional model is opened using the 3D imaging software. For each image of the images of the first set of monocular intraoperative two-dimensional training images, a virtual imaging device integrated into the 3D imaging software and reproducing the same optical properties as a so-called training imaging system is positioned in order to obtain a view of the preoperative three-dimensional model superimposable on the image in question. For the corresponding position of the virtual imaging system, it is then possible to calculate an associated depth map. The The training imaging device is identical to imaging device 1, which will be used intraoperatively. Thus, the principle of generating the Bi database is based on the association of the preoperative three-dimensional model, and therefore a depth map, with each intraoperative two-dimensional image of the first set.

[0057] More precisely, a monocular intraoperative two-dimensional training image of the first set is associated with a ground truth representing a distance map corresponding to the distances between the imaging device 1 and each pixel of the intraoperative arthroscopic image, assuming that these distances are equivalent to the distances between the virtual imaging device and each pixel of the preoperative three-dimensional model filmed virtually using the 3D imaging software in the same viewing angle as the monocular intraoperative two-dimensional image and superimposable thereon. These manipulations make it possible to generate the database Bi making it possible to carry out the training sub-phase 201.

[0058] Following the first training sub-phase 201, a second training sub-phase 202 is carried out on a second database B2 containing a second set of monocular intraoperative two-dimensional training images and a second set of associated ground truths.

[0059] The second database B2 is generated by acquiring intraoperatively using a training imaging device, similar to the imaging device 1, images at two different times t and ti and with different viewpoints. The training imaging device is tracked during these acquisitions using a training tracking device, similar to the tracking device 3, so as to know the position in space of the training imaging device at time t and at time tb. The training imaging device can be coincident with the imaging device 1. The training tracking device can be coincident with the tracking device 3. Following the acquisitions, a “stereo multiview” algorithm is implemented so as to make it possible to locate the same notable points on the image at time t and the image at time ti.Thus, the algorithm locates a remarkable point P; on the image at time t and the same remarkable point P; on the image at time tb. Knowing the positions in space of the training imaging device at times t and tb, the distance between the training imaging device and each remarkable point P; can be calculated. This Multiview Stereo method thus makes it possible to generate the second database B2 for the training sub-phase 202.

[0060] Following the second training sub-phase 202, a third training sub-phase 203 is carried out on a third database B3 containing a third set of monocular intraoperative two-dimensional training images and a third set of associated ground truths. The images of the third set are two-dimensional images of cadaveric joints, called images cadaveric images, acquired with an imaging device called a training device similar to the imaging device 1 used during the intervention.

[0061] The ground truths of the third database B3 are distance maps corresponding to the distances between the training device and the points of the cadaveric image captured by the training device. The acquisition of the ground truths is carried out in several stages, using a training tracking device, and markers positioned on various instruments but also on various anatomical structures. The tracking device can advantageously be a system of binocular infrared cameras of the NDI camera type or several calibrated infrared cameras of the Optitrack camera type. The markers can advantageously be produced using rigid bodies equipped with at least 3 reflective spheres. These systems are typically used in computer-assisted surgery and make it possible to obtain high-performance tracking that can offer a positioning accuracy of less than 0.5 mm.

[0062] More precisely, in a first step a), a first marker is placed and fixed in the femoral bone (femur) of the cadaveric joint, and a second marker is placed and fixed in the tibial bone (tibia). The first marker and the second marker will be used to know in real time the position in space of the femur and the tibia.

[0063] In a second step b), a marker is placed and fixed on the training imaging device, which will be used to know in real time the position in space of the training imaging device.

[0064] In a third step c), a marker is placed and fixed on a probe, which will be used to know in real time the position in space of the latter.

[0065] In a fourth step d), points on the femur and on the tibia are palpated with the probe. The position of the probe when it is in contact with the femur is recorded in real time in order to record a point cloud located on the surface of the femur. All of the palpated points constitute a point cloud for which the distance between the palpated point and the training imaging device is known thanks to the training tracking device and the markers. Similarly, a point cloud corresponding to the surface of the tibia is generated, for which the distance between the palpated point and the training imaging device is known.

[0066] For example, the point cloud corresponding to the femur is generated, the position of the points in space relative to the femoral marker being known. Similarly, the point cloud corresponding to the femur is generated, the position of the points in space relative to the tibial marker being known. Then, the scene is filmed using the training tracking device in the absence of the probe. Knowing the position of the training imaging device relative to the femoral marker for the femur, and knowing the position of the point cloud relative to the femoral marker, we know the distance between the training imaging device and the point cloud.

[0067] Thus, during an intervention, when a monocular intraoperative two-dimensional image is received from the imaging device at a current time t, the artificial neural network, previously trained during step 200, produces a corresponding depth map as output.

[0068] Optionally, the depth map associated with a monocular intraoperative two-dimensional image obtained in step E2 may be pre-processed. Indeed, the latter may contain aberrant data. Thus, during an intermediate step following the first calculation step E2 but preceding the second calculation step E3, filtering may be performed on the depth map obtained in step E2. Algorithms such as the RANSAC, Branch&Bound algorithms, learning algorithms via clustering, may be used.

[0069] The second calculation step E3 will now be described in more detail.

[0070] During the second calculation step E3, the current partial three-dimensional model 8 of the patient's joint at time t is calculated. The calculation uses at least one monocular intraoperative two-dimensional image obtained at and / or before the current time t, at least one depth map calculated at and / or before time t, information relating to a current position and orientation of the imaging device, and a complete preoperative three-dimensional model of the patient's joint.

[0071] For example, the calculation may use the set of monocular intraoperative two-dimensional images acquired by the imaging device 1 over a period [tl, t], with tl a time preceding the current time t and the set of corresponding depth maps. In yet another example, a sub-sampled set of monocular intraoperative two-dimensional images acquired by the imaging device 1 over a period [tl, t] may be used for the calculation. For example, one monocular intraoperative two-dimensional image out of N may be used, with N an integer between 2 and 30. N may vary depending on the movement speed of the imaging device 1 in order to obtain the best compromise between calculation accuracy and processing time. Furthermore, N may also vary depending on an amount of movement observed in the stream of monocular intraoperative two-dimensional images.

[0072] Advantageously, a SLAM (Simultaneous Localization and Mapping) type algorithm is used to calculate the current partial three-dimensional model 8. SLAM type algorithms allow, from the use of two-dimensional images, acquired by an image sensor, and the position in space of the image sensor, to produce, through depth map predictions associated with the two-dimensional images, a three-dimensional mapping of the scene observed in the two-dimensional images. The three-dimensional mapping obtained can then be located in a reference frame associated with the image sensor that acquires the two-dimensional images. Furthermore, this three-dimensional mapping is dynamic. In other words, it can be obtained in real time, as the image sensor acquires the two-dimensional images of the scene. A description of SLAM-type algorithms can be found in the article “Visual SLAM algorithm: a survey from 2010 to 2016”, T. Taketomi et al., IPSJ Transactions on Computer Vision and Applications (2017).

[0073] Advantageously, a combination of a Structure from Motion (SfM) type algorithm and a SLAM type architecture, based on graph theory, can be used.

[0074] In this case, the SfM (Structure From Motion) type algorithm allows the generation of a three-dimensional model of the observed joint from two-dimensional images captured at different times, and thus in also different positions and orientations. For example, a Multi View Stereo (MVS) type architecture can be used.

[0075] The SLAM type architecture makes it possible, by reverse mechanism, to predict the position and orientation of the camera from the analysis of two-dimensional images comprising the articulation at different times. An architecture chosen from DefSLAM, ORBSLAM, EKF-SLAM can for example be used.

[0076] Then, a graph matching solution can be used through the use of feature points as nodes to enable the construction of a Graph Neural Network (GNN) type neural network. For example, a Bayesian model or a Kalmann model as defined by graph theory can be used. The use of self-attention and cross-attention layers via transformers allows the exchange of global visual and geometric messages between nodes. A description of the transformer concept can be found in the article “DeepMatcher: A Deep Transformer-based Network for Robust and Accurate Local Feature Matching”, by Tao Xie et al.Despite the possible displacement of the observed anatomy over time, the exploitation of known extrinsic data, such as the position of the camera and the preoperative three-dimensional model of the patient's joint, allows a reduction of the quadratic complexity and a convergence of the system for the establishment of a partial three-dimensional model of the observed joint.

[0077] Other architectures that can implement a SLAM-type algorithm can be used, including LSTM networks, “Non Rigid Structure” architectures from Motion”, Shape from Template architectures, DefSLAM, ORBSLAM, MISSLAM, EKF-SLAM architectures, statistical models (graph theory type, Bayesian model, Kalmann model).

[0078] In other embodiments, alternatively to using a SLAM type algorithm, a reinforcement learning method may be used to calculate the current partial three-dimensional model 8 of the patient's joint at the current time t.

[0079] Reinforcement learning consists, for an autonomous agent, of the robot type, in learning the actions to take, from experiences, so as to optimize a quantitative reward over time. The agent is immersed in an environment and makes its decisions according to its current state. Here, the actions to take designate the creation of a correspondence between a depth map obtained in step E2 and a scene observed by the agent. The scene observed by the agent represents the environment and is a perspective view simulating the view captured by a virtual camera. The agent is trained to manipulate the complete preoperative three-dimensional model of the patient's joint so as to match a monocular intraoperative two-dimensional image obtained at a current time t and the perspective view simulating the view captured by the virtual camera.The manipulation of the preoperative three-dimensional model is performed by means of six degrees of freedom, i.e., three translation data and three rotation data of the model. At the end of the training of the reinforcement learning model, we obtain an agent capable of manipulating the preoperative three-dimensional model (by a series of translation and rotation actions) and of estimating its three-dimensional position relative to the position of the virtual camera.

[0080] Advantageously, the training of the reinforcement learning model is carried out with a nested policy of the Nested Policy Fitted A-Iteration type. The reinforcement learning model receives as input the stream of monocular intraoperative two-dimensional images, and the at least one depth map obtained in step E2. The constraints taken into account are then the position of the imaging device included in the information relating to the position and the current orientation of the imaging device and the preoperative three-dimensional model.

[0081] The environment is thus a field of view of the preoperative three-dimensional model captured by the virtual camera. The position and orientation of the virtual camera are fixed. With the reinforcement learning model, we seek to optimize a depth map corresponding to the field of view of the environment and of the size of the monocular intraoperative two-dimensional images, with the target being at least one depth map obtained in step E2.

[0082] Advantageously, during the intervention, the imaging device 1 is positioned ini initially at a characteristic position to optimize calculations with the reinforcement learning model and reduce computation time. An example of a characteristic position is to visualize the femoral trochlea and the foot of the anterolateral cruciate ligament.

[0083] For example, the training of the reinforcement learning model is carried out via a training policy called Clipped Double Q-Learning (CDQ), aiming to jointly minimize a first and a second cost function.

[0084] Thus, for example, the first cost function is representative of the distance between the depth of the pixels of the depth map that one seeks to optimize and the depth of the at least one depth map obtained in step E2. For example, the second cost function is representative of the costs imputed by the nested policy, in particular constraints linked to the intraoperative three-dimensional model and to the movement of the imaging device 1. The constraints linked to the intraoperative three-dimensional model may be linked to the intra-articular nature of the intervention, which limit the actions of the agent in space. The actions of the agent (translations and rotations of the preoperative three-dimensional model) may also be limited by the actual movement of the imaging device 1.Indeed, the information delivered by the tracking device 3 can be used to make assumptions about the correctness or incorrectness of the actions performed by the agent (for example, moving too far, taking the wrong direction, or turning the wrong way). A Markov decision process can be used for this purpose.

[0085] The training of the reinforcement learning model can be carried out in several stages. In a first training stage, an arthroscopic flow from a surgical intervention of a patient whose preoperative three-dimensional model of the joint of interest, called the training model, is known from MRI images, can be used. The arthroscopic flow is successively read and then stopped on a given image. A prediction is then made by the agent. An operator then manually positions the training model viewed by a virtual camera, so as to re-align the field of view of the virtual camera on the given image on which the arthroscopic flow was stopped. By backpropagation, the weights and biases of the reinforcement learning model are adjusted so as to take into account the result of the re-alignment by the operator. This succession of sub-stages is repeated as many times as the arthroscopic flow is read and stopped.In a subsequent training step, the ground truths (which are depth maps) from the third database B3 mentioned above can be used. In this subsequent training step, an algorithm automatically realigns the three-dimensional training model to structures identified in the ground truths of the third database B3.

[0086] Models other than the examples presented above can be used, such as DDPG, PPO, TRPO, ACHTR, SAC, DQ-AC, DDQN-AC, DDPG (Deep Deterministic Policy Gradient Algorithm), Random Compression Rehearsal solutions.

[0087] When an action is performed by the agent, the scene, in other words, the environment, is updated, as is the current partial three-dimensional model 8 of the patient's articulation of each action iteration.

[0088] Thus, from monocular intraoperative two-dimensional images containing the patient's joint and obtained up to time t, and from the associated depth maps calculated in step E2, a current partial three-dimensional model 8 of the patient's joint is obtained. This model is located in a reference system associated with the imaging device 1, which acquires the monocular intraoperative two-dimensional images.

[0089] Once the current partial three-dimensional model 8 has been calculated, the latter is recalibrated on a preoperative three-dimensional model of the patient's joint. For example, this is a preoperative three-dimensional model obtained from preoperative images of the patient's joint.

[0090] Advantageously, the preoperative three-dimensional model of the patient's joint contains theoretical positions of the bone tunnel(s) to be made in the patient's joint. These theoretical positions have, for example, been pre-planned during the planning of the surgical intervention. Thus, following the recalibration, the current partial three-dimensional model can be superimposed on these theoretical positions. Furthermore, the preoperative three-dimensional model can contain a representation of anatomical structures.

[0091] Advantageously, the surgeon performing the surgical intervention can visualize on a visualization device 12 on the one hand the superposition of the current partial three-dimensional model at a time t with the theoretical positions of the bone tunnel(s) to be made in the joint and with the representation of anatomical structures of interest, and at the same time, the position of this superposition relative to the position of the imaging device and that of the surgical drilling instrument, or more precisely of the end of the surgical drilling instrument. In this way, the surgeon can modify the position thereof so as, on the one hand, to be in agreement with the theoretical positions of the bone tunnel(s) to be made, and on the other hand, to manipulate it with full knowledge of its location relative to anatomical structures which could be areas to be avoided or preserved.

[0092] The calculation of the current partial three-dimensional model 8 is carried out in real time. Thus, the latter can also be displayed in real time on the viewing device 12, so as to allow immediate feedback to the surgeon.

[0093] [Fig.4] illustrates a device 300 for implementing methods for modeling a patient's joint for real-time assistance in creating at least one bone tunnel by arthroscopy according to one or more embodiments.

[0094] The device 300 may comprise a memory 301 for storing instructions for implementing steps of the methods for modeling a patient's joint to carry out all or part of the steps of the methods described previously.

[0095] The device 300 further comprises a control circuit 302, an input interface 303 for receiving data, in particular the training data of the learning model and / or the monocular intraoperative two-dimensional images acquired by the imaging device 2, and an output interface 304 for providing output data, such as parameters of the previously trained learning model or the current partial three-dimensional model in real time.

[0096] In one or more embodiments, the device 300 may be in the form of a computer having a display 305 and a keyboard 306 to enable easy interaction with a user. The device 300 may be a mobile terminal, a computer, a computer network, an electronic component, or other apparatus comprising a processor operably coupled to a memory, as well as, depending on the embodiment selected, a data storage unit, and other associated hardware such as a network interface and a media reader for reading and writing to a removable storage medium. The removable storage medium may be, for example, a compact disc (CD), a digital video / versatile disc (DVD), a flash drive, a USB flash drive, etc.Depending on the embodiment, the memory, data storage unit, or removable storage medium contains instructions that, when executed by the control circuit 302, cause the control circuit 302 to control the input interface 303, the output interface 304, and the memory 301.

[0097] The control circuit 302 may be a component implementing a processor or a calculation unit for training a learning model and / or calculating a current partial three-dimensional model of a patient's joint according to the proposed method and for controlling the units 301, 303 and 304 of the device 300.Furthermore, the device 300 may be implemented in software form (“software” or “firmware”), in which case it takes the form of a program executable by a processor, corresponding for example to an application downloadable and executable on a device of the smartphone or tablet type, as described above, or in hardware form (or “hardware”), such as an application-specific integrated circuit (ASIC), a system on chip (in English “System On Chip”, or SOC), or in the form of a combination of hardware and software elements, for example a software program intended to be loaded and executed on a component of the FPGA type (for “Field Programmable Gate Array” in English). SOCs are embedded systems which integrate all the . components of an electronic system in a single chip. An ASIC (Application Specified Integrated Circuit) is a specialized electronic circuit that groups together tailor-made functionalities for a given application. ASICs are generally configured during their manufacture and can only be simulated by the user. Programmable logic circuits such as FPGAs have electronic circuits that can be reconfigured by the user. The device 300 can also use hybrid architectures, for example, architectures based on a CPU+FPGA, a GPU (Graphics Processing Unit) or an MPPA (Multi-Purpose Processor Array).

[0098] Furthermore, the flowchart shown in [Fig.l] is a typical example of a program some instructions of which can be carried out by the device described. As such, [Fig.l] may correspond to the flowchart of the general algorithm of a computer program in a particular embodiment.

Claims

Claims

1. A computer-implemented method for modeling a patient's joint for real-time assistance in creating at least one bone tunnel by arthroscopy in said joint, comprising: - a step of obtaining (El), using an imaging device (1), a stream of monocular intraoperative two-dimensional images, each of the monocular intraoperative two-dimensional images being obtained at a current time t and comprising at least one portion of interest of the joint; - a first step of calculating (E2), for each monocular intraoperative two-dimensional image, an associated depth map, by implementing a previously trained learning model; - an intermediate step of filtering outlier data in each calculated depth map;- a second step of calculating (E3) a current partial three-dimensional model (8) of the patient's joint at the current time t, on the basis of at least one of said monocular intraoperative two-dimensional images obtained at and / or before the current time t, of at least one of said depth maps calculated at and / or before the time t, of information relating to a current position and orientation of the imaging device (1), and of a complete preoperative three-dimensional model of the patient's joint.;

2. Method according to claim 1, in which the second calculation step (E3) is carried out with a SLAM type algorithm.

3. The method of claim 1, wherein the second calculation step (E3) is performed using a reinforcement learning model.

4. Method according to one of the preceding claims, in which the pre-trained learning model is an artificial neural network.

5. A method according to any preceding claim, wherein the learning model has been previously trained with a set of monocular intraoperative two-dimensional training images and a set of associated ground truths, said ground truths being distances, being distances between a virtual imaging device and at least one object included in the set of monocular intraoperative two-dimensional training images and said training images being associated with virtual images from a preoperative three-dimensional model of the patient's joint.

6. Method according to one of the preceding claims, in which the preoperative three-dimensional model has been obtained from two-dimensional medical images.

7. Method according to one of the preceding claims, in which the learning model has been previously trained with a set of monocular intraoperative two-dimensional training images and a set of associated ground truths, said ground truths being distances obtained using pairs of images acquired at different times using an imaging device called the training imaging device and information relating to positions and orientations of said training imaging device.

8. Method according to one of the preceding claims, in which the learning model has been previously trained with a set of monocular intraoperative two-dimensional training images and a set of associated ground truths, said ground truths being distances measured on cadaveric joints, and said training images being two-dimensional images of said cadaveric joints.

9. A computing device comprising a circuit configured to implement a method according to one of claims 1 to 8.

10. Computer program product comprising instructions for implementing steps of the method according to one of claims 1 to 8, when this program is executed by a processor.

Images