Surgical navigation system

Abstract

This surgical navigation system can determine a pose for "M" optical markers (M1-M16) among "Ntot" optical markers (M1-M16), even if one of the "M" optical markers (M1-M16) is partially or totally hidden in the field of view of one monocular camera (C1-C3) among "N" monocular cameras (C1-C3), N≥3; two conditions must be met: - 2D images of the number "M" of optical markers (M1-M16) are captured, at a given time, by at least two monocular cameras (C1-C3) of the set, where "M" is an integer satisfying 5≤M≤Ntot; - each monocular camera of the set (C1-C3) captures 2D images, at the given time, of a cumulative number "m" of different optical markers (M1-M16) that are shared with the other "N-1" monocular cameras (C1-C3) of the set, where "m" is an integer satisfying 5≤m≤M.

Description

[0001] SURGICAL NAVIGATION SYSTEM

[0002] Technical field

[0003] The invention relates to the technical field of surgical navigation systems.

[0004] The invention is notably applicable to navigate a surgical tool during a surgical intervention.

[0005] Background art

[0006] A surgical navigation system known from prior art, notably from US 2016 / 0225192 Al, comprises a head-mounted display (HMD) which may be connected to more than one camera. Video streams from the cameras can be provided to an operational function that estimates distance to an object viewed by the cameras. The operational function can include triangulation of distance to the object based on angular offset of the object viewed in the video streams and a known distance between the cameras, §0032.

[0007] Such a surgical navigation system from prior art is not entirely satisfactory because triangulation cannot be performed in an appropriate manner when trackers (e.g. optical trackers) are totally or partially hidden. Yet, optical trackers are likely to be hidden when something or someone comes in between the cameras and the optical trackers.

[0008] Disclosure of the invention

[0009] The invention aims to completely or partly overcome the aforementioned drawbacks. To this end, one subject of the invention is a surgical navigation system, comprising:

[0010] - a set of “Ntot” optical markers, configured to track poses of objects, where “N tot is iin integer satisfying Ntot>5; the “Ntot” optical markers of the set having unknown relative positions therebetween and unknown absolute positions;

[0011] - a set of “N” monocular cameras, arranged to capture 2D images of the set of “Ntot” optical markers, where “N” is an integer satisfying N>3; the “N” monocular cameras of the set being movable therebetween, the “N” monocular cameras of the set having unknown relative positions therebetween and unknown absolute positions; each monocular camera of the set being movable with respect to the “Ntot” optical markers;

[0012] 2D images of a number “M” of optical markers being captured, at a given time, by at least two monocular cameras of the set, where “M” is an integer satisfying 5<M<Ntot; each monocular camera of the set capturing 2D images, at the given time, of a cumulative number “m” of different optical markers that are shared with the other “N-l” monocular cameras of the set, where “m” is an integer satisfying 5<m<M;

[0013] - a data processor, configured to perform steps at the given time: a) extract, for each monocular camera of the set, 2D coordinates of optical markers whose 2D images are captured by the corresponding monocular camera; b) generate all k-tuples of the 2D coordinates extracted in step a), each k-tuple associating a combination of “k” 2D coordinates extracted for “k” monocular cameras of the set, where “k” is an integer satisfying 2<k<N; c) initialize a pose for each monocular camera of the set in a 3D reference coordinate system; d) build 3D points in the 3D reference coordinate system from projection lines of the k- tuples generated in step b); e) reproject the 3D points built in step d) onto the 2D images captured by the “N” monocular cameras of the set, and determine a reprojection error for each 3D point; f) select the number “M” of 3D points reprojected in step e) that minimize the reprojection error; g) determine a pose for each monocular camera of the set in the 3D reference coordinate system that minimizes the reprojection error for the “M” 3D points selected in step f); h) compute a pose for the “M” optical markers in the 3D reference coordinate system from the poses determined in step g).

[0014] Thus, such a surgical navigation system according to the invention can determine a pose for the “M” optical markers, even if one of the “M” optical markers is partially or totally hidden in the field of view of one monocular camera of the set. To make it happen, two conditions must be met:

[0015] (i) 2D images of the number “M” of optical markers are captured, at a given time, by at least two monocular cameras of the set, where “M” is an integer satisfying 5<M<Ntot;

[0016] (ii) each monocular camera of the set captures 2D images, at the given time, of a cumulative number “m” of different optical markers that are shared with the other “N-l” monocular cameras of the set, where “m” is an integer satisfying 5<m<M.

[0017] Then, such a data processor is able to compute a pose for the “M” optical markers by performing steps a) to h), including when one of the “M” optical markers is partially or totally hidden. However, when M Ntot, the poses of the “Ntot-M” optical markers cannot be computed in so far as 2D images thereof are not captured, at the given time, by at least two monocular cameras of the set.

[0018] Therefore, such a surgical navigation system according to the invention makes it possible to track poses of objects, including when one of the “M” optical markers is partially or totally hidden.

[0019] The surgical navigation system according to the invention may include one or more of the following features.

[0020] According to one feature of the invention, the data processor is configured to performsteP g) by: gi) refining the pose initialized in step c) for each monocular camera of the set; g2) executing steps d), e) and f); gs) iterating steps gi) and g2) until the pose refined in step gi) minimizes the reprojection error determined in step e).

[0021] In other words, the data processor is configured to perform step g) by: go) retrieving the arbitrary pose initialized in step c); gi) refining the pose retrieved in step go) for each monocular camera of the set; g2) executing steps d), e) and f); go) iterating steps gi) and go) until the pose refined in step gi) minimizes the reprojection error determined in step e) .

[0022] Thus, one advantage afforded is that of improving the calculation accuracy of a pose for the “M” optical markers.

[0023] According to one feature of the invention, the data processor is further configured to perform step g) by: g4) comparing the minimized reprojection error at the end of step gs) with a predetermined threshold; gs) executing step gs) until the minimized reprojection error is inferior to the predetermined threshold.

[0024] Thus, one advantage afforded is that of defining a satisfactory safety level for the calculation of a pose for the “M” optical markers in order to improve the reliability thereof. According to one feature of the invention, the data processor is configured to perform steps c), d), e), f) and g) by implementing a data fusion algorithm, preferably a bundle adjustment algorithm.

[0025] Thus, one advantage afforded by the bundle adjustment algorithm is that of providing both the 3D points describing the scene geometry and the poses of the monocular cameras.

[0026] According to one feature of the invention, the data fusion algorithm includes a marker assignment algorithm designed to assign 2D coordinates extracted in step a) to an optical marker whose 2D images are captured by the corresponding monocular camera.

[0027] Thus, one advantage afforded is that of enhancing the reliability of the fusion data.

[0028] According to one feature of the invention, the data processor is configured to perform step c) by generating a random pose for each monocular camera of the set.

[0029] According to one feature of the invention, the data processor is configured to perform step c), at a time later than the given time, by initializing the poses with the poses determined in step g) at the given time.

[0030] In other words, the data processor is configured to update the arbitrary poses initialized during step c), at a time later than the given time, with the poses determined in step g) at the given time.

[0031] Thus, one advantage afforded is that of reducing the convergence time when performing step g) in comparison with a new arbitrary (e.g. random) pose for each monocular camera of the set.

[0032] According to one feature of the invention, the set of “Ntot” optical markers are selected from retro-reflective markers, colored marks, textured marks, geometrical patterns, coded targets, scale bars.

[0033] According to one feature of the invention, the surgical navigation system comprises a wireless module configured to connect the data processor to the set of “N” monocular cameras.

[0034] Thus, one advantage afforded is that a wireless module avoids cluttering the operation room.

[0035] According to one feature of the invention, the surgical navigation system comprises: - a pair of glasses;

[0036] - a head-up display, mounted on the pair of glasses, and connected to the data processor; wherein a monocular camera of the set is fixedly positioned with respect to the head-up display.

[0037] Thus, one advantage afforded is that of displaying useful information (e.g. poses of the tracked objects) for the person equipped with the pair of glasses. One second advantage afforded is that of having the nearest view possible of the scene, without impeding the point of view between the surgeon and the medical operation.

[0038] According to one feature of the invention, the surgical navigation system comprises a head-mounted display connected to the data processor, wherein a monocular camera of the set is fixedly positioned with respect to the head-mounted display.

[0039] Thus, one advantage afforded is that of displaying useful information (e.g. poses of the tracked objects) for the person equipped with the head-mounted display. One second advantage afforded is that of having the nearest view possible of the scene, without impeding the point of view between the surgeon and the medical operation.

[0040] According to one feature of the invention:

[0041] - the “M” optical markers are distributed among at least one optical tracker having a known geometry;

[0042] - said at least one optical tracker is configured to track poses of objects.

[0043] It should be noted that knowledge of the geometry of said at least one optical tracker is not a piece of information used when performing steps a) to h).

[0044] According to one feature of the invention, the data processor is further configured to perform, after step h), steps:

[0045] - assign 2D coordinates extracted in step a) to an optical tracker among said at least one optical tracker;

[0046] - compute poses of the objects tracked by said at least one optical tracker in the 3D reference coordinate system, from the poses computed in step h).

[0047] Thus, one advantage afforded is that of computing poses of tracked objects, including when the “M” optical markers are distributed among several optical trackers. Definitions

[0048] - The term “object” is understood to mean an object used in surgery. By way of nonlimiting examples, the term “object” may refer to a surgical tool (instrument), a bone or an implant.

[0049] - The terms “2D” and “3D” are understood to mean 2-dimensional and 3-dimensional respectively.

[0050] - The term “2D images of a number “M” of optical markers being captured, at a given time, by at least two monocular cameras of the set” is understood to mean that the “M” optical markers are entirely visible in the 2D images captured by said at least two monocular cameras of the set. In general, the term “capture 2D images of markers by a monocular camera” is understood to mean that said markers are entirely (integrally) visible in the 2D images captured by said monocular camera.

[0051] - The term “a cumulative number “m” of different optical markers that are shared with the other “N-l” monocular cameras of the set” is understood to mean that any given monocular camera of the set shares with the other “N-l” monocular cameras of the set a number (“m”) of markers that is defined by adding successive occurrences of different markers that are (entirely) visible by both the given monocular camera and each of the other monocular cameras of the set. The term “different markers” is understood to mean that an occurrence is not added when the same marker is redundantly shared by the given monocular camera and at least two of the other monocular cameras of the set. In that case, only one marker is counted in the cumulative number. In other words, the cumulative number “m” of different optical markers is defined by adding, in a non-redundant way, successive occurrences of different optical markers that are visible by both the corresponding monocular camera and each of the other “N-l” monocular cameras of the set that shares different optical markers with the corresponding monocular cameras. The cumulative number “m” is not incremented when an optical marker is not shared by the corresponding camera and one of the “N-l” monocular cameras of the set. So, the subset of “m” different markers does not have to be captured by all the monocular cameras.

[0052] - The term “data processor” is understood to mean a device, such a calculator or a computer, that performs operations on data.

[0053] - The term “configured to perform steps at the given time” is understood to mean that the data processor performs steps a) to h) at the time at which 2D images of optical markers are captured, or at a time sufficiently close thereto such that the captured images are representative of the same scene state. This term does not imply that steps a) to h) are executed strictly simultaneously with image capture (not achievable in practice due to processing and acquisition delays), but rather that such steps are performed based on image data obtained at or around said time.

[0054] - The term “k-tuple” is understood to mean a list of “k” elements, where “k” is an integer satisfying 2<k<N; each element of the list being 2D coordinates extracted from a different monocular camera. For example, when N=3, all k-tuples with 2<k<3 are generated in step b), that is to say all 2-tuples (doublets) and all 3-tuples (triplets) are generated in step b). In other words, all 2-tuples cover all combinations of two 2D coordinates extracted by two monocular cameras; all 3-tuples cover all combinations of three 2D coordinates extracted by three monocular cameras. Of course, the number of 2D coordinates extracted by each monocular camera of the set may be different, depending on the optical markers whose 2D images are captured by the corresponding monocular camera.

[0055] - The term “pose” is understood to mean the position and the spatial orientation. The pose of an entity can be defined by three parameters of position and three parameters of orientation.

[0056] - The term “3D reference coordinate system” is understood to mean a geometric reference frame used to describe poses of entities (markers, tracked objects, monocular cameras etc.) that may be obtained by a 3D geometrical reconstruction.

[0057] - The term “monocular” is understood to mean a camera that captures visual information through a single optical channel, typically comprising one or several successive lenses, and projects it onto a single photosensitive surface (e.g. an image sensor).

[0058] - The term “the Ntot optical markers of the set having unknown relative positions therebetween” is understood to mean that the positions of the optical markers relative to one another are not known in advance. In other words, the term is understood to mean an unknown position of the optical markers relative to each other.

[0059] - The term “the N monocular cameras of the set having unknown relative positions therebetween” is understood to mean that the positions of the monocular cameras relative to one another are not known in advance. In other words, the term is understood to mean an unknown position of the monocular cameras relative to each other.

[0060] - The term “each monocular camera of the set capturing 2D images” is understood to mean that each monocular camera of the set is arranged to capture 2D images (of optical markers) .

[0061] - The term “initialize a pose” is understood to mean that the data processor is configured to assign an initial, arbitrary pose for each monocular camera of the set during step c). This initial, arbitrary pose may be generated randomly. This initial, arbitrary pose serves as a starting point, and may be subsequently updated or refined by assigning the value of the pose determined during step g) .

[0062] Brief description of the drawings

[0063] Other features and advantages will become apparent in the detailed description of various embodiments of the invention, the description being accompanied by examples and references to the appended drawings.

[0064] Figure 1 is a schematic perspective view of a surgical navigation system according to the invention in a surgical room.

[0065] Figure 2 is a schematic representation of fields of view belonging to monocular cameras in a surgical navigation system according to the invention, said fields of view capturing certain optical markers at a given time.

[0066] Figure 3 is a schematic representation of fields of view belonging to monocular cameras in a surgical navigation system according to the invention, said fields of view capturing a certain number of optical markers at a given time.

[0067] Figure 4 is analogous to Figure 3, illustrating a different spatial configuration of fields of view.

[0068] Figure 5 is analogous to Figure 4, illustrating another spatial configuration of fields of view.

[0069] Figure 6 is a schematic perspective view, illustrating 3D points built in step d) and 3D points selected in step f) that minimize the reproj ection error.

[0070] It should be noted that, for the sake of readability and ease of understanding, the drawings described above are schematic and are not necessarily to scale.

[0071] Detailed description of embodiments

[0072] Elements that are identical or provide the same function will carry the same references for the various embodiments, for the sake of simplicity.

[0073] One subject of the invention is a surgical navigation system, comprising:

[0074] - a set of “Ntot” optical markers M1-M16, configured to track poses of objects, where “Ntot” is an integer satisfying Ntot>5; the “Ntot” optical markers Ml -Ml 6 of the set having unknown relative positions therebetween and unknown absolute positions;

[0075] - a set of “N” monocular cameras C1-C3, arranged to capture 2D images 11-13 of the set of “Ntot” optical markers Ml -Ml 6, where “N” is an integer satisfying N>3; the “N” monocular cameras C1-C3 of the set being movable therebetween, the “N” monocular cameras C1-C3 of the set having unknown relative positions therebetween and unknown absolute positions; each monocular camera C1-C3 of the set being movable with respect to the “Ntot” optical markers Ml -Ml 6;

[0076] 2D images 11-13 of a number “M” of optical markers M1-M16 being captured, at a given time, by at least two monocular cameras C1-C3 of the set, where “M” is an integer satisfying 5<M<Ntot; each monocular camera C1-C3 of the set capturing 2D images 11-13, at the given time, of a cumulative number “m” of different optical markers Ml -Ml 6 that are shared with the other “N-l” monocular cameras C1-C3 of the set, where “m” is an integer satisfying 5<m<M;

[0077] - a data processor DP, configured to perform steps at the given time: a) extract, for each monocular camera C1-C3 of the set, 2D coordinates of optical markers Ml -Ml 6 whose 2D images 11-13 are captured by the corresponding monocular camera C1-C3; b) generate all k-tuples of the 2D coordinates extracted in step a), each k-tuple associating a combination of “k” 2D coordinates extracted for “k” monocular cameras C1-C3 of the set, where “k” is an integer satisfying 2<k<N; c) initialize a pose for each monocular camera C1-C3 of the set in a 3D reference coordinate system; d) build 3D points P in the 3D reference coordinate system from projection lines PL of the k-tuples generated in step b); e) reproject the 3D points P built in step d) onto the 2D images 11-13 captured by the “N” monocular cameras C1-C3 of the set, and determine a reprojection error for each 3D point P; f) select the number “M” of 3D points P reprojected in step e) that minimize the reprojection error; g) determine a pose for each monocular camera C1-C3 of the set in the 3D reference coordinate system that minimizes the reprojection error for the “M” 3D points P selected in step f); h) compute a pose for the “M” optical markers Ml -Ml 6 in the 3D reference coordinate system from the poses determined in step g).

[0078] Optical markers

[0079] The surgical navigation system comprises a set of “Ntot” optical markers Ml -Ml 6, configured to track poses of objects. “Ntot” is an integer satisfying Ntot>5. In other words, optical markers Ml -Ml 6 (also known as fiducial markers) are used to locate the position and orientation of objects in combination with the set of monocular cameras C1-C3. The “Ntot” optical markers Ml -Ml 6 of the set have unknown relative positions therebetween, and unknown absolute positions.

[0080] The set of “Ntot” optical markers M1-M16 are advantageously selected from retro- reflective markers, colored marks, textured marks, geometrical patterns, coded targets, scale bars. The set of “Ntot” optical markers Ml -Ml 6 may be 2D or 3D.

[0081] The set of “Ntot” optical markers Ml -Ml 6 may be distributed among at least one optical tracker T, generally among several optical trackers T, in order to track poses of objects.

[0082] Monocular cameras

[0083] The set of “N” monocular cameras C1-C3 is arranged to capture 2D images 11-13 of the set of “Ntot” optical markers M1-M16. “N” is an integer satisfying N>3. The “N” monocular cameras C1-C3 of the set are movable therebetween. The “N” monocular cameras C1-C3 of the set have unknown relative positions therebetween and unknown absolute positions. Each monocular camera C1-C3 of the set is movable with respect to the “Ntot” optical markers Ml- M16.

[0084] 2D images 11-13 of a number “M” of optical markers Ml -Ml 6 are captured, at a given time, by at least two monocular cameras C1-C3 of the set. “M” is an integer satisfying 5<M<Ntot. The “M” optical markers may be distributed among at least one optical tracker T, generally among several optical trackers T. Said at least one optical tracker T is configured to track poses of objects.

[0085] Each monocular camera C1-C3 of the set captures 2D images 11-13, at the given time, of a cumulative number “m” of different optical markers Ml -Ml 6 that are shared with the other “N-l” monocular cameras C1-C3 of the set. “m” is an integer satisfying 5<m<M.

[0086] Each monocular camera C1-C3 of the set may comprise a light source configured to emit infrared radiations. The light source may include a set of light-emitting diodes. The light source is advantageously integrated in the corresponding monocular camera C1-C3 for compactness reasons. As illustrated in figure 1, a monocular camera C1-C3 of the set may be mounted on an autonomous support 1.

[0087] Data processor: step a)

[0088] The data processor DP is configured to extract, for each monocular camera C1-C3 of the set, 2D coordinates of optical markers M1-M16 whose 2D images 11-13 are captured by the corresponding monocular camera C1-C3.

[0089] Various techniques and algorithms known by a person skilled in the art may be employed to perform this step. These techniques may include edge detection algorithms (like Sobel or Canny), corner detection algorithms (like Harris comer detector), blob detection algorithms (like the Laplacian of Gaussian).

[0090] Data processor: step b)

[0091] The data processor DP is configured to generate all k-tuples of the 2D coordinates extracted in step a). Each k- tuple associates a combination of “k” 2D coordinates extracted for “k” monocular cameras C1-C3 of the set, where “k” is an integer satisfying 2<k<N.

[0092] For example, when N=3, all k-tuples with 2<k<3 are generated in step b), that is to say all 2 -tuples (doublets) and all 3-tuples (triplets) are generated in step b). All 2-tuples cover all combinations of two 2D coordinates extracted by two monocular cameras C1-C3. All 3- tuples cover all combinations of three 2D coordinates extracted by three monocular cameras C1-C3. As a pure illustrative (non-functional) example, let us assume that:

[0093] - 2D coordinates “ell” of one optical marker are extracted for monocular camera Cl,

[0094] - 2D coordinates “c21, c22” of two optical markers are extracted for monocular camera C2,

[0095] - 2D coordinates “c31, c32, c33” of three optical markers are extracted for monocular camera C3.

[0096] Then, {[ell, c21], [ell, c22], [ell, c31], [ell, c32], [ell, c33], [c21, c31], [c21, c32], [c21, c33], [c22, c31], [c22, c32], [c22, c33]} are all eleven 2-tuples generated in step b). In addition, { [ell, c21, c31], [ell, c21, c32], [ell, c21, c33], [ell, c22, c31], [ell, c22, c32], [ell, c22, c33] } are all six 3-tuples generated in step b).

[0097] To this end, the data processor is configured to store the 2D coordinates extracted in step a) in an array, a list, or another data structure.

[0098] Data processor: step c)

[0099] The data processor DP is configured to initialize a pose for each monocular camera Cl- C3 of the set in a 3D reference coordinate system at the given time. The data processor DP is advantageously configured to perform step c) at the given time by generating a random pose for each monocular camera C1-C3 of the set.

[0100] The data processor DP is advantageously configured to perform step c), at a time later than the given time, by initializing the poses with the poses determined in step g) at the given time.

[0101] The data processor DP is advantageously configured to perform step c) by implementing a data fusion algorithm, preferably a bundle adjustment algorithm. Examples of implementations of a bundle adjustment algorithm can be found in the document (PhD thesis): E. Mouragnon, “Reconstruction 3D et localisation simultanee de cameras mobiles : une approche temps-reel par ajustement de faisceaux local’ , 2007, or in the document R. Li et al., “Incremental bundle adjustment techniques using networked overhead and ground imagery for long-range autonomous Mars Ikover localisation” , Proc, of the 8thInternational Symposium on Artifical Intelligence, Robotics and Automation in Space — iSAIRAS, 2005.

[0102] Dafa processor: step d)

[0103] The data processor DP is configured to build 3D points P in the 3D reference coordinate system from projection lines PL of the k- tuples generated in step b).

[0104] The data processor DP is advantageously configured to perform step d) by implementing a data fusion algorithm, preferably a bundle adjustment algorithm.

[0105] Data processor: step e)

[0106] The data processor DP is configured to:

[0107] - reproject the 3D points P built in step d) onto the 2D images 11-13 captured by the “N” monocular cameras C1-C3 of the set, and

[0108] - determine a reprojection error for each 3D point P.

[0109] The data processor DP is advantageously configured to perform step e) by implementing a data fusion algorithm, preferably a bundle adjustment algorithm.

[0110] Data processor: step f)

[0111] The data processor DP is configured to select the number “M” of 3D points P reprojected in step e) that minimize the reprojection error. To this end, a pixel threshold is advantageously defined to improve the reliability of the selection.

[0112] The number “M” of 3D points P minimizing the reprojection error corresponds to the number “M” of optical markers Ml -Ml 6 whose 2D images 11-13 are captured, at the given time, by at least two monocular cameras C1-C3 of the set.

[0113] The data processor DP is advantageously configured to perform step f) by implementing a data fusion algorithm, preferably a bundle adjustment algorithm.

[0114] Data processor: step g)

[0115] The data processor DP is configured to determine a pose for each monocular camera Cl- C3 of the set in the 3D reference coordinate system that minimizes the reprojection error for the “M” 3D points P selected in step f).

[0116] The data processor DP is advantageously configured to perform step g) by: gi) refining the pose initialized in step c) for each monocular camera C1-C3 of the set; gz) executing steps d), e) and f); gs) iterating steps gi) and g2) until the pose refined in step gi) minimizes the reprojection error determined in step e).

[0117] The data processor DP is further advantageously configured to perform step g) by: g4) comparing the minimized reprojection error at the end of step gs) with a predetermined threshold; gs) executing step gs) until the minimized reprojection error is inferior to the predetermined threshold.

[0118] The data processor DP is advantageously configured to perform step g) by implementing a data fusion algorithm, preferably a bundle adjustment algorithm. The data fusion algorithm advantageously includes a marker assignment algorithm designed to assign 2D coordinates extracted in step a) to an optical marker M1-M16 whose 2D images 11-13 are captured by the corresponding monocular camera C1-C3. Various techniques and algorithms known by a person skilled in the art may be employed as a marker assignment algorithm. By way of nonlimiting examples, the Nearest Neighbor Assignment, the Joint Probabilistic Data Association (JPDA), the Multiple Hypothesis Tracking (MHT), the Global Nearest Neighbor (GNN), the Probabilistic Data Association Filter (PDAF) may be employed as marker assignment algorithms.

[0119] Data processor: step h)

[0120] The data processor DP is configured to compute a pose for the “M” optical markers MIMI 6 in the 3D reference coordinate system from the poses determined in step g). The data processor may be configured to perform step h) by using a matrix transformation between coordinate systems known by a skilled person in the art. The matrix transformation may be a rigid transformation referring to a geometric transformation of a Euclidean space that preserves the distance between every pair of points. The rigid transformation may include rotations, translations, reflections, or a combination thereof. By way of a non-limiting example, the data processor DP may implement Kabsch-Umeyama algorithm to perform step h). Other algorithms (Iterative Closest Point ICP, Random Sample Consensus RANSAC) are also conceivable.

[0121] The “M” optical markers may be distributed among at least one optical tracker T having a known geometry, generally among several optical trackers T. Said at least one optical tracker T is configured to track poses of objects. When the “M” optical markers M1-M16 are distributed among at least one optical tracker T having a known geometry, the data processor DP is further advantageously configured to perform, after step h), steps: - assign 2D coordinates extracted in step a) to an optical tracker T among said at least one optical tracker T, by implementing a marker assignment algorithm;

[0122] - compute poses of objects tracked by said at least one optical tracker T in the 3D reference coordinate system, from the poses computed in step h), by using a matrix transformation between coordinate systems.

[0123] It should be noted that knowledge of the geometry of said at least one optical tracker T is not a piece of information used when performing steps a) to h).

[0124] Equipment

[0125] The surgical navigation system advantageously comprises a wireless module (not illustrated) configured to connect the data processor DP to the set of “N” monocular cameras C1-C3.

[0126] According to one embodiment, the surgical navigation system comprises:

[0127] - a pair of glasses;

[0128] - a head-up display HUD, mounted on the pair of glasses, and connected to the data processor DP.

[0129] A monocular camera C1-C3 of the set is fixedly positioned with respect to the head-up display HUD. Said monocular camera is advantageously mounted on the eyeglass frame of the pair of glasses, preferably by clipping (snap-fit assembly).

[0130] According to another embodiment, the surgical navigation comprises a head-mounted display HMD connected to the data processor DP. A monocular camera C1-C3 of the set is fixedly positioned with respect to the head-mounted display HMD.

[0131] The surgical navigation system may be associated with a robotic arm 2 for assisting a user (e.g. a surgeon) during a surgical intervention when a patient lies on an operating table 3.

[0132] Example n°1

[0133] As illustrated in figures 1 and 2, Ntot=16 and N=3.

[0134] Four optical markers M13-M16 are only seen by the monocular camera C3.

[0135] 2D images of twelve markers Ml -Ml 2 are captured, at a given time, by at least two monocular cameras C1-C3 of the set: M=12. Indeed, 2D images of four optical markers Ml- M4 are captured by only two monocular cameras Cl, C3. 2D images of four optical markers M5-M8 are captured by all monocular cameras Cl, C2, C3. 2D images of four optical markers M9-M12 are captured by only two monocular cameras C2, C3.

[0136] Each monocular camera C1-C3 of the set captures 2D images, at the given time, of a cumulative number “m” of different optical markers that are shared with the other two monocular cameras of the set, where “m” is an integer satisfying 5<m<M. For monocular camera Cl: m=ml=4 [M5-M8 shared with monocular camera C2] + 8 [M1-M8 shared with monocular camera C3] — 4 [M5-M8 are redundant] = 8.

[0137] For monocular camera C2: m=m2=4 [M5-M8 shared with monocular camera Cl] + 8 [M5-M12 shared with monocular camera C3] — 4 [M5-M8 are redundant] = 8.

[0138] For monocular camera C3: m=m3=8 [M1-M8 shared with monocular camera Cl] + 8 [M5-M12 shared with monocular camera C2] — 4 [M5-M8 are redundant] = 12.

[0139] Example n°2

[0140] As illustrated in figure 3, Ntot=x+y+z+10 and N=3. “x” is the number of optical markers only seen by the monocular camera Cl. “y” is the number of optical markers only seen by the monocular camera C2. “z” is the number of optical markers only seen by the monocular camera C3.

[0141] 2D images of ten markers (4+ 3+ 1+2) are captured, at a given time, by at least two monocular cameras C1-C3 of the set: M=10. Indeed, 2D images of four optical markers are captured by only two monocular cameras Cl, C3. 2D images of three optical markers are captured by only two monocular cameras Cl, C2. 2D images of one optical marker are captured by all monocular cameras Cl, C2, C3. 2D images of two optical markers are captured by only two monocular cameras C2, C3.

[0142] Each monocular camera C1-C3 of the set captures 2D images, at the given time, of a cumulative number “m” of different optical markers that are shared with the other two monocular cameras of the set, where “m” is an integer satisfying 5<m<M.

[0143] For monocular camera Cl: m=ml=4 [shared with monocular camera C2] + 5 [shared with monocular camera C3] — 1 [shared with both monocular cameras C2, C3: redundant] = 8.

[0144] For monocular camera C2: m=m2=4 [shared with monocular camera Cl] + 3 [shared with monocular camera C3] — 1 [shared with both monocular cameras Cl, C3: redundant] =

[0145] 6.

[0146] For monocular camera C3: m=m3=5 [shared with monocular camera Cl] + 3 [shared with monocular camera C2] — 1 [shared with both monocular cameras Cl, C2: redundant] =

[0147] 7.

[0148] Example n°3

[0149] As illustrated in figure 4, Ntot=x+y+z+9 and N=3. “x” is the number of optical markers only seen by the monocular camera Cl. “y” is the number of optical markers only seen by the monocular camera C2. “z” is the number of optical markers only seen by the monocular camera C3. 2D images of nine markers (2+2+3+2) are captured, at a given time, by at least two monocular cameras C1-C3 of the set: M=9. Indeed, 2D images of two optical markers are captured by only two monocular cameras Cl, C3. 2D images of two optical markers are captured by only two monocular cameras Cl, C2. 2D images of three optical markers are captured by all monocular cameras Cl, C2, C3. 2D images of two optical markers are captured by only two monocular cameras C2, C3.

[0150] Each monocular camera C1-C3 of the set captures 2D images, at the given time, of a cumulative number “m” of different optical markers that are shared with the other two monocular cameras of the set, where “m” is an integer satisfying 5<m<M.

[0151] For monocular camera Cl: m=ml=5 [shared with monocular camera C2] + 5 [shared with monocular camera C3] — 3 [shared with both monocular cameras C2, C3: redundant] = 7.

[0152] For monocular camera C2: m=m2=5 [shared with monocular camera Cl] + 5 [shared with monocular camera C3] — 3 [shared with both monocular cameras Cl, C3: redundant] = 7.

[0153] For monocular camera C3: m=m3=5 [shared with monocular camera Cl] + 5 [shared with monocular camera C2] — 3 [shared with both monocular cameras Cl, C2: redundant] = 7.

[0154] The two optical markers captured by the monocular cameras Cl, C2 (or by the monocular cameras Cl, C3, or by the monocular cameras C2, C3) can be hidden without affecting the ability to compute the pose for the “M” optical markers in step h). In that case, M=9-2=7; ml=m2=7-2=5; m3=7.

[0155] Example n°4

[0156] As illustrated in figure 5, Ntot=x+y+z+ll and N=3. “x” is the number of optical markers only seen by the monocular camera Cl. “y” is the number of optical markers only seen by the monocular camera C2. “z” is the number of optical markers only seen by the monocular camera C3.

[0157] 2D images of eleven markers (2+2+5+2) are captured, at a given time, by at least two monocular cameras C1-C3 of the set: M=ll. Indeed, 2D images of two optical markers are captured by only two monocular cameras Cl, C3. 2D images of two optical markers are captured by only two monocular cameras Cl, C2. 2D images of five optical markers are captured by all monocular cameras Cl, C2, C3. 2D images of two optical markers are captured by only two monocular cameras C2, C3. Each monocular camera C1-C3 of the set captures 2D images, at the given time, of a cumulative number “m” of different optical markers that are shared with the other two monocular cameras of the set, where “m” is an integer satisfying 5<m<M.

[0158] For monocular camera Cl: m=ml=7 [shared with monocular camera C2] + 7 [shared with monocular camera C3] — 5 [shared with both monocular cameras C2, C3: redundant] = 9.

[0159] For monocular camera C2: m=m2=7 [shared with monocular camera Cl] + 7 [shared with monocular camera C3] — 5 [shared with both monocular cameras Cl, C3: redundant] = 9.

[0160] For monocular camera C3: m=m3=7 [shared with monocular camera Cl] + 7 [shared with monocular camera C2] — 5 [shared with both monocular cameras Cl, C2: redundant] = 9.

[0161] Any monocular camera C1-C3 can be hidden (losing four optical markers) without affecting the ability to compute the pose for the “M” optical markers in step h). In that case, assuming the monocular camera Cl being hidden, M=11-4=7; ml =9-4=5; m2=m3=9.

[0162] Monocular cameras capturing 3D images

[0163] Another subject of the invention is a surgical navigation system, comprising:

[0164] - a set of “Ntot” optical markers M1-M16, configured to track poses of objects, where “Ntot” is an integer satisfying Ntot>5; the “Ntot” optical markers Ml -Ml 6 of the set having unknown relative positions to each other and unknown absolute positions;

[0165] - a set of “N” monocular cameras C1-C3, arranged to capture 3D images 11-13 of the set of “Ntot” optical markers Ml -Ml 6, where “N” is an integer satisfying N>3; the “N” monocular cameras C1-C3 of the set being movable therebetween, the “N” monocular cameras C1-C3 of the set having unknown relative positions to each other and unknown absolute positions; each monocular camera C1-C3 of the set being movable with respect to the “Ntot” optical markers Ml -Ml 6;

[0166] 3D images 11-13 of a number “M” of optical markers Ml -Ml 6 being captured, at a given time, by at least two monocular cameras C1-C3 of the set, where “M” is an integer satisfying 5<M<Ntot; each monocular camera C1-C3 of the set being arranged to capture 3D images 11-13, at the given time, of a cumulative number “m” of different optical markers Ml -Ml 6, the cumulative number “m” of different optical markers being defined by adding, in a non- redundant way, successive occurrences of different optical markers that are visible by both the corresponding monocular camera and each of the other “N-l” monocular cameras C1-C3 of the set that shares different optical markers with the corresponding monocular camera, where “m” is an integer satisfying 5<m<M;

[0167] - a data processor DP, configured to perform the following steps at the given time: a) extract, for each monocular camera C1-C3 of the set, 3D coordinates of optical markers M1-M16 whose 3D images 11-13 are captured by the corresponding monocular camera C1-C3; b) generate all k-tuples of the 3D coordinates extracted in step a), each k-tuple associating a combination of “k” 3D coordinates extracted for “k” monocular cameras C1-C3 of the set, where “k” is an integer satisfying 2<k<N; c) initialize an arbitrary pose for each monocular camera C1-C3 of the set in a 3D reference coordinate system; d) build 3D points P in the 3D reference coordinate system from projection lines PL of the k-tuples generated in step b); e) reproject the 3D points P built in step d) onto the 3D images 11-13 captured by the “N” monocular cameras C1-C3 of the set, and determine a reprojection error for each 3D point P; f) select the number “M” of 3D points P reprojected in step e) that minimize the reprojection error; g) determine a pose for each monocular camera C1-C3 of the set in the 3D reference coordinate system that minimizes the reprojection error for the “M” 3D points P selected in step f); h) compute a pose for the “M” optical markers M1-M16 in the 3D reference coordinate system from the poses determined in step g).

[0168] According to one embodiment, the “N” monocular cameras C1-C3 are configured to directly capture 3D images 11-13 of the set of “Ntot” optical markers M1-M16. To this end, each monocular camera C1-C3 is typically a depth image sensor (e.g. a LIDAR “Ught Detection and Ranging’ sensor, a time-of-flight sensor etc.) providing directly a 3D image.

[0169] A monocular camera C1-C3 captures visual information through a single optical channel, typically comprising one or more several successive lenses, and projects it onto a single photosensitive surface (e.g. an image sensor, such as a CMOS sensor). The image sensor is typically a 2D image sensor that captures a 2D image by recording light intensity across a grid of pixels. It provides data in two dimensions, with no depth information.

[0170] According to another embodiment, the monocular camera C1-C3 may combine a depth acquisition mechanism to the 2D image sensor in order to capture 3D images 11-13 of the optical markers M1-M16. To this end, the depth acquisition mechanism may comprise a projector configured to project a structured light onto the optical markers, the structured light having predetermined patterns (e.g. dots, stripes, grids, or other geometrically recognizable features). The projector is rigidly attached to the photosensitive surface so that it is possible to compare the predetermined patterns with the distorted patterns (reflected by the optical markers) in order to get depth information.

[0171] It is noted that the aforementioned features described in relation to steps a) to h) also apply to this subject of the invention.

[0172] The invention is not limited to the described embodiments. A person skilled in the art is capable of considering all technically feasible combinations thereof and of substituting them with equivalents.

Claims

CLAIMS1. A surgical navigation system, comprising:- a set of “Ntot” optical markers (M1-M16), configured to track poses of objects, where “Ntot” is an integer satisfying Ntot>5; the “Ntot” optical markers (M1-M16) of the set having unknown relative positions to each other and unknown absolute positions;- a set of “N” monocular cameras (C1-C3), arranged to capture 2D images (11-13) of the set of “Ntot” optical markers (M1-M16), where “N” is an integer satisfying N>3; the “N” monocular cameras (C1-C3) of the set being movable therebetween, the “N” monocular cameras (C1-C3) of the set having unknown relative positions to each other and unknown absolute positions; each monocular camera (C1-C3) of the set being movable with respect to the “Ntot” optical markers (Ml -Ml 6);2D images (11-13) of a number “M” of optical markers (Ml -Ml 6) being captured, at a given time, by at least two monocular cameras (C1-C3) of the set, where “M” is an integer satisfying 5<M<Ntot; each monocular camera (C1-C3) of the set being arranged to capture 2D images (11-13), at the given time, of a cumulative number “m” of different optical markers (M1-M16), the cumulative number “m” of different optical markers being defined by adding, in a non- redundant way, successive occurrences of different optical markers that are visible by both the corresponding monocular camera and each of the other “N-l” monocular cameras (CIGS) of the set that shares different optical markers with the corresponding monocular camera, where “m” is an integer satisfying 5<m<M;- a data processor (DP), configured to perform the following steps at the given time: a) extract, for each monocular camera (C1-C3) of the set, 2D coordinates of optical markers (Ml -Ml 6) whose 2D images (11-13) are captured by the corresponding monocular camera (C1-C3); b) generate all k-tuples of the 2D coordinates extracted in step a), each k-tuple associating a combination of “k” 2D coordinates extracted for “k” monocular cameras (C1-C3) of the set, where “k” is an integer satisfying 2<k<N; c) initialize an arbitrary pose for each monocular camera (C1-C3) of the set in a 3D reference coordinate system; d) build 3D points (P) in the 3D reference coordinate system from projection lines (PL) of the k-tuples generated in step b);e) reproject the 3D points (P) built in step d) onto the 2D images (11-13) captured by the “N” monocular cameras (C1-C3) of the set, and determine a reprojection error for each 3D point (P); f) select the number “M” of 3D points (P) reprojected in step e) that minimize the reprojection error; g) determine a pose for each monocular camera (C1-C3) of the set in the 3D reference coordinate system that minimizes the reprojection error for the “M” 3D points (P) selected in step f); h) compute a pose for the “M” optical markers (Ml -Ml 6) in the 3D reference coordinate system from the poses determined in step g).

2. The surgical navigation system according to claim 1, wherein the data processor (DP) is configured to perform step g) by: go) retrieving the arbitrary pose initialized in step c); gi) refining the pose retrieved in step go) for each monocular camera (C1-C3) of the set; gz) executing steps d), e) and f); go) iterating steps gi) and go) until the pose refined in step gi) minimizes the reprojection error determined in step e).

3. The surgical navigation system according to claim 2, wherein the data processor (DP) is further configured to perform step g) by: g4) comparing the minimized reprojection error at the end of step go) with a predetermined threshold; go) executing step go) until the minimized reprojection error is inferior to the predetermined threshold.

4. The surgical navigation system according to any claim 1 to 3, wherein the data processor (DP) is configured to perform steps c), d), e), f) and g) by implementing a data fusion algorithm, preferably a bundle adjustment algorithm.

5. The surgical navigation system according to claim 4, wherein the data fusion algorithm includes a marker assignment algorithm designed to assign 2D coordinates extracted in step a) to an optical marker (M1-M16) whose 2D images (11-13) are captured by the corresponding monocular camera (C1-C3).

6. The surgical navigation system according to any claim 1 to 5, wherein the data processor (DP) is configured to perform step c) by generating a random pose for each monocular camera (C1-C3) of the set.

7. The surgical navigation system according to any claim 1 to 6, wherein the data processor (DP) is configured to update the arbitrary poses initialized during step c), at a time later than the given time, with the poses determined in step g) at the given time.

8. The surgical navigation system according to any claim 1 to 7, wherein the set of “Ntot” optical markers (M1-M16) are selected from retro-reflective markers, colored marks, textured marks, geometrical patterns, coded targets, scale bars.

9. The surgical navigation system according to any claim 1 to 8, comprising a wireless module configured to connect the data processor (DP) to the set of “N” monocular cameras (C1-C3).

10. The surgical navigation system according to any claim 1 to 9, comprising:- a pair of glasses;- a head-up display (HUD), mounted on the pair of glasses, and connected to the data processor (DP); wherein a monocular camera (C1-C3) of the set is fixedly positioned with respect to the head-up display (HUD).

11. The surgical navigation system according to any claim 1 to 10, comprising a headmounted display (HMD) connected to the data processor (DP), wherein a monocular camera (C1-C3) of the set is fixedly positioned with respect to the head-mounted display (HMD).

Images