Method for digitally reading the measurement of the movement of at least one moving part of an instrument

Abstract

The invention relates to a method for digitally reading the measurement of the movement of at least one moving part of an instrument, the measurement being represented on the instrument by a moving indicator (3, 5) whose position on the instrument changes according to the movement of the mobile part, wherein the instrument comprising at least one marker (10, 20) consisting of a visible image comprising a characteristic pattern is placed in the field of view of at least one camera filming the instrument and items of data characteristic of each marker (10, 20) and of the instrument are stored by a computer processing system to which the camera is connected, the system processing the images of the instrument taken by the camera in real time, at least one display screen being provided to display the results of the processing operations performed by the computer system.

Description

[0001] DESCRIPTION

[0002] Method for digitally reading the measurement of the displacement of at least one moving part of an instrument

[0003] The invention relates to a method for digitally reading the measurement of the displacement of at least one moving part of an instrument. The invention also relates to a digital reading device which is particularly intended for implementing the method. The technical fields to which the invention relates are in particular those of computer-assisted vision and augmented reality applied to the recognition of the positioning of instruments in a room. The present invention is more particularly in the field of reading the measurement of displacements of parts of instruments, for example used in orthopedic surgery.

[0004] Computer vision is already known and used in operating rooms to help surgeons understand the positions and orientations of various objects, tools as well as parts of the human body. Systems using augmented reality for navigation in an area of ​​interest in an operating room are also known, using, for example, 2D markers as spatial references to provide visual assistance to an operator. These markers must be fixed in various locations, for example on the observed instruments.

[0005] However, none of the known solutions has yet used the navigation of classic mechanical instruments, offering a reading of variables that they measure in real time without the contribution of additional marker devices. As a simple example to help understanding, we can cite the specific case of knee ligament balancing instruments, called tensors, which play a fundamental role in ensuring the alignment of the prosthesis during total knee arthroplasty, by allowing the reading of the spacing and angulation on the instrument, parameters characteristic of knee flexion and extension.

[0006] The aim of the invention is to provide a method for direct digital reading of such parameters, applicable to conventional commercial instruments, i.e. usable without the contribution of the invention, and which are slightly modified to fall within the scope of the invention. In particular, the idea is to improve the ergonomics of these instruments by providing them with a "digital magnifying glass" effect, because the physical graduations which appear there are generally very small. More generally, the invention makes it possible to automatically read the measurements of numerous mechanical orthopedic instruments.

[0007] At the same time, the aim of the invention is to simplify the use of augmented reality for the navigation of mechanical instruments, with a view to superimposing on the visualized reality information on the measurement of movements of parts of these instruments, providing the surgeon with more relevant indications, for example during the protocol for placing the prosthesis, than those which he may have during simple manual use of the non-navigated instrument.

[0008] In short, the invention makes it possible to automatically read the measurements of many common mechanical orthopedic instruments on the market, which will be mentioned in more detail later.

[0009] According to a first aspect, the invention relates to a method for digitally reading the measurement of the displacement of at least one moving part of an instrument, said measurement being manifested on the instrument by a moving marker whose position on the instrument changes as a function of the displacement of the moving part, said instrument comprising at least one marker consisting of a visible image comprising a remarkable pattern, and being placed in the field of vision of at least one camera filming the instrument, characteristic data of each marker and of the instrument being stored by a computer processing system to which the camera is connected, said system processing in real time images of the instrument taken by the camera, at least one display screen being provided to display results of processing carried out by the computer system, the method being such that it comprises the following steps:

[0010] - Detection of at least one marker present on a surface of the instrument by searching for geometric characteristics of the marker stored in the computer system;

[0011] - Estimation of the 3D pose of the instrument from the image of said marker taken by the camera and the characteristic data stored in the computer system;

[0012] - Search for at least one area of ​​interest whose characteristics are stored in the computer system;

[0013] - Identification, in each area of ​​interest, of a moving reference point reflecting the movement of a moving part of the instrument;

[0014] - Detection of the position of this moving marker allowing a measurement of said movement;

[0015] - Projection on the screen of values ​​calculated from the measurements of the movements of each moving part.

[0016] In the solutions of the prior art, independent markers are fixed on the instruments, whereas in the solution of the invention, the marker is a remarkable pattern which is an adaptation of an area of ​​the instrument possibly already comprising a pattern which fulfills a function for this instrument. This pre-existing pattern is then completed and adapted to make it an area of ​​interest of a marker within the meaning of the invention, compatible with reading by computer vision which requires in particular well-defined contrasts. An example of a remarkable pattern having its own function in the instrument and which can serve as a basis for the creation of a marker within the meaning of the invention can be a graduation area which makes it possible to measure the displacement of moving parts of the instrument.A method such as that of the invention makes it possible to envisage the reading by the camera of measurements of movements of moving parts of the instrument thanks to these areas of interest which are the graduations existing on the instruments, by means of some adaptations and additions of complementary elements to make them markers specific to the invention: corners, patterns, marked contrasts.

[0017] It is therefore necessary, according to the method, to start by identifying a marker on the instrument, then to estimate its pose - so as to know the pose of the instrument - in the field of vision of the camera, that is to say in the frame of reference of said camera. The pose is in this case defined by the position and orientation of the object in the frame of reference of the camera. The determination of this pose allows the computer processing system to have information on the images to be processed, during the search for one or more areas of interest in the sense of the objective pursued by the method, namely the search for areas of the instrument which comprise a moving frame of reference reflecting the movement of a moving part of the instrument. The camera can therefore measure the movement of each frame of reference relative to the instrument in order to deduce therefrom the movement of each moving part associated with said frame of reference.The calculated displacement values ​​are projected onto a system screen, for example in conjunction with augmented reality glasses, so that an operator can access them.

[0018] According to one possibility, in the invention, a static reference mark is identified in each marker and / or in each area of ​​interest. This is not systematic, because in certain configurations, the marker acts as a (static) reference mark from which the displacement of an object or marking that serves as a moving reference mark is measured. This is for example the case when the area of ​​interest is close to, or even included in, the marker. The separate identification of a static reference mark is, on the other hand, necessary when the area of ​​interest is at a certain distance from the marker, sometimes placed on a part that is movable relative to the part of the instrument having the reference marker, parts which are therefore likely to include mechanical play introducing a measurement bias which potentially distorts the assessment of poses made from a first reference.

[0019] In the particular case of an instrument comprising two moving parts according to two degrees of freedom, it may be advantageous to couple the marker of the instrument with a local area of ​​interest comprising a moving reference reflecting the movement of a part mechanically linked to the part of the instrument comprising said marker.

[0020] In this case, the method of the invention is more precisely such that it comprises the following steps:

[0021] - Detection of a first marker present on the instrument by searching for geometric characteristics of the marker stored in the computer system;

[0022] - Estimation of the 3D pose of the instrument from the image of said marker taken by the camera and the characteristic data stored in the computer system;

[0023] - Detection, in the image containing said marker, of a local area of ​​interest whose characteristics are stored in the computer system;

[0024] - Identification, in said local area of ​​interest, of a first moving reference point reflecting the movement of a first moving part;

[0025] - Detection of the position of this first moving marker allowing a measurement of said movement;

[0026] - Application, from the first marker, of a predetermined geometric transformation, including the calculated displacement of the first moving part from the measurement of the displacement of the first moving reference point, making it possible to locate at least one second marker on the instrument having at least one first remote area of ​​interest whose characteristics are stored in the computer system;

[0027] - Identification, in said at least one second area of ​​interest, of a second moving reference point reflecting the movement of a second moving part of the instrument;

[0028] - Detection of the position of the second moving marker allowing the measurement of said movement;

[0029] - Projection on the screen of values ​​deduced from the measurements of the movements of the moving parts.

[0030] Where appropriate, particularly if the second moving part of the instrument is mechanically connected to the first moving part and has no direct connection to the part of the instrument comprising the first marker, it is preferable to identify a static marker in the second marker presenting the second area of ​​interest, as mentioned previously.

[0031] The moving reference mark may be a moving part of the instrument, or a marking on a moving part of the instrument. Such a mark may, for example, be a needle secured to a moving part, capable of moving in front of a dial provided with a graduation of possible values, or a radial line made on a rotating pinion.

[0032] To be able to implement their detection under good conditions, each area of ​​interest may comprise at least one region of geometry, patterns and positioning predefined in the processing computer system. These areas, their constituent elements / patterns, some of their remarkable points etc. are therefore known to the processing computer system. Thus, the detection of a first rectangular marker comprising a black pattern present on the instrument may comprise, in the method of the invention, the following steps:

[0033] - Segmentation of the instrument image and contouring by thresholding of black regions constituting potential markers;

[0034] - Detection of corners of identified potential markers;

[0035] - Increase in the size of potential markers and looping through the two previous steps;

[0036] - Perspective transformation to present potential markers in frontal view;

[0037] - Validation of the first marker sought by detecting a white mark near one of the corners.

[0038] We therefore first detect marker candidates by applying processing to make their images usable, before rejecting all candidates who do not meet the conditions set by the system.

[0039] A search is then carried out for a mobile reference mark in the first marker, said reference mark being for example a radial line placed on a rotating element of circular geometry of the instrument included in the first marker and forming a local area of ​​interest there. This search is carried out by detecting the contours appearing on the marker, then applying a mask distinguishing an annular region of said rotating element in which the only residual pattern is the radial line forming the mobile reference mark.

[0040] The computer processing system stores the characteristics of this area of ​​interest, and its recognition therefore largely proceeds from comparisons facilitated by the processing applied to the input image in the camera. The calculation of the slope of the radial line of the moving reference frame then makes it possible to calculate the displacement of a first moving part of the instrument, as will be seen in more detail below.

[0041] In the context of the tensor instrument more particularly taken as an example in the description, and according to the invention, a second marker is detected, by applying the aforementioned geometric transformation applied to the center of the first marker, said transformation being in particular defined by the position of a point known to the computer system and characteristic of the location of a first rectangular distant zone of interest belonging to the second marker, said zone of interest comprising a mobile reference mark consisting of a mobile needle rotating relative to the instrument.

[0042] As mentioned, the application, from the first marker, of this predetermined geometric transformation, takes into account by including it the already calculated displacement of the first moving part, calculation carried out from the measurement of the position of the first moving reference.

[0043] More precisely, for the detection of this second marker whose characteristics are recorded in the computer system, we carry out:

[0044] - Projection onto the input image in the camera of the points forming the corners of the rectangular area known to the computer system,

[0045] - A perspective transformation to present the first distant area of ​​interest in frontal view,

[0046] - Contour detection by thresholding,

[0047] - The detection of two rectangular marks framing the first distant area of ​​interest, the dimensions of which are known to the computer system,

[0048] - Recording in the computer system the coordinates of the four corners of each rectangular mark. This second area of ​​interest has the particular advantage of defining local static reference points, to avoid, as mentioned and as explained in more detail below, possible measurement biases induced by mechanical play between the tensor frame which contains the main marker, and the upper mobile column which contains the needle.

[0049] In the method of the invention, applied to such a tensor defined in more detail below, a second distant area of ​​interest is defined, adjacent to the first distant area of ​​interest and zoomed in on the needle, including the estimation of the pose of the rectangular marks, and a projection onto the input image in the camera of points delimiting the second area of ​​interest.

[0050] This second area of ​​interest is intended to be a more stable and accurate region, in which the needle angle measurement can be performed based on local landmarks on a more zoomed-in image, avoiding the aforementioned possibilities of error.

[0051] According to the method, first a thresholding of the second area of ​​interest is carried out to convert it into a black and white binary image, then a detection of the contours to detect the lines belonging to the needle, and finally a calculation of the slope of these lines.

[0052] This slope, which in fact gives the measurement of an angle separating two parts of the observed tensor, makes it possible to measure the angular displacement of a second moving part of the instrument, and to visualize it on the screen so that the operator can be aware of it.

[0053] As mentioned, the invention also relates to a digital reading device for implementing a method as described up to now, and which comprises a platform comprising a computer processing system, a camera and a screen, said platform being assemblable to a spectacle frame, the assembly constituting smart glasses.

[0054] By measuring parameters related to relative movements of moving parts of an instrument and projecting them in front of a user's eyes, the invention actually makes it possible to have a reading of spacing and angulation, for example in the context of the installation of the components of a total knee prosthesis, using a computer vision system with augmented reality via smart glasses.

[0055] The attached drawings represent, by way of example already mentioned, the implementation of the method of the invention on the basis of a ligament balancing device used for knee prosthesis fitting operations, called a tensor.

[0056] Figure 1 shows an elevation view of a tensor in its two versions adapted respectively to a left knee and a right knee

[0057] Figure 2 shows a very schematic flowchart of the general operation.

[0058] Figure 3 shows a front view of the two tensor markers which are particular areas of a normal tensor.

[0059] Figure 4 shows a flowchart presenting the schematic steps leading to the measurement of the values ​​to be measured and read.

[0060] Figure 5 represents a first processing step used in the marker detection process. Figure 6 shows a second step, illustrating the detection of the corners of the image of a marker candidate.

[0061] Figure 7 represents the detection of a mark in the image for the purpose of validating the marker detection.

[0062] Figure 8 represents the detection of the moving marker of the first local area of ​​interest on the first marker.

[0063] Figure 9 represents more precisely the calculation of the angle of said moving reference.

[0064] Figure 10 represents, on the tensor, the application of the geometric transformation resulting in a second marker.

[0065] Figure 11 illustrates the extraction of ROI 1 and ROI 2 areas of interest from the second marker.

[0066] Figure 12 represents the detection of the needle constituting the moving marker in ROI 2.

[0067] Figure 13 illustrates the creation of a virtual line passing through the center of rotation of the needle for the calculation of the slope and therefore the angulation of a part of the tensor relative to another part.

[0068] The instruments shown in Figure 1 are intended to balance the knee in preparation for surgical placement of a total knee replacement. The instrument on the left is intended for a left knee and the instrument on the right is suitable for a right knee. When placing a total knee replacement, particular attention must be paid to the following two points: - the alignment of the leg in extension, for which the hip center, the knee center and the ankle center tend to be totally or partially aligned after the operation;

[0069] - ligament tension at the level of the medial and lateral ligaments in order to guarantee good stability of the joint.

[0070] The instruments in Figure 1 refer more specifically to the second point since they allow ligament balancing to be performed in preparation for total knee arthroplasty.

[0071] Very schematically, the instrument comprises a femoral plate 1 and a tibial plate 2 whose spacing height h and angle a are adjustable. A needle 3 moving in front of and relative to a graduated plate 4 makes it possible to have the measurement of the angle a between the femoral plate 1 and the tibial plate 2. A radial mark 5 appearing on the front surface of a pinion whose drive makes it possible to move, for example via a rack, a bracket comprising the femoral plate 1, moves opposite a graduation made in another plate 6 of the instrument. The pinion, not visible, is concentric with the circle and / or the annular zone appearing on the plate 6. The method of the invention allows the digital measurement of the movements of the mobile markers that are the radial mark 5 and the needle 3, with a view to calculating the spacing (the height h) and the angulation (angle a) on the instrument for the cases of knee flexion and extension.

[0072] The method of the invention is implemented by means of a navigation system shown in Figure 2, which in practice allows the direct reading of the spacing and angulation on the tensor in cases of flexion and extension of the knee. The reading is carried out using smart glasses, including a camera and a screen, as well as processing means allowing the detection of balancing instruments mentioned in Figure 2 which is in fact part of a computer processing system allowing computer-aided vision.

[0073] The instrument actually includes two markers 10, 20 comprising the marks allowing the measurement of the angles (pinion and needle), which are illustrated in figure 3 and which are in substance practically identified with the plates 4 and 6 which are the supports. The main marker 10 is identified with the plate 6 and is in the area of ​​the pinion. It is the first marker 10, which makes it possible to give the position and the rotation (the pose) of the instrument in relation to the camera of the glasses. The second marker 20, identified with the plate 4 of the needle 3, makes it possible to correctly set up the reference base for the measurement of the angle.

[0074] Determining the pose of a marker 10, 20 of an instrument makes it possible to define the position of other parts of this instrument by following a precise geometric transformation obtained by predetermined metrological measurement, the characteristics of which are stored in the computer processing system of the system. Thanks to the calibration of the camera and the precise knowledge of the geometry of the instrument, the system can determine the spatial coordinates and the orientation of "points of interest" of the instrument.

[0075] In the implementation architecture of the method, as shown in Figure 4, the detection of the instrument does not consist solely of identifying it in a "dictionary", i.e. via the characteristics of a marker stored in the processing computer system to which the detected characteristics would be compared, as was the case for the simple markers of the prior art attached to the instruments. The detection is rather carried out from the position of the instrument, by determining the rotation of two marks 5, 3 in the region of interest of each marker 10, 20. A more detailed explanation of the operation of each step of the detection appearing in the blocks of Figure 4 is made in the remainder of this document, with reference to the following figures. Individually, most of these steps are known per se, they consist of successive image processing in order to carry out the required final detections.During the first stages, candidate markers, that is to say at these stages potential markers, are detected, some being discarded during the process to keep only the true marker(s) of the instrument, validly recognized as such.

[0076] A first transformation step, which aims in particular to outline objects in the image and to enhance contrasts, is implemented to facilitate the subsequent detection of certain characteristics specific to the markers. This step uses adaptive thresholding that divides the image into small regions and uses the average of the neighboring area as a threshold value for each region. Therefore, different threshold values ​​are retained for different regions of the used image. Then, the contours around all black regions are extracted from the processed image, knowing that the computer system looks for at least one marker with a black pattern. Additional filtering is applied to remove unwanted contours such as large contours, small contours or contours too close to each other. The result appears for example in Figure 5, applied to the configuration of the first marker 10.Morphological elements are then applied in a second step to remove noise from the image, removing a large amount of unwanted information from the processed image. This first step of image transformation for further processing is done with a reduced image size.

[0077] Next, corner detection implements a function that schematically replaces the identified contours with other contours containing fewer vertices. All undetected contours are then filtered as rectangular (therefore including 4 vertices). The extracted corners are considered as the first and last point of each approximated contour. An example of detected corners of a tensor marker is shown in Figure 6. Since candidate markers are detected at a lower resolution, another detection is necessary to obtain a better marker localization, creating a new candidate list for the original instrument image, which are ranked in order of contour size.

[0078] Still related to the process steps presented in Figure 4, to finalize the marker identification, the marker candidates are processed with other morphological operators to verify if they are actually markers. A perspective transformation is then applied to the marker candidate to present it in a frontal view with a predefined size. A thresholding is finally applied to obtain a black and white binary image.

[0079] To validate a candidate as a marker, the first step is to check whether it contains one and only one white mark in one of its four corners. To do this, the marker candidate is inspected corner by corner. The upper left corner is analyzed in all cases and the candidate is rotated four times to retrieve information from all corners. A candidate is considered a marker if it contains 1 white mark in one of its corners, and if the other three corners are black. This is shown in Figure 7, which illustrates a thresholded marker candidate that corresponds to marker 10, the first column showing it in the four control positions. In the second column, the outline of the location of the mark, known to the processing computer system, is checked.Then, in the third column, the content of the location supposed to contain the mark is checked to verify that one of the corners does indeed have a white mark, which is the case in the third row: the marker candidate is therefore validated, it is indeed a first marker of the instrument. We note that these figures actually show an area of ​​interest ROI 0 of the first marker 10, which is notably well shown in the rightmost image of figure 8.

[0080] After further checks, particularly on the intrinsic quality of the marker, the marker image moves on to the step of determining the angle of the pinion, which will allow the calculation of the height h (see Figure 4). A processing process is used to obtain the edges of the image, then a mask is applied so that the only visible feature is the radial moving marker 5 marked on the pinion, as shown in Figure 8. In practice, the starting image on the left of the figure showing the ROI 0 area of ​​interest is contoured, then a mask is applied to it which ultimately leaves visible only the contours of the line forming the moving marker 5 attached to the pinion.

[0081] In Figure 9, with the coordinates of the initial and final points of the detected lines, it is possible to calculate the slope, and therefore the angle of the line with respect to the lateral reference of the tensor, the zero of the graduation. To be declared as a pinion angle of the instrument, the detected line must pass through the center of the image (the point in the center of the right image) with a limited distance error. Knowledge of the pinion angle allows the calculation of the height h of the femoral plateau 1 with respect to the tibial plateau 2, once the pose is known.

[0082] The pose estimation step of Figure 4 can actually be initiated knowing that the best corners are known in the correct order and that additional data is available in the processing computer system, including the parameters associated with the camera. Knowing the pose of the instrument relative to the camera, it is possible to define the distance and rotation limits of the instrument to avoid entering an ambiguity zone where the readings are not reliable. The angle of the pinion displaced relative to the marker graduation reference, depending on the side of the instrument, is to be multiplied by the mechanical ratio of the angular displacement of the pinion relative to the translational displacement of a rack, to obtain the height displacement h of Figure 1.

[0083] In the instrument of said figure 1, it has been seen that there is a second marker 20 which allows a calculation of the angle a of a needle 3 which visualizes the angular difference between the femoral 1 and tibial 2 plateaus. From the marker 10, detected and validated as shown above, a geometric transformation T is applied according to the method of the invention which allows a zone of interest to be located in the second marker 20.

[0084] The first region of interest RO1 1 distant from the marker 10 is the first approximation obtained from the transformation T applied to the center of the marker 10, taken as a fixed reference. This transformation T, illustrated by an arrow in figure 10, is notably defined by the position of the upper right corner (or left, depending on whether the instrument is intended for a left or right knee) of an upper rectangle 21 relative to the central reference of the marker 10, that is to say a fixed reference of the marker 10. This position is materialized by a point 23. A lower rectangle 22 is also provided to help define the area of ​​interest ROI 1.

[0085] Since the position of the needle changes with the height h of the plate 1 , said height h must be added, along the vertical axis Y of Figure 10, to the original reference position of the area of ​​interest ROI 1. This is possible because the height h and the vertical y axis of the reference of the coordinates of the marker 10 of the tensor are aligned, as shown in Figure 10. By using this transformation T and defining the size of the ROI 1 using its known parameters, stored in the processing computer system, the four boundary corners of the area of ​​interest ROI 1 , namely the two upper corners of the rectangle 21 and the two lower corners of the rectangle 22, can be projected and, by distorting the image, the perspective is modified to have a frontal view of the ROI 1 . Once this view of the ROI 1 is determined, a threshold is applied and then a definition of the contours of the binary image is carried out.These steps are performed in order to search for the marks consisting of the two black rectangles 21, 22 of the dial of plate 4 which are supposed to be inside the ROI 1. A filtering of the contours is applied, keeping only those which can be approximated to a four-sided polygon. Then, the regions with closed contours with 4 sides are analyzed to determine if they are the black rectangular marks 21, 22, and in particular if their size corresponds to the expected size because known to the computer system. Only then can they be defined as being indeed the upper black marks 21 or lower black marks 22, according to their position in the image. For a final validation of the rectangular marks 21 and 22, they must be at a certain distance from each other. This confirms that they belong to the dial of hand 3. The four corners of each rectangular mark 21, 22 of the side dial are recorded.

[0086] Another, more precise area of ​​interest ROI 2 is then sought. This area of ​​interest 2, also visible in Figure 10, to the right of the first distant area of ​​interest ROI 1, is intended to be a more stable and precise region where the measurement of the angle of the needle 3 can be carried out according to local landmarks on a more zoomed image. The detection of its pose and that of the area of ​​interest ROI 1 makes it possible to overcome an error potentially obtained during the estimation of the pose of the marker 10 of the tensor. To do this, a new pose (position and orientation) is therefore estimated, but this time the object is composed of the rectangular marks 21, 22 of the plate 4 forming the lateral dial. With this pose, a new projection of points is necessary to define the new area of ​​interest ROI 2 delimiting an image which only contains needle 3, therefore more targeted and a little offset from the area of ​​interest ROI 1 (as seen in figure 1 1).It is determined by the parameters - known to the processing computer system - of ROI 2, the upper corners of rectangular mark 21 and lower corners of rectangular mark 22, and the known and stored parameters of the camera.

[0087] In essence, a problem here lies in the fact that physically, needle 3 of ROI 2 is not exactly in the same plane as the "static" markings consisting of rectangular marks 21, 22 of ROI 1. It is necessary to take into account a small offset on the horizontal axis of figure 10 (X axis) and the axis perpendicular to the plane of the figure (Z axis), therefore to carry out a modification of perspective, with an offset. In fact, as already mentioned, the rotation of needle 3 in ROI 2 is expressed in a reference frame determined by the pose of the reference frame of ROI 1 rather than determined by the pose of the main marker, in order not to be subject to the bias of mechanical clearances between the two main components of the instrument.

[0088] Having found the ROI 2 of the needle, a thresholding operation is applied to it to convert it into a binary image. Then, an edge detection is performed on the binary image, these operations being shown in Figure 12. The right image of said figure is subjected to a processing that returns a list containing the initial and final pixel coordinates of all the detected lines. Before identifying a line as part of the needle 3, a correction of the pixel coordinates is performed, to clearly define the points on the edges. The edge is defined as the white border of the central white band in the binary ROI image of the center.

[0089] With the corrected initial and final points, the slope of the line is calculated, and then a perfect line L is virtually created (see Figure 13), which corresponds to a line that has the same slope as the previously detected line but that passes exactly through the center of rotation 30 of the needle. Then, the distances between the initial and final points of the detected line and those of the perfect line are calculated, determining whether the points belong to a line that passes through the center of rotation 30 of the needle 3, whose position is known thanks to the determination of the pose of the two markers (the first and the second) combined with the information stored in the memory of the processing computer system.To verify that the preserved line is part of the needle and not another element of the image, both sides (top and bottom) are inspected for enough pixels (1 1 ) corresponding to the width of the needle (1 mm) with a tolerance of +0.3 mm / -0.2 mm, in a hardware configuration of a 4K resolution RGB color camera type, with a diagonal field of view of 78°.

[0090] The final result gives the reading of the angle a between the femoral 1 and tibial 2 plateaus.

[0091] As indicated, all these process steps are implemented on a platform comprising a camera and a screen adaptable to glasses with a conventional frame, together constituting smart glasses. A software application comprising the algorithms for detecting markers and calculating the movements of the moving parts of the instrument is implemented on this platform. The input data are obtained by the camera, a visual sensor, and the output data are provided by the screen, a visualization device.

[0092] The processing of visual signals and information includes comparisons with data and parameters, particularly geometric data, of the instrument stored in a processing unit of the platform, which constitutes the computer processing system which also performs the calculations necessary for the restitution on the screen of information useful to the operator, for example the values ​​of height h and angle a of the femoral 1 and tibial 2 plates of the instrument, giving the spacing and angulation between the tibia and the femur during flexion and extension of the knee.

Claims

Claims 1. Method for digitally reading the measurement of the displacement of at least one moving part of an instrument, said measurement being manifested on the instrument by a moving mark (3, 5) whose position on the instrument changes according to the displacement of the moving part, said instrument comprising at least one marker (10, 20) consisting of a visible image comprising a remarkable pattern, and being placed in the field of vision of at least one camera filming the instrument, characteristic data of each marker (10, 20) and of the instrument being stored by a computer processing system to which the camera is connected, said system processing in real time images of the instrument taken by the camera, at least one display screen being provided to display results of processing carried out by the computer system, characterized in that it comprises the following steps - Detection of at least one marker (10, 20) present on a surface of the instrument by searching for geometric characteristics of the marker (10, 20) stored in the computer system; - Estimation of the 3D pose of the instrument from the image of said marker (10, 20) taken by the camera and the characteristic data stored in the computer system; - Search for at least one area of ​​interest (ROI 0, ROI 1, ROI 2), the characteristics of which are stored in the computer system; - Identification, in each area of ​​interest (ROI 0, ROI 1, ROI 2), of a moving reference point (3, 5) reflecting the movement of a moving part of the instrument; - Detection of the position of this mobile marker (3, 5) allowing a measurement of said displacement; - Projection on the screen of values ​​calculated from the measurements of the movements of each moving part.

2. Method for digitally reading the measurement of the displacement of at least one moving part of an instrument according to the preceding claim, characterized by the identification of a static reference in a marker and / or in an area of ​​interest.

3. Method for digitally reading the measurement of the displacement of at least one moving part of an instrument according to one of the preceding claims, characterized in that it comprises the following steps: - Detection of a first marker (10) present on the instrument by searching for geometric characteristics of the marker stored in the computer system; - Estimation of the 3D pose of the instrument from the image of said marker (10) taken by the camera and the characteristic data stored in the computer system; - Detection, in the image containing said marker (10), of a local area of ​​interest (ROI 0) whose characteristics are stored in the computer system; - Identification, in said local area of ​​interest (ROI 0) of a first moving reference point reflecting the movement of a first moving part; - Detection of the position of this first moving marker allowing a measurement of said movement; - Application, from the first marker (10), of a predetermined geometric transformation T, including the calculated displacement of the first moving part from the measurement of the displacement of the first moving reference mark (5), making it possible to locate at least one second marker (20) on the instrument having at least a first zone of interest (ROI 1) remote whose characteristics are stored in the computer system; - Identification, in said at least one first remote area of ​​interest (ROI 1), of a second moving reference point (3) reflecting the movement of a second moving part of the instrument; - Detection of the position of the second mobile marker (3) allowing the measurement of said movement; - Projection on the screen of values ​​deduced from the measurements of the movements of the moving parts.

4. Method for digitally reading the measurement of the displacement of at least one moving part of an instrument according to one of the preceding claims, characterized in that each area of ​​interest (ROI 0, ROI 1, ROI 2) comprises at least one region of geometry, patterns and positioning predefined in the computer processing system.

5. Method for digitally reading the measurement of the displacement of at least one moving part of an instrument according to the preceding claim, characterized in that the detection of a first rectangular marker (10) comprising a black pattern present on the instrument comprises the following steps: - Segmentation of the instrument image and contouring by thresholding of black regions constituting potential markers; - Detection of corners of identified potential markers; - Increase in the size of potential markers and looping through the two previous steps; - Perspective transformation to present potential markers in frontal view; - Validation of the first marker (10) sought by detecting a white mark near one of the corners.

6. Method for digitally reading the measurement of the displacement of at least one moving part of an instrument according to claim previous, characterized by searching for a mobile reference mark (5) in the first marker (10), said mark (5) being a radial line placed on a rotating element of circular geometry of the instrument included in the first marker (10) and forming there a local area of ​​interest (ROI 0), by detecting the contours appearing on the marker (10), then applying a mask distinguishing an annular region of said rotating element in which the only residual pattern is the radial line forming the mobile reference mark (5).

7. Method for digitally reading the measurement of the displacement of at least one moving part of an instrument according to the preceding claim, characterized by calculating the slope of the radial line of the moving reference mark (5) to then calculate the displacement of a first moving part of the instrument.

8. Method for digitally reading the measurement of the displacement of at least one moving part of an instrument according to one of claims 3 to 7, characterized by the detection of a second marker (20) by application of the geometric transformation T applied to the center of the first marker (10), said transformation T being defined by the position of a point (23) known to the computer system and characteristic of the location of a first rectangular distant zone of interest (RO1 1 ) belonging to the second marker (20), said zone of interest (ROI 1 ) comprising a moving reference mark consisting of a needle (3) movable in rotation relative to the instrument.

9. Method for digitally reading the measurement of the movement of at least one moving part of an instrument according to the preceding claim, characterized by: - Projection onto the input image in the camera of the points forming the corners of the rectangular area known to the computer system, - A perspective transformation to present the first distant area of ​​interest (ROI 1) in frontal view, - Contour detection by thresholding, - The detection of two rectangular marks (21, 22) framing the first distant area of ​​interest (ROI 1), the dimensions of which are known to the computer system, - Recording in the computer system the coordinates of the four corners of each rectangular mark (21, 22).

10. Method for digitally reading the measurement of the displacement of at least one moving part of an instrument according to the preceding claim, characterized by the definition of a second distant zone of interest (ROI 2) adjacent to the first distant zone of interest (RO1 1 ), and zoomed on the needle (3), including the estimation of the pose of the rectangular marks (21 , 22), and by a projection on the input image in the camera of points delimiting the second zone of interest (ROI 2). 1 1. Method for digitally reading the measurement of the displacement of at least one moving part of an instrument according to the preceding claim, characterized by thresholding the second area of ​​interest (ROI 2) to convert it into a black and white binary image, then by detecting the contours to detect the lines belonging to the needle (3), and by calculating the slope of these lines.

12. Digital reading device for implementing a method according to the preceding claims, characterized in that it comprises a platform comprising a computer processing system, a camera and a screen, said platform being assemblable to a spectacle frame, the assembly constituting smart glasses.

Images