System and method for dynamically calibrating an endoscopic camera

Abstract

The invention relates to a method for determining calibration parameters (7) for an endoscopic camera (1), which acquires a video stream (3) in which at least one image of the video stream displays at least one tool (10) comprising at least one straight edge (100), the method comprising: − a step E1 of receiving a plurality of coordinates of pixels associated with at least one straight edge (100) of the at least one tool (10) on one of the images (2) of the video stream (3); − a step E2 of determining a static calibration cost function from the received plurality of pixel coordinates; − a step E3 of determining calibration parameters for the endoscopic camera (1) by solving an optimization problem in which the cost function comprises at least the static calibration cost function.

Description

[0001] DESCRIPTION

[0002] TITLE OF THE INVENTION: SYSTEM AND METHOD FOR DYNAMIC CALIBRATION OF AN ENDOSCOPIC CAMERA

[0003] Technical field of the invention

[0004] The present invention relates to the technical field of calibrating video cameras, in particular an endoscopic camera.

[0005] The invention relates to a system and method for calibrating an endoscopic camera.

[0006] Technological background

[0007] Modern surgery and endoscopy increasingly rely on computer vision to improve navigation and anatomical localization. The use of computer vision methods, particularly 3D vision, requires precise calibration of the camera used. This calibration involves estimating the camera's internal parameters, sometimes called intrinsic parameters, which describe the geometric transformation between the coordinates of a pixel in an image and the projection ray passing through the camera's center. However, these internal parameters change with adjustments such as zoom and focus, posing a fundamental challenge to maintaining accuracy throughout the surgical procedure.

[0008] Currently, the calibration of cameras used in surgery and endoscopy is performed using static methods with calibration targets. While accurate, these methods are cumbersome because they require operators to set the camera parameters at the beginning of the procedure, making it impossible to update the parameters without a complete recalibration, thus interrupting the procedure. Furthermore, the need for a calibration target complicates the preparation and execution of procedures, thereby limiting their flexibility and efficiency.

[0009] In this context, self-calibration methods, which do not use a target, seemed a promising solution. However, these methods are not robust enough for use in demanding medical environments, where accuracy and reliability are essential. Self-calibration often fails to maintain the stability of internal parameters, particularly in dynamic situations where viewing conditions change frequently.

[0010] There is therefore a need to develop an approach that enables reliable and flexible calibration of the internal parameters of endoscopic cameras, offering the possibility of dynamically updating the calibration during its use and eliminating the dependence on a calibration target.

[0011] Objectives of the invention

[0012] The invention aims to provide, in at least one embodiment, a method for updating the current calibration of an endoscopic camera.

[0013] The invention aims to provide, in at least one embodiment, a method for calibrating an endoscopic camera that does not require a calibration target.

[0014] Description of the invention

[0015] To this end, the invention relates to a method for determining calibration parameters of an endoscopic camera, acquiring a video stream, each image of said video stream representing a scene, of which at least one image of said video stream displays at least one tool comprising at least one straight edge, the method being characterized in that it comprises the following steps:

[0016] - a step of receiving a plurality of pixel coordinates associated with at least one straight edge from at least one tool on one of the images of said video stream;

[0017] - a step of determining a static calibration cost function from said plurality of received pixel coordinates;

[0018] - a step of determining calibration parameters of said endoscopic camera by solving an optimization problem whose cost function, called overall cost function, includes at least said static calibration cost function.

[0019] By proceeding in this manner, the invention proposes to solve the problem of calibrating an endoscopic camera without the need for a calibration target, even in a dynamic environment. Receiving a plurality of pixel coordinates associated with at least one straight edge of a tool on one of the video stream frames makes it possible to take advantage of features present in the scene captured by the endoscopic camera, in particular the straight edges of the tools, which are often well-defined and easily detectable. By using these edges as references, the invention avoids the need for a specific calibration target, which greatly simplifies the process and makes it applicable in dynamic environments where installing a target would be impractical.

[0020] Next, determining a static calibration cost function from the plurality of received pixel coordinates allows us to formalize the calibration problem mathematically. This cost function represents, for example, the difference between the observed pixel coordinates and the expected coordinates for a given camera model. Thus, by minimizing this cost function, which includes at least the static calibration cost function, it is possible to quickly and algorithmically obtain the values ​​of the calibration parameters. By proceeding in this way, the combination of the static constraint derived from the tool geometry and the classical constraints improves the overall stability and, consequently, the resulting optimization solution.

[0021] This is particularly advantageous in medical or industrial environments where conditions can change rapidly and where continuous and reliable calibration is required.

[0022] In the context of the present invention, "camera calibration" means the process of estimating the internal parameters of a camera.

[0023] In the case where the camera is a stereoscopic camera, we will speak of camera calibration to refer to the process of estimating internal parameters as well as external parameters, including parameters relating to the position and relative orientation of the two optical sensors of said stereoscopic camera.

[0024] In the context of the present invention, the term "straight edge of a tool displayed on an image" refers to a series of pixels corresponding to the edge of a tool as it appears on an image. This edge may be a straight line when the image exhibits no optical distortion, but may also take the form of a pronounced curve in the case of significant optical distortion.

[0025] In the context of camera calibration, the internal parameters of a camera model refer to the parameters that describe the camera's internal geometry. These parameters are used to transform the 2D coordinates of the image into 3D spatial coordinates, specifically to find the projection lines passing through the 3D points corresponding to the 2D points. These internal parameters are divided into affine and nonlinear internal parameters.

[0026] The internal affine parameters of a camera are contained in a matrix, usually denoted K, of size 3 x 3, whose structure is:

[0027] [Math.l] where f x and f y are the camera's effective focal lengths in pixels, which describe the camera's zoom level and aspect ratio; c x etc y are the coordinates of the principal point in pixels, which is the point where the optical axis intersects the image plane; and the parameter T represents the obliquity, which is close to zero for most cameras and commonly replaced by this default value.

[0028] The nonlinear internal parameters are the distortion coefficients contained in the vector D. The distortion coefficients describe the optical distortion of the camera; this can be radial or tangential.

[0029] Advantageously and according to the invention, the calibration parameters of the endoscopic camera include the internal affine parameters and the internal nonlinear distortion parameters.

[0030] Advantageously, if the endoscopic camera is a stereoscopic camera comprising a pair of optical sensors, it follows that a pair of internal parameters must be determined, as well as external parameters representing the relative position of the two cameras, encoded by a rigid motion defined by a 3D rotation and translation. According to this aspect of the invention, the calibration parameters of the endoscopic camera also include internal distortion parameters. These internal distortion parameters are necessary to correct optical distortions introduced by the camera lenses, such as barrel or pincushion distortion.By taking these parameters into account, the process makes it possible to rectify distorted images, which allows the direct use of the rectified image as input for other image processing methods that require rectified images to function, particularly those requiring a high-fidelity perspective model.

[0031] Advantageously and according to the invention, the method includes a step of determining a self-calibration cost function from a plurality of point correspondences between images of said video stream; so that the overall cost function includes, in addition to said static calibration cost function, said self-calibration cost function.

[0032] According to this aspect of the invention, determining a self-calibration cost function from a plurality of point correspondences between images in the video stream improves the calibration accuracy of the endoscopic camera. By using point correspondences between different images, it is possible to capture variations and distortions that would not be detectable based solely on a single image. This allows for the consideration of dynamic factors and more effective constraints on the camera model, leading to greater reliability of the measurements performed by the camera.

[0033] The overall cost function, which includes both the static calibration cost function and the self-calibration cost function, allows the advantages of both approaches to be combined to achieve a more robust and accurate calibration.

[0034] Advantageously, and according to the invention, the overall cost function comprises two weighting parameters for weighting the static calibration cost function and the self-calibration cost function, in particular through a linear combination of said static and self-calibration cost functions. According to this aspect of the invention, the inclusion of weighting parameters in the overall cost function makes it possible to effectively weight the static calibration cost function and / or the self-calibration cost function. This weighting allows for fine-tuning the relative importance of each component in the optimization process. Consequently, this results in a more precise and robust calibration of the endoscopic camera, taking into account the variations and uncertainties inherent in different image acquisition conditions.

[0035] Thus, according to this embodiment, the overall cost function J takes the following form:

[0036] [Math.2]

[0037] J = / AE(PAF< PDI< PNU) + / ES(PAF< PDI< PNU) where f AE is a self-calibration cost function, ES a static calibration cost function, A and p of the weighting parameters, p AF and p m two vectors containing the affine internal parameters and the distortion internal parameters, for all images, and p NU a vector containing the nuisance parameters.

[0038] The term "nuisance parameters" refers to the set of parameters necessary for the expression of cost functions that are not used subsequently.

[0039] Thus, the overall cost function can be modulated according to the relative importance of the static calibration cost function compared to the self-calibration cost function. The weighting parameters could be determined dynamically based on the specific environmental conditions of the scene captured by the endoscopic camera, thereby ensuring greater flexibility in the calibration process. For example, in environments where the 3D shape of the observed organs changes rapidly, the weighting parameters could be adjusted to give more weight to the static calibration cost function, allowing for rapid and accurate adaptation to new conditions without requiring assumptions about the dynamics of the observed scene.

[0040] Advantageously and according to the invention, the method includes a step prior to the step of determining a self-calibration cost function, of determining said plurality of point correspondences between images of said video stream.

[0041] According to this aspect of the invention, the step of determining the plurality of point correspondences between images of the video stream makes it possible to identify common points between different images of the video stream, thus facilitating the reconstruction of the 3D geometry of the observed scene. By determining these point correspondences, the system can establish the spatial relationship between the different images.

[0042] Advantageously and according to the invention, said point correspondences between images of said video stream can be determined by a keypoint determination method, in particular a method among: scale-invariant feature transform “SIFT”, robust and invariant binary keypoints “BRISK”, nonlinear scale space detector and descriptor “KAZE” or methods based on artificial neural networks such as SuperPoint or on optical flow.

[0043] According to this aspect of the invention, the point correspondences between images of the video stream can be determined by a keypoint determination algorithm, such as the calculation of scale-invariant feature transforms, known by the acronym SIFT, from the English "scale-invariant feature transform", which generally ensures good accuracy in the correspondence of points between images.

[0044] Scale-invariant feature transform (SIFT) is a robust and widely used method for detecting and describing points of interest in images. SIFT is distinguished by its ability to produce descriptors that are invariant to scale and rotation, and partially invariant to changes in illumination. This method is particularly effective for identifying reliable correspondences between different views of the same object or scene, even in the presence of significant geometric transformations. In the context of endoscopic camera calibration, SIFT offers high accuracy in locating key points, resulting in more reliable estimation of camera parameters.

[0045] The application of robust and invariant binary keypoints (BRISK) provides a fast and efficient method for detecting and describing points of interest in images. BRISK is particularly well-suited to environments where processing speed is critical, such as real-time video streams. The BRISK method maintains high performance in terms of matching accuracy while reducing computation time, which is important for dynamic and responsive calibration of endoscopic cameras.

[0046] The use of the nonlinear scale space detector and descriptor (KAZE) enables the capture of fine details and complex structures in images. KAZE is capable of detecting key points with high accuracy even under varying lighting conditions and on textured surfaces. This ability to handle nonlinear variations in the image improves the quality of point matching, resulting in more precise calibration parameters for the endoscopic camera.

[0047] Alternatively, said point correspondences between images of said video stream may be determined by a method using other detectors, descriptors and / or matching mechanisms, including a tracking mechanism, in particular through a neural network such as Deep Simple Online Realtime Tracking (DeepSORT).

[0048] Advantageously and according to the invention, the method includes a preliminary step of detecting straight edge points of tools on the images of said video stream; the detection of straight edge points of tools being carried out by a specific detection method, such as an image segmentation method based on a generic artificial neural network such as Segment Anything Meta (SAM) or specifically trained for this task such as Augmented Reality Tool Network (ART-Net).

[0049] According to this aspect of the invention, the preliminary step of detecting straight edge points of tools on the video stream images allows for the precise identification of the straight edges of the tools present in the images, which is advantageous for the accurate calibration of the endoscopic camera. By using robust algorithms such as RANSAC or the Hough transform for low-dimensional curvature models, the method ensures reliable detection even in the presence of noise or disturbances in the images, thus guaranteeing a stable foundation for subsequent calibration parameter calculation steps.

[0050] Advantageously and according to the invention, the preliminary step of detecting straight edge points of tools on the images of the video stream is carried out by a neural network specially designed and trained to detect straight edge points of tools.

[0051] Using a specially designed and trained neural network to detect straight edge points on tools significantly improves the accuracy and speed of edge detection in images acquired by the endoscopic camera. Neural networks are capable of processing large amounts of visual data and learning complex features from it, making them particularly effective at identifying specific patterns such as straight edges. This ability to accurately detect straight edges of tools in images provides more precise pixel coordinates for subsequent calibration steps, contributing to a more accurate determination of the camera's calibration parameters.

[0052] Advantageously and according to the invention, the method comprises a step, following the step of detecting straight edge points of tools on the images of the video stream, of fitting a model of geometric curves, in particular models of Bézier curves or B-Splines, to said detected straight edge points.

[0053] According to this aspect of the invention, fitting a model of geometric curves, particularly Bézier curves or B-splines, to the detected straight edge points allows for a smoother and more continuous representation of tool edges. Bézier curves and B-splines are particularly effective for modeling complex shapes with high accuracy while using a reduced number of parameters; moreover, they provide a mathematical expression of the curve, which is useful for establishing calibration constraints. In other words, this geometric representation provides a precise mathematical description of the edges, leading to a more accurate determination of the endoscopic camera's calibration parameters.

[0054] Alternatively, the detection of straight edge points of tools in video stream images can be performed using edge detection algorithms based on spatial filtering techniques, such as Sobel or Canny filters. These algorithms identify brightness transitions in the image, which often correspond to object edges. Once the edge points are detected, they can be used to fit geometric curve models, such as Bézier curves or B-splines, to more accurately and continuously represent the straight edges of the tools.

[0055] Alternatively, instead of using Bézier curves or B-splines, other geometric curve models, such as Lagrange polynomials or cubic splines, can be used to adjust the detected straight edge points. These models offer different smoothing and continuity properties that can be advantageous depending on the specific characteristics of the tool edges in the video stream images. Using these alternative models allows for adaptation to a variety of edge shapes and configurations, thus increasing the flexibility of the calibration process.

[0056] Advantageously and according to the invention, the fitting of said geometric curve model is carried out by a robust regression method, for example a gradient descent method or iteratively reweighted least squares or a Gauss-Newton or Levenberg-Marquardt method, etc.

[0057] According to this aspect of the invention, the geometric curve model is fitted using a regression method, resulting in increased accuracy in the calibration of the endoscopic camera. By employing robust regression methods such as gradient descent-optimized M-estimators or iteratively reweighted least squares, with variants using Gauss-Newton or the Levenberg-Marquardt method, it is possible to minimize errors between observed pixel coordinates and model-predicted coordinates, with a tolerance for noise and detection errors. This leads to a better match between the actual data and the theoretical model, thereby improving the reliability and accuracy of the determined calibration parameters and reducing sensitivity to outliers.

[0058] The least squares method, in particular, is effective at reducing the sum of squared differences between observed and predicted values, which is advantageous for obtaining an optimal fit of geometric curves. This method is simple to implement and offers a fast and efficient solution for linear and nonlinear regression problems.

[0059] The Gauss-Newton method, on the other hand, solves least-squares problems using an iterative technique that progressively refines the model parameters by minimizing the cost function. It is particularly useful for nonlinear regression problems, where it can converge rapidly to an optimal solution, even when the initial conditions are far from the final solution.

[0060] Finally, the Levenberg-Marquardt method combines the advantages of gradient descent and Gauss-Newton methods, offering better convergence in situations where other methods might fail. It is particularly effective for complex nonlinear regression problems, where it can fit model parameters more stably and accurately.

[0061] Advantageously and according to the invention, the method comprises a step, following the step of fitting a model of geometric curves, of determining a plurality of algebraic constraints on the calibration parameters from said fitted models of geometric curves.

[0062] According to this aspect of the invention, the subsequent step of adjusting a model of geometric curves allows for refining the accuracy of the endoscopic camera's calibration parameters. By adjusting these geometric curve models, it becomes possible to better represent the straight edges of the tools present in the video stream images. This increased accuracy in modeling the geometric curves results in a better correspondence between the pixel coordinates and the actual edges of the tools, thus improving the overall calibration quality.

[0063] If desired, determining constraints on calibration parameters could include using regularization techniques to impose additional constraints on possible solutions. For example, regularization terms or L2 can be added to the cost function.

[0064] Advantageously and according to the invention, additional algebraic constraints on the calibration parameters may relate, in particular, to the aspect ratio, the obliquity or the equality of the parameters for a plurality of images of the video stream.

[0065] According to this aspect of the invention, algebraic constraints on calibration parameters, such as the aspect ratio, ensure that the proportions of objects captured by the endoscopic camera remain true to reality. This is advantageous for medical applications where precise measurements are needed to diagnose and treat conditions. By maintaining a known or constant aspect ratio, it is ensured that the relative dimensions of objects in the images are not distorted.

[0066] According to this aspect of the invention, the obliquity of the calibration parameters is also taken into account. Obliquity refers to the angle between the axes of the camera sensor. By imposing constraints on the obliquity, angular distortions that might occur can be corrected. This results in more precise and accurate images, thus facilitating visual interpretation and measurements performed on the images.

[0067] According to this aspect of the invention, the equality of parameters for a plurality of images in the video stream is another important algebraic constraint. By ensuring that the calibration parameters remain constant across multiple images, visual and metric consistency is guaranteed throughout the video stream, and, more importantly, constraint redundancy is ensured, guaranteeing reliable camera calibration. This consistency is therefore important for dynamic analyses where comparisons between different images are necessary, such as monitoring the progression of a medical condition or endoscopic navigation. This minimizes errors and variations that could otherwise complicate the interpretation of video data.

[0068] Advantageously and according to the invention, the method comprises a step, following the step of determining calibration parameters of said endoscopic camera, of updating the calibration parameters of said endoscopic camera from a cost function, called dynamic overall cost function, comprising said overall cost function on the one hand, and a cost function dependent on a distance between the current values ​​of the calibration parameters and said calibration parameter values ​​to be determined on the other hand.

[0069] According to this aspect of the invention, the overall cost function J under consideration is of the form where f AE is a self-calibration cost function, ES a static calibration cost function, A, p and 7] three weighting parameters, d a distance function, p AF and p mtwo vectors containing the affine internal parameters and the internal distortion parameters to be determined, p AE and p E)i the old values ​​of the internal affine parameters and distortion parameters, for all images, p NU a vector containing the nuisance parameters.

[0070] According to this aspect of the invention, the subsequent step of updating the endoscopic camera's calibration parameters using a dynamic overall cost function ensures increased accuracy and continuous system adaptability. By incorporating a cost function dependent on the distance between the current calibration parameter values ​​and the target values, the method minimizes deviations and optimizes camera performance in real time. This dynamic approach ensures that adjustments are constantly refined.

[0071] The use of a dynamic overall cost function, which includes both the overall cost function and a component dependent on the distance between the current parameters and those to be determined, allows for the efficient management of variations and potential disturbances in the camera environment. This ensures increased robustness of the calibration system, maintaining optimal performance even in the presence of changes in shooting conditions or tool wear.

[0072] By incorporating this dynamic update step, the process also reduces the need for frequent manual recalibrations, which can be time-consuming and prone to human error. The system's ability to self-adjust based on detected variations in calibration parameters improves operational efficiency and the overall reliability of the endoscopic camera.

[0073] Alternatively, receiving multiple pixel coordinates associated with at least one straight edge of the tool on at least one frame of the video stream could be achieved using more sophisticated edge detection algorithms, such as Hough transforms or convolutional neural networks specialized in detecting geometric shapes. Using these methods improves the accuracy of straight edge detection, even under varying lighting conditions or in the presence of noise.

[0074] Alternatively, determining a static calibration cost function from the plurality of received pixel coordinates could include adaptive weighting techniques, where each pixel coordinate is weighted according to its reliability or relevance in the context of the image.

[0075] Alternatively, determining the calibration parameters of the endoscopic camera by solving an optimization problem can be performed using stochastic optimization methods, such as genetic algorithms or simulated annealing algorithms. These methods allow for a more thorough exploration of the space of possible solutions and avoidance of certain local minima, thus leading to a more precise and reliable camera calibration.

[0076] Alternatively, updating the endoscopic camera's calibration parameters using a dynamic overall cost function could be achieved by integrating online learning mechanisms, where the calibration parameters are continuously adjusted in real time as new images of the video stream are acquired. This would maintain optimal camera calibration even in the presence of dynamic changes in the environment or operating conditions.

[0077] Advantageously, and according to the invention, the distance between the current values ​​of the calibration parameters and said calibration parameter values ​​is a distance derived from a Lebesgue standard, commonly known as the L standard p or a distance from Mahalanobis.

[0078] According to this aspect of the invention, the use of a distance derived from a standard L p to measure the distance between current calibration parameter values ​​and reference values, allowing for extensive flexibility in evaluating deviations. The L standard pwhere p can be adjusted, offers an adaptable way to quantify differences, which can be particularly useful for accommodating different data distributions or specific accuracy requirements. This flexibility allows the calibration process to be optimized according to the specific characteristics of the endoscopic camera and the conditions of use, thus improving the accuracy and robustness of the calibration.

[0079] According to this aspect of the invention, using the Mahalanobis distance to measure the distance between current calibration parameter values ​​and reference values ​​allows for the consideration of correlations between different parameter dimensions. The Mahalanobis distance is particularly effective for identifying deviations in multidimensional spaces where variables may be correlated. By incorporating this measurement, the calibration process becomes more sensitive to significant variations and less influenced by random fluctuations, leading to a more precise and reliable determination of the endoscopic camera's calibration parameters.

[0080] The invention also relates to a system for determining calibration parameters of an endoscopic camera acquiring a video stream; each image of the video stream representing a scene may include at least one tool, each tool including at least one straight edge, the system comprising:

[0081] - an endoscopic camera;

[0082] - a computing unit capable of receiving a plurality of pixel coordinates associated with at least one straight edge of at least one tool on one of the images of said video stream; said computing unit being capable of determining calibration parameters of said endoscopic camera by solving an optimization problem, the cost function of which, called the overall cost function, includes at least one static calibration cost function obtained from said plurality of received pixel coordinates.

[0083] According to this aspect of the invention, the endoscopic camera is used to acquire a video stream in which at least one image represents a scene including at least one tool with a straight edge. This configuration allows for the capture of precise and detailed images of the tools, thus facilitating the identification of the straight edges necessary for the calibration process.

[0084] The processing unit is designed to receive multiple pixel coordinates associated with at least one straight edge of the tool in one of the video stream frames. This receiving capability allows the processing unit to gather precise information about the position of the straight edges, which is fundamental for calculating calibration parameters. By processing these pixel coordinates, the processing unit can analyze the visual data in a detailed and accurate manner.

[0085] The computing unit is also capable of determining the calibration parameters of the endoscopic camera by solving a numerical optimization problem. The cost function, or overall cost function, includes at least one static calibration cost function obtained from the received pixel coordinates. This approach optimizes the calibration parameters by minimizing errors and ensuring accurate camera calibration. Solving the optimization problem allows for the estimation of calibration parameters that are consistent with the data, thus improving the accuracy and reliability of the measurements performed by the endoscopic camera.

[0086] Alternatively, the endoscopic camera can be equipped with additional sensors to capture depth information or spectral data, in addition to standard images. These additional sensors can provide complementary data, enabling more precise calibration adapted to complex or varied environments.

[0087] Alternatively, the processing unit can be integrated directly into the endoscopic camera, creating a compact and self-contained device. This integration reduces data transmission latency and improves the speed of the calibration process. Furthermore, such a configuration can facilitate camera use in environments where space is limited or mobility is essential.

[0088] Alternatively, the processing unit can be equipped with real-time processing capabilities, enabling continuous calibration of the endoscopic camera during video stream acquisition. This dynamic approach allows calibration parameters to be adjusted according to scene variations or lighting conditions, thus ensuring optimal image quality throughout the endoscopic procedure.

[0089] Alternatively, camera models can be specifically adapted to the endoscopic camera being used. These models can be pre-defined in the laboratory and stored in the computing unit, thus enabling more precise correction of distortions inherent in the camera's optics. This approach benefits the fidelity of the images obtained and the accuracy of the measurements performed on them.

[0090] Alternatively, the system can include machine learning algorithms capable of progressively improving calibration parameters based on data accumulated over time. These algorithms can analyze the results of previous calibrations and adjust the models used to better suit the specific characteristics of each endoscopic procedure. This self-improvement capability allows the system to become increasingly efficient and accurate as it is used.

[0091] The proposed method for calibrating the endoscopic camera can be used in two independent or combined ways: the first to replace static calibration and avoid the use of a calibration target, and the second to update internal parameters during the procedure. These methods allow for great flexibility and adaptability to the specific needs of the procedure. They can be used independently or in combination.

[0092] Advantageously, according to the invention, the first method allows calibration to be performed at the beginning of the procedure. This process begins with the acquisition of a video stream containing two or more images showing the work area and a tool. The steps include establishing correspondences between the images in the video stream using key points such as SIFT or specific anatomical points, forming a self-calibration cost function from these correspondences, and detecting potential tool edge points in the images using appropriate methods such as edge detectors or deep neural networks. Next, a curve model, for example, a B-spline curve estimated by RANSAC, is fitted to the detected points to retain only the grouped tool edge points. A static calibration cost function is then formed from these points.Constraints on the internal affine parameters, such as aspect ratio and obliquity, are then defined. Finally, a numerical optimization problem is solved to minimize a function combining the costs of self-calibration and static calibration, subject to the defined constraints.

[0093] Advantageously, and according to the invention, the second method is used to update the internal parameters during the intervention. Steps 1 to 6 are identical to those of the first method. However, in this case, the current affine and distortion internal parameters are used as initial conditions for updating the parameter vectors. A numerical optimization problem is solved to produce an estimate of the current parameters. This optimization incorporates not only the self-calibration and static calibration cost functions, but also an additional term to weight the influence of the previous estimate on the update, using an appropriate distance function between vectors.

[0094] Advantageously, and according to the invention, a third possibility combines the two methods. In this configuration, a combined calibration is performed at the beginning of the procedure, and then the internal parameters are updated during the procedure. This approach ensures accurate initial calibration without requiring strong prior knowledge of the parameter values ​​and subsequently allows these parameters to be dynamically adjusted based on variations observed during the procedure, thus guaranteeing optimal calibration quality throughout. The parameters can be adjusted in real time using a time-domain filter, such as a Kalman filter, to continuously improve calibration accuracy based on previous parameter values ​​and newly acquired data.

[0095] List of figures

[0096] Other objects, features and advantages of the invention will become apparent from the following description, given by way of non-limiting example only, and which refers to the accompanying figures in which:

[0097] [Fig. 1] is a schematic view of a method for determining calibration parameters of an endoscopic camera according to an embodiment of the invention.

[0098] [Fig. 2A] is a schematic view of a pair of images acquired by an endoscopic camera of a scene showing two straight-edged tools and a plurality of points of interest in the form of point correspondences between said images.

[0099] [Fig. 2B] is a schematic view of a pair of images acquired by an endoscopic camera of a scene showing two straight-edged tools and a plurality of points of interest in the form of point correspondences between said images.

[0100] [Fig. 3] is a schematic view of a calibration system for an endoscopic camera according to an embodiment of the invention.

[0101] Detailed description of an embodiment of the invention

[0102] In the figures, the scales and proportions are not strictly respected, for the purposes of illustration and clarity.

[0103] In addition, identical, similar or analogous elements are designated by the same references in all figures.

[0104] Figure 1 schematically and partially represents a method for determining calibration parameters of an endoscopic camera according to an embodiment of the invention.

[0105] The process begins with a preliminary step (PI) of detecting pixels with straight edges of tools on the images of the video stream. This detection is performed by a curve detection method based on the RANSAC algorithm.

[0106] After the detection of the straight edge points, a subsequent step (P2) consists of fitting a geometric curve model of the B-Splines type to the detected points.

[0107] Following the geometric curve fitting step, a Cl step is performed to determine a plurality of constraints on the calibration parameters from the fitted curve models. These constraints are based on the mathematical expressions of the determined B-splines.

[0108] Next, a step El is performed to receive a plurality of pixel coordinates associated with at least one straight edge of the tool on at least one of the images of the video stream.

[0109] The received pixel coordinates are then used in step E2 to determine a static calibration cost function. This cost function quantifies the difference between the observed and expected coordinates according to a camera model, thus formalizing the calibration problem in mathematical terms.

[0110] The next step, E3, involves determining the calibration parameters of the endoscopic camera by solving an optimization problem. The cost function, or overall cost function, includes the static calibration cost function. By minimizing this cost function, the camera parameters are adjusted to best match the observed reality.

[0111] The process includes a step 1E1 for determining a self-calibration cost function from point correspondences between images in the video stream. This cost function is integrated into the overall cost function to further improve calibration accuracy by taking into account dynamic variations in the scene.

[0112] Finally, after determining the calibration parameters, the DI step consists of updating these parameters by solving an optimization problem whose cost function is a dynamic global cost function. This function includes a component dependent on the Mahalanobis distance between the current values ​​of the calibration parameters and the determined values. This allows for continuous and precise adaptation of the calibration parameters to the changing conditions of the camera's environment.

[0113] Figure 2A represents, schematically and partially, an IM1 image acquired by an endoscopic camera of a scene showing two straight-edged tools 100 and a plurality of points of interest in the form of point correspondences between said images.

[0114] Figure 2B represents, schematically and partially, an IM1 image acquired by an endoscopic camera of a scene showing two straight-edged tools 100 and a plurality of points of interest in the form of point correspondences between said images.

[0115] Images IM1 and IM2 in figures 2A and 2B represent two different views of the same endoscopic scene, taken at distinct times, but chronologically ordered.

[0116] The key points pl, p2, p3, p4, p5, and p6 are marked on both images. These key points are characteristic points automatically detected by the SIFT algorithm. These key points pl, p2, p3, p4, p5, and p6 establish correspondences between images IM1 and IM2 and allow for the creation of a self-calibration cost function.

[0117] The numbers 10 and 100 indicate the edges of the tools present in the scene. The primary edge of the tool is marked as 10, and its secondary edges as 100. These edges are detected by appropriate methods, such as edge detectors using ad hoc filtering or deep neural networks.

[0118] Point 201 represents a distance indicator between points, illustrating that the images are distinct and chronologically ordered. This indication helps to understand the variations in point position between successive images, which is important for monitoring and dynamically adjusting calibration parameters.

[0119] The set of points and edges detected in the images is then used to fit a curve model, such as a B-spline curve estimated by RANSAC. This model allows the points at the edges of the tools to be grouped and a static calibration cost function to be formed.

[0120] Figure 3 schematically represents a calibration system for the parameters of an endoscopic camera that acquires a video stream. This system is designed to improve the accuracy of the captured images by using an optimization process to calibrate the camera parameters.

[0121] The endoscopic camera 1 captures a continuous video stream 3 of the endoscopic scene. At least one image of the video stream (2) represents a scene comprising at least one tool (10), each tool comprising at least one straight edge (100).

[0122] The video stream images, captured by the endoscopic camera, show scenes with instruments and their straight edges. The images are transmitted to the computer for analysis and processing. The video stream represents the sequence of images captured by the endoscopic camera.

[0123] If desired, the video stream is continuously transmitted to the computer for real-time processing.

[0124] The computer centralizes the image processing and calibration. It receives images from the video stream and coordinates the various calculation steps necessary to determine the calibration parameters. Storage memory is used to store the video stream images as well as intermediate and final data related to the calibration process. The stored information includes the pixel coordinates of the straight edges of the tools, the calculated calibration parameters, and other relevant data.

[0125] The processing unit is capable of receiving a plurality of pixel coordinates associated with the straight edges of tools present in the video stream images. It uses these coordinates to determine the calibration parameters of the endoscopic camera by solving a constrained optimization problem. The cost function, known as the overall cost function, includes at least one static calibration cost function obtained from the received pixel coordinates.

[0126] The calibration parameters are the final results of the calculations performed by the processing unit. These parameters include corrections for optical distortions and other adjustments necessary to ensure that the images captured by the camera are accurate and faithful to the camera model.

[0127] In this system, the computational steps include establishing correspondences between images in the video stream, generating cost functions for self-calibration and static calibration, detecting edge points of the tools, and fitting curve models to the detected points. Finally, a numerical optimization problem is solved to minimize the cost functions under the defined constraints, allowing the determination of optimal calibration parameters for the endoscopic camera.

[0128] Figure 3 shows a workflow where images captured by the endoscopic camera are processed by a computer and a computing unit to determine the necessary calibration parameters. These parameters are then applied to correct distortions and improve the accuracy of the endoscopic images and their subsequent automated processing.

Claims

DEMANDS 1. Method for determining calibration parameters (7) of an endoscopic camera (1), acquiring a video stream (3), each image (2) of said video stream representing a scene, of which at least one image of said video stream displaying at least one tool (10) comprising at least one straight edge (100), the method being characterized in that it comprises the following steps: a step (E1) of receiving a plurality of pixel coordinates associated with at least one straight edge (100) of at least one tool (10) on one of the images (2) of said video stream (3); a step (E2) of determining a static calibration cost function from said plurality of received pixel coordinates; a step (E3) of determining calibration parameters of said endoscopic camera (1) by solving an optimization problem whose cost function, referred to as the overall cost function, comprises at least said static calibration cost function.

2. Method for determining calibration parameters of an endoscopic camera (1) according to claim 1, characterized in that the calibration parameters (7) of the endoscopic camera (1) include internal affine parameters and internal distortion parameters.

3. Method for determining calibration parameters (7) of an endoscopic camera (1) according to any one of claims 1 to 2, characterized in that it comprises a step (F1) of determining a self-calibration cost function from a plurality of point correspondences (p1; p2; p3; p4; p5) between images of said video stream (3); and in that the overall cost function comprises, in addition to said static calibration cost function, said self-calibration cost function.

4. A method for determining calibration parameters (7) of an endoscopic camera (1) according to claim 3, characterized in that said overall cost function comprises a cost function weighting parameter static calibration and / or said self-calibration cost function.

5. Method for determining calibration parameters of an endoscopic camera according to any one of claims 1 to 4, characterized in that it comprises a preliminary step (PI) of detecting straight edge points (100) of tools on the images of said video stream, the detection of straight edge points (100) of tools being carried out by a curve detection method.

6. Method for determining calibration parameters of an endoscopic camera according to claim 5, characterized in that it comprises a step (P2), consecutive to the step (PI) of detecting straight edge points (100) with tools on the images of the video stream, of adjusting a model of geometric curves.

7. Method for determining calibration parameters of an endoscopic camera according to claim 6, characterized in that it comprises a step (Cl), following the step (P2) of fitting a model of geometric curves, of determining a plurality of algebraic constraints on the calibration parameters from said fitted models of geometric curves.

8. Method for determining calibration parameters of an endoscopic camera according to any one of claims 1 to 7, characterized in that it comprises a step (D1), following the step (E3) of determining calibration parameters of said endoscopic camera, of updating the calibration parameters of said endoscopic camera from a cost function, said dynamic overall cost function, comprising said overall cost function on the one hand, and a cost function dependent on a distance between the current values ​​of the calibration parameters and said values ​​of calibration parameters to be determined on the other hand.

9. A method for determining the calibration parameters of an endoscopic camera according to claim 8, characterized in that the distance between the current values ​​of the calibration parameters and said calibration parameter values ​​is a Lebesgue distance L p or a distance of Mahalanobis.

10. System for determining calibration parameters of an endoscopic camera (1) acquiring a video stream (3); at least one image of the video stream (3) representing a scene, of which at least one image of said video stream displaying at least one tool (10) comprising at least one straight edge (100), characterized in that it comprises: an endoscopic camera; a computing unit capable of receiving a plurality of pixel coordinates associated with at least one straight edge (100) of at least one tool (10) on one of the images of said video stream, said computing unit being capable of determining calibration parameters of said endoscopic camera by solving an optimization problem, the cost function of which comprises at least one static calibration cost function obtained from said plurality of received pixel coordinates.

Images