Camera-Based Tracking System

The control device for optical tracking systems accelerates thermal stabilization by adjusting temperature and energy consumption modes, addressing the inaccuracy issues during startup, ensuring rapid and accurate marker tracking.

JP2024542234AActive Publication Date: 2024-11-13KONINKLIJKE PHILIPS NV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2024529870
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-12-03
Filing Date
2022-11-28
Publication Date
2024-11-13
Estimated Expiration
2042-11-28

AI Technical Summary

Technical Problem

Optical tracking systems in surgical navigation take a long time to warm up and stabilize, leading to temporary inaccuracies due to thermal expansion and changes in the geometric relationship between cameras, affecting the accuracy of marker tracking during cold starts.

Method used

A control device for a camera-based tracking system that includes a controller to generate calibration compensation instructions, adjusting the temperature and camera output to match predetermined conditions, using high energy consumption modes to accelerate thermal stabilization and improve accuracy during startup.

Benefits of technology

The solution reduces warm-up time and minimizes temporary accuracy degradation by ensuring the system reaches stable operating conditions faster, maintaining high precision in marker tracking.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024542234000001_ABST
    Figure 2024542234000001_ABST
Patent Text Reader

Abstract

To provide accurate tracking at an earlier stage, a control device for the operation of a camera-based tracking system is provided. The control device has a controller and a control signal output. The controller generates an operation signal for the tracking system with a calibration compensation instruction during a start-up phase of the tracking system. The calibration compensation instruction has instructions for at least one of i) actively adjusting a temperature of the camera-based tracking system, and ii) adapting a calibration-related camera output of the camera-based tracking system used in the tracking calculation. The control signal output provides the operation signal to the tracking system. In one approach, pre-heating is provided to achieve a (steady-state) calibration state by providing a control signal to the camera for operation in a high thermal energy generation mode. In another approach, calibration is provided as a dynamic calibration model that allows calibration for system conditions outside the steady state.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a control device for the operation of a camera-based tracking system, a camera-based tracking system, and a method for operating a camera-based tracking system. [Background technology]

[0002] Navigation systems are used in medical interventions to track instruments. For example, the instruments are provided with markers that are detected by the tracking system. An example of a tracking marker is the use of an optical tracking system, where a camera tracks a marker in the image data provided by the camera. As an example, as the surgeon moves the instrument, the position of the tracked instrument relative to the patient's anatomy is shown on the patient's image. The surgeon therefore uses the system to "navigate" the position of the instrument. The feedback of the instrument position provided by the system can be useful in situations where the surgeon cannot actually see the tip of the instrument, for example in minimally invasive surgery. Tracking devices are also provided in computer-assisted surgery (CAS) systems, which are increasingly used for pre-operative planning and for navigation of instruments during surgical procedures.

[0003] As an example, using a surgical navigation system, a surgeon uses special instruments that are tracked by the navigation system.

[0004] As a further example, US Patent Application Publication No. 2018 / 235715(A1) relates to bone and tool tracking in computer-assisted orthopedic surgery and robotic computer-assisted surgery. To this end, the tracking device may include a 3D camera for performing range imaging. Such range tracking requires that the 3D camera be calibrated to achieve appropriate accuracy and precision of tracking, for example using a calibration pattern.

[0005] Tracking or navigation systems may be moved between operating rooms. Clinical use and schedules may require that the system be easy to set up, start up quickly, and be ready to use within minutes. However, it has been shown that optical tracking systems used for surgical navigation can take as long as 30 minutes or even an hour to warm up and stabilize after a cold start. Because of this start-up behavior, the system's ability to track markers attached to the patient and surgical instruments may be temporarily compromised, leading to inaccurate results. Summary of the Invention [Problem to be solved by the invention]

[0006] Thus, for example, improved and accurate tracking at earlier stages may be required. [Means for solving the problem]

[0007] The object of the present invention is solved by the subject matter of the independent claims, further embodiments are incorporated in the dependent claims. It is noted that the below described aspects of the present invention also apply to an apparatus for operation of a camera-based tracking system, to a camera-based tracking system and to a method for operating a camera-based tracking system.

[0008] According to the present invention, a control device for the operation of a camera-based tracking system is provided. The control device comprises a controller and a control signal output. The controller is configured to generate at least one operating signal for the tracking system, comprising at least one calibration compensation instruction, at least during a start-up phase of the tracking system. The at least one calibration compensation instruction comprises instructions for at least one of the group: i) actively adjusting a temperature of the camera-based tracking system, and ii) adapting a calibration-related camera output of the camera-based tracking system, which is used for tracking calculations. The control signal output is configured to provide the at least one operating signal to the tracking system.

[0009] As an effect, compensation instructions are provided to at least partially compensate for discrepancies between a current state of a camera-based tracking system and predetermined conditions of the system assumed for calibration of the system, which improves the accuracy of the tracking system.

[0010] According to one example, the calibration compensation instructions include at least one control signal for temporarily operating the camera-based tracking system in an increased heat generation mode.

[0011] According to one example, to temporarily operate the camera-based tracking system in a high heat generation mode, the calibration compensation instructions have at least one control signal for temporarily operating at least one of the at least two cameras in a heating mode to generate heat by at least one of the at least two cameras. The at least one control signal is configured to operate at least one of the at least two cameras in a high energy consumption mode, in which a higher energy consumption by the camera occurs when the camera is operated for tracking than in a normal tracking mode. The control signal output unit is configured to provide at least one operating signal to the tracking system to operate at least one of the at least two cameras in the high energy consumption mode, thereby heating at least the camera for a higher energy consumption compared to the normal tracking mode.

[0012] According to one example, for a high heat mode, the calibration compensation instructions include at least one control signal for temporarily activating a heating element of a camera-based tracking system.

[0013] In one example, markers are used for tracking, while in another example, markerless tracking is provided.

[0014] This addresses the desirability, for optical tracking systems, of reducing the warm-up time and minimizing the temporary accuracy loss of the system during cold start. A further advantage is that the camera can be configured with an external computer. Moreover, this is based on the fact that the power consumption of the camera is directly related to the generation of heat by the camera and the subsequent dissipation of this heat to the parts of the system that are thermally connected to the camera (heat sink). More heat generation in the camera results in a more rapid temperature rise of the camera and the thermally connected components. It is noted that much of the thermal stabilization process is done inside the camera. This is addressed by the present solution without the need for major modifications to the camera, e.g. the camera housing expands and the sensor location moves. The warm-up of the camera to the level required for stable operation of the system can be further accelerated by using a set of camera parameters that demands more power consumption than is required for actual normal operation.

[0015] The tracking of markers, for example on the patient and on the equipment or instruments, is achieved by determining their exact spatial position relative to the tracking system. Knowledge of the spatial position of the tracking system in a local reference grid or spatial reference system in the operating room makes it possible to link and embed the markers in the local reference grid. The determination of the spatial position of the markers is essentially achieved by trigonometry. For example, the distances to the markers are measured from different sources, i.e. different cameras. By trigonometric calculations, the spatial position relative to the cameras can be determined. Trigonometric calculations make it possible to spatially align the images provided by the cameras. However, this requires precise knowledge of the actual (spatial) positions of the cameras and their (spatial) orientation. This precise geometrical knowledge is provided by the calibration of the camera system. To implement a camera system, several cameras are provided, which must be fixed to a reference. For this reason, the cameras are physically mounted on a common support structure. Since the support structure constantly undergoes thermal expansion, depending on the material, design and size, there are always geometric changes that affect the position and orientation of the camera and, as a result, the accuracy of detecting the markers. The thermal expansion is mainly provided by the internal heat generated by the operation of the camera. The heat results in thermal expansion of the camera itself (optical system relative to the image detection element), the mount by which the camera is fixed to the support structure, and the support structure itself. As a result, a calibration of the system is provided that takes into account conditions that are assumed to be stable. Such conditions are taken with respect to the situation during normal operation of the tracking system, i.e., when the system is in use.

[0016] The present invention addresses this by providing a means to match the current conditions with the conditions assumed for the calibration: by providing additional heat, the system conditions better match the assumed conditions applied for the calibration.

[0017] According to one example, the controller is configured to take into account measured external conditions for the control signal when generating the operating signal, the external conditions including at least one of the following group: ambient temperature, lighting conditions, and indoor airflow.

[0018] According to one example, the controller is configured to provide a dynamic calibration model for the camera-based tracking system. The dynamic calibration model includes a plurality of calibration coefficients depending on a temperature-related state of the system. For a current temperature-related state of the camera-based tracking system, the controller is also configured to determine a matching calibration coefficient from the plurality of calibration coefficients. Further, the control signal output is configured to provide the matching calibration coefficient for an application during operation of the camera-based tracking system.

[0019] In one example, the controller is configured to provide calibration compensation instructions including operational instructions for operating at least one of the at least two cameras in a high energy consumption mode. The controller is also configured to provide a dynamic calibration model for the camera-based tracking system and to determine and provide matching calibration coefficients. Furthermore, the controller is further configured to operate the at least two cameras in an imaging mode while operating the cameras in the high energy consumption mode. The controller is further configured to apply the matching calibration coefficients to the generated images.

[0020] According to the present invention, there is also provided a camera-based tracking system, the system having a plurality of at least two cameras and a control device according to one of the previous examples, wherein at least one operating signal is provided for operating at least one of the at least two cameras.

[0021] Optionally, the camera-based tracking system is a camera-based medical tracking system for use in a hospital or other medical related setting.

[0022] According to one example, the high energy consumption mode includes image acquisition with energy consumption at least 50% higher than the energy consumption during a normal tracking mode of the system.

[0023] Optionally, the high energy consumption mode includes image acquisition having at least one of the following groups that is at least a 50% increase compared to the system's normal tracking mode: frame rate, exposure time, bit depth, bit packing, region of interest, clocking frequency, color format, image compression, and on-board image pre-processing.

[0024] According to one example, a set of markers that can be tracked by the system is provided, the markers being attachable to at least one of the following groups: a subject, an object, and a device.

[0025] In another example, markerless tracking is provided.

[0026] According to the present invention there is also provided a method for operating a camera-based tracking system, the method comprising the following steps. generating, at least during a start-up phase of the tracking system, at least one operating signal for the tracking system, the at least one calibration compensation instruction having instructions for at least one of the group of: i) actively adjusting a temperature of the camera-based tracking system, and ii) adapting a calibration-related camera output of the camera-based tracking system that is used in the tracking calculations. Providing at least one motion signal to a tracking system. · Operating the camera-based tracking system based on an operating signal including at least one calibration compensation command.

[0027] According to one aspect, tracking errors due to calibration conditions that have not yet been performed are minimized by providing countermeasures to compensate for the tracking errors.

[0028] According to one approach, pre-heating is provided to achieve a (steady) calibration state (applied for system calibration) by providing a control signal to the camera for operation in a high heat energy generation mode. Thus, existing equipment is used to generate heat. Thus, the heat energy is provided exactly where it has the greatest effect, i.e. at the camera position. The pre-heating mimics the heat generated during normal operation, but in a faster boost mode, for example.

[0029] According to another approach, instead of trying to achieve a steady state by pre-heating, which forms the basis of a static calibration, the calibration is provided as a dynamic calibration model which allows calibration also for states of the system outside the steady state: even if the operating temperature has not yet been reached, a respectively adapted calibration is provided by the dynamic model.

[0030] According to a further approach, combining the above two approaches, a dynamic model is provided in combination with a heating procedure with a special operating signal for the camera. In combination with an optional temperature sensor, the accuracy can be further improved.

[0031] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter.

[0032] Exemplary embodiments of the invention are described below with reference to the following drawings: [Brief description of the drawings]

[0033] [Figure 1] 1 illustrates a schematic of an example controller for operation of a camera-based tracking system. [Diagram 2] 1 illustrates an example of a camera-based tracking system in the context of a catheterization lab. [Figure 3a] 1 illustrates an example of a camera-based tracking system within a housing structure. [Figure 3b]3b shows an example of the camera-based tracking system of FIG. 3a without the housing structure. [Figure 4] 4 illustrates steps of an example method for operating a camera-based tracking system. [Diagram 5] 1 is a graph showing tracking error versus time after start. [Figure 6] 1 is a graph showing temperature of the system versus time after initiation. [Figure 7] 4 shows a graph illustrating system temperature over time for different operating modes. [Figure 8] 1 shows a graph illustrating an example warm-up pattern for system temperature and camera power consumption over time. [Figure 9] 13 shows a graph illustrating another example of a warm-up pattern for system temperature and camera power consumption over time. [Figure 10] 13 shows graphs illustrating further examples of warm-up patterns for system temperature and camera power consumption over time. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] Certain embodiments will now be described in detail with reference to the accompanying drawings. In the following description, like drawing reference numbers are used for like elements in different drawings. Matters defined herein, such as detailed configurations and elements, are provided to aid in a comprehensive understanding of the exemplary embodiments. Also, well-known functions or configurations will not be described in detail since they would obscure the embodiments in unnecessary detail. Furthermore, phrases such as "at least one of," when preceding a list of elements, modify the entire list of elements and do not modify individual elements of the list.

[0035] Surgical navigation systems that rely on optical cameras may have an inherent accuracy of marker tracking in the range of 0.5 to 1 mm. The inherent 3D marker tracking accuracy of the system is optimal (minimum) when the system is in the same "state" as when it was calibrated. If the system is in a different state, this results in less accurate marker tracking and therefore larger 3D errors. An example is that the relative geometric position and / or orientation of the optical cameras has changed due to mechanical forces (shock, vibration). This may require recalibration of the system. Furthermore, when using optical camera systems, temperature is a significant factor that can introduce changes in the actual image that result in additional tracking errors of the same order of magnitude as the desired accuracy. As an example, during start-up of the navigation system, tracking accuracy can be temporarily degraded when the hardware is in a "different state" due to temperature-related effects. There are two main effects that cause measurable thermally induced errors. First, the frame suspending the optical cameras undergoes some degree of temperature-related expansion, causing a temporary different geometric relationship between the cameras. Second, thermally related changes in the camera housing and lens system cause measurable differences in the projection of the image on the camera imaging sensor. Both effects contribute to a temporary decrease in accuracy as the calibration is performed when the system is in thermal steady state and while the system is in (temporarily) different transient states.

[0036] Generally, the start-up of an optical tracking system can be divided into two phases. In the first phase, also called phase 1, the system warms up. At the start, the temperature of the camera is close to the ambient temperature of the system. It increases due to the heat dissipation of the active components, e.g. the camera processor, the imaging chip, etc. The temperature-related tracking error is maximum at the start of this phase and minimum at the end of this phase. In the second phase, also called phase 2, the steady state of the system is reached. The hardware reaches thermal equilibrium and the optical tracking accuracy is at the same level as in the calibration state.

[0037] In one example, the thermal behavior of the optical tracking system may be more complex, since there may be a certain delay in the heat transfer from the camera to the other parts of the system that are thermally connected and act as a heat sink. Furthermore, preferably, the warm-up behavior of the navigation system is such that the thermal response is minimal and the time required to reach a steady state after start-up is minimal. This solution does not necessarily require smart hardware selection and / or design of the system, thus avoiding additional construction-related costs. Another aspect for triggering calibration to affect changes can be provided by the difference in ambient temperature, for example, by moving the camera from a hot storage room to an air-conditioned operating room.

[0038] When an optical tracking system is warming up from its surrounding ambient temperature, its internal components inevitably change size and shape, resulting in start-up behavior. The present solution speeds up the warm-up of the optical tracking system to a steady-state level and reduces temperature-related differences in tracking accuracy, so that tracking can be performed with an optimal level of accuracy sooner (faster) after start-up without adding hardware and / or hardware complexity to the system.

[0039] FIG. 1 shows a schematic diagram of an example of a control device 10 for the operation of a camera-based tracking system. The control device comprises a controller 12 and a control signal output 14. The controller 12 is configured to generate at least one operating signal for the tracking system, including at least one calibration compensation instruction, at least during a start-up phase of the tracking system. The at least one calibration compensation instruction comprises instructions for at least one of the following groups: i) actively adjusting the temperature of the camera-based tracking system, and ii) adapting a calibration-related camera output of the camera-based tracking system, which is used for tracking calculations. The control signal output 14 is configured to provide at least one operating signal to the tracking system for operating the camera-based tracking system.

[0040] In an option indicated in Fig. 1 by a dashed line, a signal input 16 is provided. The signal input 16 is configured to receive a signal for activating a camera-based tracking system having at least two cameras. Based on the activation signal, the controller is configured to generate at least one operating signal.

[0041] Thus, at least one calibration compensation command is provided at system start-up to allow faster use of the system.

[0042] In another option, the signal input 16 is configured to receive a signal for ongoing operation of the camera-based tracking system. Based on the ongoing operation signal, the controller is configured to generate at least one operation signal as a check of the calibration.

[0043] Thus, at least one calibration compensation instruction may be provided upon start-up of the system and during use of the system to enable improved accuracy, for example, during operation in changing environmental conditions.

[0044] A first arrow 18 indicates a signal for activating a camera-based tracking system. A second arrow 20 indicates the output of at least one operating signal, for example to a camera-based tracking system, indicated by a dotted frame 22. A further frame 24 indicates that the controller 12, the control signal output 14 and the signal input 16 can be provided within a common structure, such as a housing. However, they can be located separately but data connected.

[0045] Compensation instructions are provided to at least partially compensate for differences between a current state of the camera-based tracking system and a predetermined state of the system assumed for calibration of the system.

[0046] Active regulation of the temperature of the camera-based tracking system is provided to at least partially compensate for the discrepancy.

[0047] Adaptation of the camera output is provided to at least partially compensate for the difference.

[0048] The control signal output unit 14 is configured to provide at least one operating signal to the camera-based tracking system to operate the camera-based tracking system based on the operating signal including at least one calibration compensation instruction, thereby reducing the discrepancy between a current state of the camera-based tracking system and a predetermined state of the system assumed for calibration of the system.

[0049] Calibration compensation instructions, which may also be referred to as calibration correction instructions, are provided to adjust or modify system settings related to calibration.

[0050] Calibration compensation instructions are provided to minimize the offset between the assumed state on which the calibration is based and the current state of the system.

[0051] As an effect, the system is provided with improved accuracy even in situations where a static calibration state has not yet been achieved, such as during the start-up phase.

[0052] The start-up phase refers to that part of the operation of the system from the start of operation until the predetermined conditions of the system that are assumed for the calibration of the system are reached.

[0053] In a first option, not shown in further detail in FIG. 1, the calibration compensation instructions include at least one control signal for temporarily operating the camera-based tracking system in a high heat generating mode.

[0054] In an option not shown in further detail in FIG. 1 , the calibration compensation instructions include at least one control signal for temporarily operating at least one of the at least two cameras in a heating mode to generate heat by at least one of the at least two cameras. The at least one control signal is configured to operate at least one of the at least two cameras in a high energy consumption mode, in which a higher energy consumption by the camera occurs than in a normal tracking mode when the camera is operated to track the marker. The control signal output unit 14 is configured to provide at least one operating signal to the tracking system for operating at least one of the at least two cameras in a high energy consumption mode, thereby heating at least the camera for a higher energy consumption compared to a normal tracking mode.

[0055] The difference may also be referred to as an inconsistency.

[0056] In one option, the heating mode is provided by operating a (already existing) camera in a heating mode to generate thermal energy.

[0057] In another option, the heating modes are provided by operating a separate heating element to generate thermal energy. A separate heating element is provided with the tracking system. The heating element can be provided with a mount or support structure for the camera, by a separately attached heating element, or by a heating element integrally formed with the camera. The heating element can also be provided by identifying an operational element, such as a circuit within the system, that can be operated in a specified thermal energy generation mode.

[0058] In one example, when in a high heat generation mode, the calibration compensation instructions include at least one control signal for temporarily activating a heating element of a camera-based tracking system.

[0059] In one example, the heating element includes at least one integral heating element, for example, provided within the camera mount, to provide thermal energy at a location where the camera, in an operational mode, transfers thermal energy to the support structure.

[0060] In another example, an integral heating element is provided within the camera adjacent to the location of the electrical circuitry that generates heat when the camera is in its operational mode.

[0061] An example of such an additional heating element is a resistive stripe attached to the existing structure. Thus, targeted heat generation (and pre-heating or warm-up) is possible with less energy consumption compared to heating the complete apparatus of the system, for example, in the warm-up area of ​​the operating room or even in a separate warm-up room.

[0062] In one example, higher energy consumption is provided by increased heat generation due to operation: for example, a particular part of an electrical circuit is activated which uses the same energy as other parts during normal operation, but this part or component generates more temperature, i.e. more heat available for dissipation and heating of the system.

[0063] As an effect, heat (to reach the operating conditions on which the calibration is based) is generated exactly where it should be generated. As an advantage in terms of economy and reduced system complexity, this is achieved by using hardware that is already available. Another advantage results in the functionality of the tracking system being maintained. Even at the start of the heating process, the tracking may not be very accurate, and the system is available for other purposes where this may not be an issue, for example already providing visualization of the area of ​​interest to allow the surgeon to place the interventional device in a convenient place while maintaining a proper line of sight, etc.

[0064] The term "activate" relates to the initiation of operation of the tracking system when the system is started, e.g. converted or transitioned from OFF mode to ON mode. This is also referred to as initiation. Activation or initiation also includes transitioning the system from standby or sleep mode to ON mode. ON mode is the mode in which the system is ready for use, i.e. for normal, specified operation. ON mode can also be referred to as an operating mode. It should be noted that the term operation relates to the system in this context and does not necessarily imply that a (medical) operation is performed.

[0065] "High energy consumption mode" refers to a mode that results in higher energy consumption of the camera than in the absence of an operating signal for the high energy consumption mode. The high energy consumption mode can also be referred to as a higher energy consumption mode. Energy consumption can also be referred to as the use of (electrical) power.

[0066] The "high energy consumption mode" may also be referred to as an artificial or triggered warm-up mode, or a boosted warm-up mode.

[0067] As a result, by operating in a high energy usage mode, the warm-up period is reduced.

[0068] In the present context, the term "normal operation" relates to system settings that are applied when producing normal images with the camera, for example for tracking purposes.

[0069] The term "normal operation" relates to requirements such as camera frame rate, i.e. images per second, etc. As an example, in order to "smooth" the surgical navigation task, according to current requirements, cameras are adjusted to provide at least 12 frames per second (fps).

[0070] In one example, the term "high energy consumption mode" refers to the camera being operated "harder" during warm-up, for example at 30 frames per second, which significantly reduces the warm-up time.

[0071] In one example, an optical tracking system is provided in which during start-up, an optical camera in the system is configured to consume more power than during normal operation. This is achieved by setting camera parameters, such as frame rate, acquisition time, color format, image compression, on-board image pre-processing, camera timing, to values / states that require more on-board image processing and therefore more power consumption. More power consumption results in more heat dissipation, which leads to a quicker warm-up of the camera and thermally connected components than would be the case with a lower power consumption setting. Once a temperature corresponding to that normally reached with the low power consumption setting is reached, the camera settings are adjusted accordingly. As a result, less time is required to bring the optical system to operating temperature.

[0072] In one example, not shown in detail in FIG. 1, a control device for operation of a camera-based tracking system is provided. The control device has a signal input, a controller, and a control signal output. The signal input is configured to receive a signal for activating a camera-based tracking system having at least two cameras. Based on the activation signal, the controller is configured to generate at least one operation signal for the tracking system, the operation signal having at least one control signal for temporarily operating at least one of the at least two cameras in a heating mode generating heat by at least one of the at least two cameras. The at least one control signal is configured to operate at least one of the at least two cameras in a high energy consumption mode, in which a higher energy consumption by the camera is performed than in a normal tracking mode when the camera is operated to track a marker. The control signal output is configured to provide at least one operation signal to the tracking system for operating at least one of the at least two cameras in the high energy consumption mode, thereby heating at least the camera for a higher energy consumption compared to the normal tracking mode.

[0073] In one example, the controller is configured to provide an operational signal to operate the at least one camera in a high energy consumption mode until a predetermined thermal condition is reached.

[0074] Optionally, the controller is configured to provide an operating signal to maintain the high energy consumption mode until at least one of the group of at least one camera and the system reaches its normal operating temperature, which is the basis for the calibration of the system.

[0075] In another option, the controller is configured to provide an operational signal to provide a higher energy consumption mode beyond the point at which the at least one camera reaches its normal operating temperature.

[0076] In one example, the controller is configured to provide a normal operation signal for operating the at least one camera under normal conditions after heating the at least one camera by operation of the at least one camera of the at least two cameras in the high energy consumption mode.

[0077] Optionally, the controller is configured to modify settings for normal operation based on the difference between the system temperature at the time of calibration and the expected steady state temperature of the system under real-time conditions.

[0078] In one example, not further shown in Figure 1, the controller is configured to take into account measured external conditions for the control signal when generating the operating signal, the external conditions including at least one of the following group: ambient temperature, lighting conditions, and indoor airflow.

[0079] In one example, the external temperature conditions are measured and taken into account: for example, if the ambient temperature is high, the thermal equilibrium temperature of the optical system will also be high, in which case the threshold temperature for switching from high power to low power consumption should be adjusted accordingly, for example sooner.

[0080] In one example, lighting conditions are measured and taken into account: the brightness of the lighting can cause a higher (2-3° C.) steady-state camera temperature, and the brightness can be directly subtracted from the images recorded by the camera, thus serving as an additional input; similarly, if the external lighting cannot be controlled, the brightness of the captured images can still be adjusted via the camera parameters (e.g., exposure time).

[0081] In one example, indoor airflow is taken into account, also referred to as airflow, and is either provided and sensed or provided as a known parameter for the supply of clean air or for cooling or heating purposes, and air movement also affects the camera due to convection effects.

[0082] In one example, a signal input 16 is provided configured to provide a measured external condition.

[0083] In one example, not further shown in Fig. 1, to generate at least one operating signal for the tracking system, the controller is configured to provide a model of the warm-up of the system and its dynamic thermal behavior and to take these into account for the at least one control signal, the at least one control signal having a signal sequence with a plurality of varying control sub-signals.

[0084] In one example, the dynamic thermal behavior of the system is taken into account by using a model of the system's warm-up: rather than simply switching from a high power consumption to a low power consumption setting based on a threshold temperature, a priori knowledge of the system thermal behavior can be used to fine-tune the exact moment at which camera parameters that affect power consumption change during warm-up.

[0085] In another option, not shown in further detail in FIG. 1 , the controller 12 is configured to provide a dynamic calibration model for the camera-based tracking system. The dynamic calibration model has a number of calibration coefficients depending on a temperature-related state of the system. For a current temperature-related state of the camera-based tracking system, the controller 12 is configured to determine a matching calibration coefficient from the number of calibration coefficients. The control signal output 14 is configured to provide the matching calibration coefficient for application during operation of the camera-based tracking system.

[0086] In a basic version of this option, the current temperature related state of the camera-based tracking system is information regarding activation of the system switching from an OFF state to an ON state. In another version, information regarding the time period since the system was activated is provided. In a further version, one or more temperature measurements are provided, such as temperature readings from internal and / or external sensors.

[0087] In a further option, the controller 12 is configured to provide calibration compensation instructions having operational instructions for operating at least one of the at least two cameras in a high energy consumption mode. The controller 12 is also configured to provide a dynamic calibration model for the camera-based tracking system and to determine and provide matching calibration coefficients. The controller 12 is further configured to operate the at least two cameras in an imaging mode while operating the cameras in the high energy consumption mode. The controller 12 is configured to apply the matching calibration coefficients to generated images.

[0088] In one example, a signal input 16 is provided that is configured to provide a current temperature-related state of the camera-based tracking system. The controller is configured to provide a dynamic calibration model for the camera-based tracking system. The controller is also configured to determine a matching calibration factor from the plurality of calibration factors. The control signal output is configured to provide the matching calibration factor for application during operation of the camera-based tracking system.

[0089] In one example, also referring to the first and second options above, the controller 12 is configured to provide a calibration compensation instruction having an operation instruction for operating at least one of the at least two cameras in a high energy consumption mode. The controller 12 is also configured to provide a dynamic calibration model for the camera-based tracking system and to determine and provide matching calibration coefficients. The controller 12 is further configured to operate the at least two cameras in an imaging mode while operating the cameras in the high energy consumption mode. The controller 12 is further configured to apply the matching calibration coefficients to the generated images.

[0090] 2 shows an example of a camera-based tracking system 50 in the context of a catheterization lab. The camera-based tracking system 50 has a plurality of at least two cameras 52. An example of the control device 10 according to one of the above and below examples is provided. At least one operating signal is provided to operate at least one of the at least two cameras.

[0091] Optionally, the camera-based tracking system 50 is a camera-based medical tracking system for use in a hospital or other medical related setting.

[0092] In another option, the camera-based tracking system 50 is a camera-based tracking system for use in a non-medical setting.

[0093] In one example, a medical optical tracking system is provided that has a camera-based system capable of imaging a medical device / object and also has a means for directly or indirectly measuring the transient warm-up period.

[0094] 2 shows four cameras 52 mounted on a ring-like support structure 54. A data connection 56, shown diagrammatically, connects the four cameras and the controller 10. For example, the controller 10 is connected to a console 58 provided for operation of different equipment in the catheter lab. By way of example, an X-ray imaging system 60 is provided with a C-arm 62 having an X-ray source 64 and an X-ray detector 66 mounted on either end of the C-arm 62. The C-arm 62 is movably mounted on a ceiling rail structure. Furthermore, a subject support 68, also referred to as a patient table, is provided for receiving a subject 70. Furthermore, a display device 72 is shown.

[0095] 3a shows an example of a camera-based tracking system 50 in a housing structure 74 having four cameras 52a, 52b, 52c, and 52d. It should also be noted that fewer cameras, e.g., two or three, or more cameras, e.g., five, six, seven, eight, nine, ten or more cameras, can be provided.

[0096] Figure 3b shows the example of Figure 3a without the housing structure 74. Shown is a ring support 76 to which the four cameras 52a, 52b, 52c and 52d are attached. Further, a connection flange 78 is also provided. The lines 80 below the cameras indicate their overlapping fields of view.

[0097] In one example, the high energy consumption mode has image acquisition with energy consumption at least 50% higher than the energy consumption during a normal tracking mode of the system. Optionally, the high energy consumption mode is provided having image acquisition with at least one of the following group increased by at least 50% compared to the normal tracking mode of the system: frame rate, exposure time, bit depth, bit packing, region of interest, clocking frequency, color format, image compression, and on-board image pre-processing.

[0098] The exposure time can also be referred to as the acquisition time.

[0099] Image compression may also involve debayering.

[0100] As an example, the increased frame rate is at least 50% higher than the frame rate used in the normal tracking mode of the system.

[0101] In one example, the increased frames are (1+1 / 2) times the normal tracking frame rate.

[0102] In one example, the high energy consumption mode includes image acquisition with energy consumption that is at least (2+1 / 2) times the energy consumption during a normal tracking mode of the system.

[0103] In one example, the increased frame rate is provided as more than twice the normal tracking frame rate, for example, (2+1 / 2) times the normal tracking frame rate.

[0104] In one example, at least one of the following groups is provided in a double or (2+1 / 2) times amount: exposure time, bit depth, bit packing, region of interest, clocking frequency, color format, image compression, and on-board image pre-processing.

[0105] In one example apparatus or system, the control signal is configured to cause at least one camera to be operated in a high energy consumption mode until a predetermined thermal condition is reached.

[0106] Optionally, the control signal is configured such that the high energy consumption mode is maintained until at least one of the group of at least one camera and the system reaches a normal operating temperature, which is the basis for the calibration of the system.

[0107] As a further option, the control signal is configured such that a higher energy consumption mode is provided beyond the point at which the at least one camera has reached its normal operating temperature.

[0108] In one example of the device or system, the control signal is configured such that after heating at least one of the at least two cameras by operating the at least one camera in a high energy consumption mode, a normal operation signal is provided such that the at least one camera is operated under normal conditions.

[0109] Optionally, the control signal is configured such that the settings for normal operation are modified based on the difference between the system temperature at the time of calibration and the expected steady state temperature of the system under real-time conditions.

[0110] In one example of the device or system, the controller for generating at least one operating signal for the tracking system is configured to provide a model of the warm-up of the system and its dynamic thermal behavior and to take these into account for the at least one control signal. Optionally, it is provided that the at least one control signal has a signal sequence with a plurality of varying control sub-signals.

[0111] In one example, at least one temperature sensor is provided for monitoring a camera temperature, the camera temperature being taken into account for generating the operating signal, and optionally the controller is configured to generate an updated operating signal.

[0112] In one example, the camera's built-in temperature sensor is used to monitor the camera temperature, which can be read out using the existing computer-camera communication interface without the need for an additional temperature measurement system.

[0113] In one example, not shown, the controller 12 is configured to modify the operating signal based on at least one of the group of sensor data and system conditions.

[0114] For example, an internal temperature sensor measures the system temperature and the controller then determines whether additional heating would be beneficial.

[0115] For example, if the system is turned off for only a few minutes to move from one operating room to the next and is still close to the steady state temperature assumed for calibration, no further additional heating may be required.

[0116] For example, if the system is stored in an unconditioned storage room during a heat wave, no additional heating may be required.

[0117] In another option, a temperature sensor provides temperature readings throughout normal operation of the system, and the controller compares the readings to a threshold below which the controller provides for the initiation of an intermediate warm-up procedure in which at least one of the at least two cameras is operated in a high energy consumption mode, thereby heating the camera for higher energy consumption compared to a normal tracking mode. This ensures that the system is operating in conditions that are the basis for calibration.

[0118] In one example, not shown, a number of sensors 84 (see FIG. 2) are provided that monitor the thermal behavior of the system. The thermal behavior monitored by the sensors is used to provide a more accurate model of the system warm-up.

[0119] Optionally, a model of the system warm-up is provided that is used in combination with multiple temperature readings from the sensor during warm-up to dynamically adjust the system calibration.

[0120] Optionally, the temperature history of the system is provided and the time remaining is predicted and optionally shown before the system reaches the normal operating stage.

[0121] In one example, a model of the thermal behavior of the system combined with temperature readings during warm-up and normal operation is used to dynamically adjust the optical calibration of the system. In one example, an optical calibration look-up table is created by calculating the optical calibration (internal and external camera parameters) at various times during system warm-up under various ambient temperature conditions. Optionally, this look-up table is then used to dynamically adjust the optical calibration of the system based on the thermal conditions of the system.

[0122] In one example, multiple temperature sensors are used at various locations on the optical tracking system so that a more accurate model can be applied to the dynamic thermal behavior of the system. Within the camera, the temperature changes most dramatically (typically up to 70°C ambient temperature) and rapidly (as heat is generated within the camera), while the camera housing already exhibits a significantly different temperature range (up to 50°C ambient temperature) and delayed warm-up. The warm-up of other components of the optical tracking system that are thermally connected to the camera (e.g., camera mount, frame) is delayed and the thermal equilibrium temperature will be lower in the absence of other heat sources. Using multiple sensors to monitor the temperature of various parts of the system helps to build a comprehensive thermal behavior model of the system and to more accurately fine-tune the warm-up settings.

[0123] In one example, the camera-lens system is thermally modeled. In another example, the complete device, i.e., system, frame, cover and camera, is thermally modeled. Based on the modeling, it is possible to determine the response (transfer function) of the system without having a priori knowledge of the system. In one example, the main inputs to the model are a combination of the signal from the temperature sensor, the camera imaging parameters, e.g., frame rate, system power consumption and image-based parameters, e.g., image brightness.

[0124] In one example, the temperature history is provided as a current temperature history, which may also be referred to as a thermal history.

[0125] In one example, the thermal state of the system based on temperature history is used to predict the time remaining before the system can be used for normal operation. Preferably, that information can be provided as feedback, for example as a countdown timer, progress bar, etc.

[0126] Another option is to provide a modified set of parameters after the start-up phase, for example, the system is operated at a significantly lower ambient temperature than during calibration, and therefore provide for the camera to run, i.e. operate, at a slightly increased frame rate to compensate for the lower ambient temperature.

[0127] In an embodiment not shown, the controller is configured to analyze at least two images from different times during warm-up to determine the current displacement speed and adjust the high energy consumption mode accordingly.

[0128] In one example, a set of markers 82 is provided that can be tracked by the system, as shown as an option in Figure 2. The markers 82 can be attached to at least one of the following groups: a subject, an object, and a device.

[0129] The markers 82 are also referred to as fiducials.

[0130] The present solution avoids the high cost of navigation systems. Moreover, the clinical adoption of surgical navigation systems is also supported, since the use of navigation technology is encouraged and it fits well into clinical routine due to its accelerated warm-up. Such medical, e.g., surgical navigation systems work quickly and easily with simplified procedures, and therefore do not complicate the workflow too much. The navigation system is also suitable to be moved between operating rooms. The system is easy to set up, quickly starts up, and is ready for use within minutes.

[0131] Applying the above operating scheme results in a particular energy consumption pattern, which can be detected, for example, by measuring the power consumption of the cameras in a given optical tracking system, for example using a USB power meter or by measuring voltages and currents in the hardware.

[0132] In one example, a temperature measurement system is provided to measure the temperature of the camera and / or other parts of the system.

[0133] In one example, a processor is provided for analyzing the at least two images and determining a current displacement velocity.

[0134] In one example, a timer is provided to monitor the length of the warm-up period.

[0135] In one example, a processor is provided that is capable of processing transient warm-up period measurement information, configuring the camera, and receiving and processing image data from the camera.

[0136] It is provided that thermal stabilization of the optical tracking system is accelerated by configuring the camera to consume more power during warm-up than during normal operation, and once the desired thermal conditions are reached, the camera is configured to provide the desired performance with the lowest possible power consumption.

[0137] Regarding the warm-up of an optical tracking system, roughly two phases can be distinguished: in phase 1, the temperature of the system rises rapidly due to self-heating of active components in the camera; in phase 2, the system is approaching or reaching thermal equilibrium. Phase 2 may be provided for longer (several hours) than phase 1 (without sustained heating, between 30 minutes and an hour, significantly shorter with sustained (boosted) heating).

[0138] In one example, two fixed sets of image acquisition parameters are used: a first setting associated with high power demands on the camera to heat it, and a second setting associated with normal image acquisition by the camera.

[0139] In one example, during the first stage of camera warm-up, a high power setting is used (FIG. 8, curve I) to speed up the camera warm-up to a temperature level that would have been reached using the normal power setting (FIG. 8, curve II). Once the preset temperature is reached, the system switches to the settings required for normal image acquisition by the camera.

[0140] In one example, a machine vision camera can be controlled by setting image acquisition parameters such as frame rate, exposure time, binning, bit depth, bit packing, region of interest, clocking frequency, and level of on-board image pre-processing, e.g., image compression, debayering. An example is shown in Figure 7. The warm-up time of the camera imaging chip and the camera housing are accelerated by about 3.5 times and about 1.4 times, respectively, by using a higher frame rate for the first part of start-up.

[0141] In one example, similar to the previous example, except that the image acquisition parameters for normal use of the camera do not have fixed values, once phase 2 is reached, the parameters are dynamically changed until a temperature close to or equal to the temperature at the moment of calibration is reached. The main advantage of this feature is that the system can to some extent mitigate the effects of external conditions (e.g., ambient temperature, exposure to other heat sources such as surgical lights, etc.).

[0142] In a further example similar to the previous example, except that image acquisition parameters for normal use during stage 2 are dynamically adjusted, and external conditions, e.g., ambient temperature, acquisition parameters are varied to maintain a system temperature close to the calibration temperature.

[0143] A further example is provided that is similar to the previous example, except that the system uses a model of the thermal behavior of the system to predict how to adjust image acquisition parameters so that a desired thermal stage / system temperature can be reached and maintained. Preferably, the system is self-learning and adjusts its model based on recorded data of previous warm-up cycles and / or based on input from various sources, e.g., temperature sensors, imaging sensors.

[0144] In another example, similar to the previous example, except that the camera is modeled and controlled according to minimum time control (or optimal control), the system reaches equilibrium with only two control actions (e.g., a transient as fast as possible). In such a case, the camera is initially configured to consume as much power as possible, and once a state is reached, the camera is configured to consume as little power as possible until a desired state (steady state) is reached. The camera is then configured for the steady state.

[0145] 4 shows the steps of an example of a method 100 for the operation of a camera-based tracking system. The method 100 comprises the following steps: In a first step 102, at least one operating signal is generated during a start-up phase of the tracking system, the at least one calibration compensation instruction comprising at least one instruction from the group of: i) actively adjusting a temperature of the camera-based tracking system, and ii) adapting a calibration-related camera output of the camera-based tracking system, which is used for the tracking calculations. In a second step 104, at least one operating signal is provided to the tracking system. In a third step 106, the camera-based tracking system is operated based on an operating signal having at least one calibration compensation instruction.

[0146] Actively tuning a camera-based tracking system involves adjusting the system in its geometric setup. Active tuning modifies the actual configuration settings. Adapting the camera output involves modifying the image processing with respect to the spatial alignment of the images provided by the camera. Adapting the camera output modifies the virtual setup provided by the calibration.

[0147] In one example, indicated by the hashed lines in FIG. 3, at an initial step 108, a signal is received to activate a camera-based tracking system.

[0148] In one example, a method option is provided in which generating at least one operating signal for the tracking system includes generating at least one control signal for temporarily operating at least one of the at least two cameras of the system in a heating mode to generate heat by at least one of the at least two cameras. The at least one control signal is configured to operate at least one of the at least two cameras in a high energy consumption mode in which a higher energy consumption by the camera occurs than in a normal tracking mode when the camera is operated to track the marker. In this example, operating the camera-based tracking system includes operating at least one of the at least two cameras in a high energy consumption mode, thereby heating at least the camera with a higher energy consumption compared to the normal tracking mode.

[0149] A reduction in the discrepancy between the current state of the camera-based tracking system and the predetermined conditions of the system assumed for the calibration of the system is provided by heating of at least one camera with higher energy consumption compared to a normal tracking mode.

[0150] In one example, a method for operating a camera-based tracking system is provided, the method comprising: receiving a signal to activate a camera-based tracking system having at least two cameras; generating at least one operating signal for the tracking system, the at least one operating signal having at least one control signal for temporarily operating at least one of the at least two cameras in a heating mode to generate heat by at least one of the at least two cameras based on the activation signal; the at least one control signal is configured to operate at least one of the at least two cameras in a high energy consumption mode, resulting in higher energy consumption by the camera than in a normal tracking mode in which the camera is operated to track the marker; providing at least one operational signal to the tracking system; operating at least one of the at least two cameras in a high energy consumption mode, thereby heating at least the camera with a higher energy consumption compared to a normal tracking mode; has.

[0151] Thus, a faster warm-up curve is provided to reach a temperature at least essentially similar to the calibration temperature so that errors due to temperature dependence in the optical system are minimized.

[0152] In one example of the method, at least one camera is operated in a high energy consumption mode until a predetermined thermal condition is reached.

[0153] Optionally, the high energy consumption mode is maintained until at least one of the groups of at least one camera and the system reaches its normal operating temperature, which is the basis for the calibration of the system.

[0154] In another option, the high energy consumption mode is provided beyond the point at which at least one camera has reached its normal operating temperature.

[0155] The normal operating temperature may also be referred to as the calibration temperature of the system.

[0156] In one example, at least one camera is operated in a high energy consumption mode until a predetermined temperature of the camera is reached.

[0157] In another example, at least one camera is operated in a high energy consumption mode for a predetermined period of time.

[0158] In one example, the high power consumption is maintained for longer than required to reach the normal operating temperature of the camera so that the part of the system that acts as a heat sink, such as the camera holder or frame, reaches the target temperature more quickly, after which the system switches to a lower power consumption setting.

[0159] In one example, the portion of the system that serves as a heat sink for the camera is designed such that thermal equilibrium of the system is reached as quickly as possible, e.g., providing that the heat flow from the camera to the frame is optimized such that delays in warming up the frame and associated suboptimal system performance are minimized.

[0160] Preferably, the part of the system that acts as a heat sink for the camera is designed so that thermal equilibrium of the system is reached as quickly as possible. One example could be to optimize the heat flow from the camera to the frame so that delays in frame warm-up (and the associated suboptimal system performance) are minimized.

[0161] In one example of the method, after heating at least one of the at least two cameras by operating the at least one camera in a high energy consumption mode, a normal operation signal is provided and the at least one camera is operated under normal conditions.

[0162] Optionally, the settings for normal operation are modified based on the difference between the system temperature at the time of calibration and the expected steady state temperature of the system under real-time conditions.

[0163] In one example, the settings used for normal operation are not fixed, but are adjusted based on the difference between 1) the system temperature at the time of calibration and 2) the expected steady state temperature of the system under real-time conditions. For example, if the system temperature during normal operation is higher or lower than the temperature at the time of calibration, the camera parameters are selected to cause the camera to consume less or more power, thereby bringing the system closer to the calibrated state.

[0164] In one example, the camera settings for normal operation are optimized for minimum power consumption while providing adequate image data. This aids in the steady state temperature of the optical system during normal operation, thus reducing the temperature difference between the cold start condition (where the camera is at room temperature) and the temperature at which thermal equilibrium has been reached. As a result, not only is the time required to reach normal operating temperature shorter, but the system also experiences fewer temperature fluctuations (and associated performance degradation) during start-up.

[0165] In one example of the method, a model of the warm-up of the system and its dynamic thermal behavior is provided and taken into account for at least one control signal, the at least one control signal having a signal sequence with a plurality of varying control sub-signals, in order to generate at least one operational signal for the tracking system.

[0166] One example of the method provides for analyzing at least two images from different times during warm-up to determine the current displacement speed and adjust the high energy consumption mode accordingly.

[0167] In one example of the method, multiple sensors are provided that monitor the thermal behavior of the system. The thermal behavior monitored by the sensors is used to provide a more accurate model of the system's warm-up. Optionally, the model of the system's warm-up is used in combination with multiple temperature readings from the sensors during warm-up to dynamically adjust the system's calibration. As a further option, a temperature history of the system is provided and a remaining time is predicted before the system reaches a normal operating phase. Optionally, the remaining time is indicated.

[0168] In an example provided as another option of the method, providing a dynamic calibration model for the camera-based tracking system, the dynamic calibration model having a number of calibration coefficients that are dependent on a temperature-related state of the system; providing a current temperature related state of the camera-based tracking system; determining a matching calibration coefficient from the plurality of calibration coefficients; providing matching calibration coefficients for application during operation of the camera-based tracking system; is provided.

[0169] Therefore, a modified calibration is provided during the start-up phase that provides a matching temperature that is at least essentially similar to the dynamic calibration temperature so that errors due to temperature dependence in the optical system are minimized.

[0170] In one example, the temperature related conditions include temperature measurements from a sensor. For example, the sensor is provided within the system. Optionally, the sensor is integrated and provides a temperature signal for other system operation aspects as well. Optionally, the sensor is provided exclusively for calibration adjustments. According to one aspect, the temperature dependent behavior of the optical system is measured and modeled. In addition to knowing the changing characteristics in advance (e.g., based on simulations or based on empirical data), additional temperature sensors can be provided to monitor the temperature of the camera and supporting hardware in real time.

[0171] Rather than using a static calibration associated with a steady state, a dynamic calibration model is provided. In one example, the dynamic calibration model depends on the time elapsed since start-up. In one example, the dynamic calibration depends on the current temperature conditions, for example using a temperature sensor reading. In one example, the dynamic calibration model is used taking into account the time elapsed since start-up and the current temperature conditions.

[0172] In one example, a look-up table is provided in advance.

[0173] In one basic approach, only the time from start-up is used to pick up the correct set of calibration parameters from a look-up table.

[0174] Another basic approach provides measurements from a temperature sensor that are used to pick up the correct set of calibration parameters from a look-up table.

[0175] For example, an internal temperature sensor in one of the camera chips is used to select the set of calibration parameters that likely gives the best results.

[0176] In another more advanced approach, the starting temperature of the camera is used in combination with the ambient temperature to estimate the warm-up curve and thus provide the correct calibration coefficients.

[0177] In one example, the current temperature-related conditions are updated, for example, constantly or at varying intervals, to provide two or more updated calibration factors during system warm-up.

[0178] 5 shows a graph with a curve 200 showing the tracking error with respect to time after initiation. The vertical axis 202 is the tracking error E TR in millimeters, and the horizontal axis 204 is time after start-up t AST 2 shows the time (in minutes) required for the optical tracking system to warm up. During the warm-up of the optical tracking system, the tracking error gradually decreases as the temperature of the system stabilizes until it reaches the baseline level 206 (intrinsic accuracy) of the system. Preferably, the time A required to reach the steady state from the maximum error level 208 is short to allow the user to use the system immediately after start-up. The difference B in the tracking error associated with different temperature conditions of the system, e.g., cold start or operating temperature, is a measure of the robustness of the system against temperature variations. A vertical line 210 separates a first stage S1, also called stage 1, from a second stage S2, also called stage 2.

[0179] 6 shows a graph with a curve 220 showing the temperature of the system versus time after start-up. The vertical axis 222 shows the temperature T in °C (Celsius) and the horizontal axis 224 shows the time after start-up t AST 2 shows the time in minutes. The graph can be divided into a warm-up phase, e.g., phase S1, and a more steady operation phase, e.g., phase S2. The separation is indicated by vertical line 226.

[0180] FIG. 7 shows a graph 230 with different curves showing the system temperature over time for different operating modes. The vertical axis 232 shows the temperature T in °C, and the horizontal axis 234 shows the time t in minutes. In an optical tracking system with an industrial standard machine vision camera, the measured power consumption of the camera can be increased by changing the image acquisition parameters. In this particular example, the power consumption is increased from 2.6 W to 2.96 W by increasing the frame rate from 15 frames per second (fps) to 30 frames per second (fps). The warm-up of the camera imaging chip and the camera housing was accelerated by 3.5 times and 1.4 times, respectively, by boosting the warm-up using a higher frame rate. A first curve 236 shows the camera internal temperature for 30 fps, and a second curve 238 shows the camera internal temperature for 15 fps. A third curve 240 shows the external temperature, i.e. the temperature outside the camera, for 30 fps, and a fourth curve 242 shows the external temperature, i.e. the temperature outside the camera, for 15 fps. A further curve 244 shows the air temperature. As can be seen, an increase in frame rate results in a significant increase in temperature, during an initial phase where the temperature rises sharply, then levels off towards a steady state temperature.

[0181] 8 shows graphs illustrating an example of a warm-up pattern of system temperature and power consumption of a camera over time. In the upper graph, the vertical axis 250 shows temperature T in °C and the horizontal axis 252 shows time t after start-up. AST In the lower graph, the vertical axis 254 represents the power consumption P of the camera in W versus the time in minutes after start-up t AST is again shown on the horizontal axis 256. During phase S1, the default high power demand setting D EF The H-PS is used to accelerate the warm-up of the system. When stage S2 is reached, the system warms up according to the temperature curve T S For comparison, a further curve 262 is shown for the default normal power setting, D EFIn the upper graph, a horizontal line 284 indicates the calibration temperature T C As can be seen, application of a higher power demanding camera setting results in an increase in temperature and a faster time to achieve operating temperature.

[0182] 9 shows graphs illustrating another example of a warm-up pattern of system temperature and camera power consumption over time. In the upper graph, vertical axis 270 shows temperature T in °C and horizontal axis 272 shows time t in minutes. In the lower graph, vertical axis 274 shows camera power consumption P in W and horizontal axis 276 shows time t in minutes. At the start of stage S2, image acquisition parameters and associated camera power consumption are dynamically adjusted until the calibrated temperature is reached. The first portion 278 of the curves shows the warm-up pattern at the default high power setting, D EF The second portion of the curve 280 illustrates the application of the dynamic power setting, D YN The curve 282 in the lower graph shows the varying camera power consumption CPC. Also in the upper graph, a horizontal line 310 indicates the application of the calibration temperature T C As can be seen, a dynamic adaptation of the power consumption is provided in the transition period at the boundary between the first stage S1 and the second stage S2.

[0183] 10 shows graphs illustrating further examples of warm-up patterns of system temperature and camera power consumption over time. In the upper graph, the vertical axis 290 is system temperature T S in ° C., and the horizontal axis 292 is the time after start-up t AST In the center graph, the vertical axis 294 represents the ambient temperature T A in ° C., and the horizontal axis 296 is the time after start-up t AST In the lower graph, the vertical axis 298 represents the power consumption P of the camera in W, and the horizontal axis 300 represents the time after start-up t AST During stage S2, image acquisition parameters (and associated camera power consumption) are dynamically adjusted to mitigate the effects of fluctuations in external conditions, e.g., ambient temperature, such that a temperature close to the system temperature at the time of calibration is maintained. The first portion 302 includes a default high power setting, D EFThe second part of the curve 304 illustrates the application of the dynamic power setting, D YN 3 shows the application of PS. The middle graph curve 306 shows a decrease in ambient temperature. The lower graph curve 308 shows the changing camera power consumption CPC. As can be seen, even after the initial heating, further adjustments in the system temperature can be accommodated taking into account the calibration during operation. As an example, a change in ambient temperature is shown. In response, the power consumption of the camera is adjusted accordingly.

[0184] The term "subject" may also be referred to as an individual. A "subject" may also be referred to as a patient, although it should be noted that this term does not indicate whether any disease or illness is actually present in the subject.

[0185] In one example, a computer program or program element for controlling an apparatus according to one of the above examples is provided, which program or program element is configured to perform the method steps of one of the above method examples when executed by a processing unit.

[0186] Thus, the computer program element may be stored in a computing unit or distributed across two or more computing units that may be part of an embodiment of the present invention. This computing unit may be configured to execute or direct the execution of the steps of the above-mentioned method. Furthermore, it may be configured to operate the components of the above-mentioned apparatus. The computing unit may be configured to operate automatically and / or to execute a user's order. The computer program may be loaded into a working memory of a data processor. The data processor may thus be configured to execute the method of the present invention.

[0187] Aspects of the invention may be embodied in a computer program product, which may be a collection of computer program instructions stored on a computer-readable storage device that may be executed by a computer. The instructions of the invention may be any interpretable or executable code mechanism, including, but not limited to, a script, an interpretable program, a dynamic link library (DLL), or a Java class. The instructions may be provided as a complete executable program, a partial executable program, a modification (e.g., an update) to an existing program, or an extension (e.g., a plug-in) to an existing program. Additionally, parts of the processing of the invention may be distributed across multiple computers or processors.

[0188] As mentioned above, a processing unit, e.g., a controller, performs the control method. This controller can be implemented in various ways using software and / or hardware to perform the various functions required. The processor is one example of a controller that uses one or more microprocessors that can be programmed using software (e.g., microcode) to perform the functions required. However, the controller may be implemented with or without a processor, or as a combination of dedicated hardware to perform some functions and a processor (e.g., one or more programmed microprocessors and associated circuitry) to perform other functions.

[0189] Examples of controller components used in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs).

[0190] This exemplary embodiment of the invention encompasses both computer programs that use the invention from the outset, and computer programs that, by means of an update, turn existing programs into programs that use the invention.

[0191] Moreover, the computer program element may be capable of providing all the steps required to fulfill the procedures of the exemplary embodiments of the methods described above.

[0192] According to a further exemplary embodiment of the present invention, a computer readable medium, such as a CD-ROM, is presented, having stored thereon a computer program element, which is described by the previous section. The computer program may be stored and / or distributed on a suitable medium, such as an optical storage medium or a solid-state medium, supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless communication systems.

[0193] However, the computer program may also be presented over a network such as the World Wide Web and can be downloaded from such a network into the working memory of a data processor. According to a further exemplary embodiment of the invention, a medium for making available for downloading a computer program element is provided, this computer program element being configured to perform a method according to one of the aforementioned embodiments of the invention.

[0194] It should be noted that the embodiments of the present invention are described with reference to different subject matters. In particular, some embodiments are described with reference to method type claims, and other embodiments are described with reference to device type claims. However, those skilled in the art will understand from the above and the following description that, unless otherwise indicated, any combination of features belonging to one type of subject matter, as well as any combination between features relating to different subject matters, is disclosed in the present application. However, all features can be combined to provide a synergistic effect that is higher than the simple sum of the features.

[0195] While the invention has been illustrated and described in detail in the drawings and the foregoing description, such illustration and description are to be considered exemplary or explanatory and not restrictive. The invention is not limited to the disclosed embodiments. Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure and the dependent claims.

[0196] In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be interpreted as limiting the scope.

Claims

1. 1. A controller for operation of a camera based tracking system, the controller comprising: A controller; A control signal output unit; having the controller is configured to generate, at least during a start-up phase of the tracking system, at least one operating signal for the tracking system having at least one calibration compensation command; the at least one calibration compensation instruction comprises instructions for at least one of the group of: i) actively adjusting a temperature of the camera-based tracking system; and ii) adapting a calibration-related camera output of the camera-based tracking system used in tracking calculations; the control signal output is configured to provide the at least one operating signal to the tracking system. Control device.

2. The controller of claim 1 , wherein the calibration compensation instructions include at least one control signal for temporarily operating the camera-based tracking system in a high heat generating mode.

3. To temporarily operate the camera-based tracking system in the high heat generation mode, the calibration compensation instructions have at least one control signal for temporarily operating at least one of the at least two cameras in a heating mode in which heat is generated by at least one of the at least two cameras, the at least one control signal being configured to operate the at least one of the at least two cameras in a high energy consumption mode in which a higher energy consumption is achieved by the camera than in a normal tracking mode in which the camera is operated for the tracking; the control signal output unit is configured to provide the at least one operating signal to the tracking system to operate the at least one of the at least two cameras in the high energy consumption mode, thereby heating at least the camera with a higher energy consumption compared to the normal tracking mode. The control device according to claim 2.

4. The controller of claim 2 , wherein for the high heat production mode, the calibration compensation instructions include at least one control signal for temporarily activating a heating element of the camera-based tracking system.

5. The controller is configured to take into account measured external conditions relative to the control signal when generating the operating signal, the external conditions comprising: The ambient temperature, The lighting conditions and Indoor airflow, having at least one of the following groups:

5. A control device according to claim 2, 3 or 4.

6. To generate the at least one operating signal for the tracking system, the controller is configured to provide a model of the warm-up of the system and its dynamic thermal behavior and to take these into account for the at least one control signal, the at least one control signal comprises a signal sequence having a plurality of varying control sub-signals; A control device according to any one of claims 2 to 5.

7. the controller is configured to provide a dynamic calibration model for the camera-based tracking system, the dynamic calibration model having a plurality of calibration coefficients responsive to a temperature-related condition of the system; for a current temperature-related state of the camera-based tracking system, the controller is configured to determine a matching calibration coefficient from the plurality of calibration coefficients; the control signal output is configured to provide the matching calibration coefficients for application during operation of the camera-based tracking system. A control device according to any one of claims 1 to 6.

8. A plurality of at least two cameras; A control device according to any one of claims 1 to 7; In a camera-based tracking system having the at least one operating signal is provided for operating at least one of the at least two cameras; The camera-based tracking system is a camera-based medical tracking system for use in a hospital or other medical related setting. Camera-based tracking system.

9. the high energy consumption mode comprises image acquisition having an energy consumption at least 50% higher than an energy consumption during a normal tracking mode of the system; The high energy consumption mode includes: Frame rate, Exposure time, Bit depth and Bit packing and A region of interest; The clocking frequency, Color format, Image compression and On-board image pre-processing and at least one of the groups has image acquisition that is at least 50% increased compared to a normal tracking mode of the system; The system of claim 8.

10. a plurality of sensors are provided for monitoring the thermal behavior of the system; the thermal behavior monitored by the sensor is used to provide a more accurate model of the warm-up of the system; the model of warm-up of the system is used in combination with multiple temperature readings from a sensor during warm-up to dynamically adjust a calibration of the system; A temperature history of the system is provided, and a time remaining before the system reaches a normal operating stage is predicted, and the time remaining is indicated.

10. A system according to claim 8 or 9.

11. 11. The system of claim 8, wherein the controller is configured to analyze at least two images from different times during the warm-up to determine a current displacement speed and adjust the high energy consumption mode accordingly.

12. A set of markers capable of being tracked by the system is provided; The marker is attachable to at least one of the following groups: an object, an object, and a device. A system according to any one of claims 8 to 11.

13. 1. A method for operation of a camera-based tracking system, comprising: generating, at least during a start-up phase of the tracking system, at least one operating signal for the tracking system having at least one calibration compensation command, the at least one calibration compensation instruction comprises instructions for at least one of the group of: i) actively adjusting a temperature of the camera-based tracking system; and ii) adapting a calibration-related camera output of the camera-based tracking system used in tracking calculations; providing the at least one operating signal to the tracking system; operating the camera-based tracking system based on the operating signal having at least one calibration compensation command; The method according to claim 1,

14. A computer program enabling a processor to carry out the method according to claim 13.

15. A computer readable medium storing a computer program according to claim 14.

Citation Information

Patent Citations

  • Bone and tool tracking in robotized computer-assisted surgery

    US20180235715A1

  • Methods and System for Performing 3-D Tool Tracking by Fusion of Sensor and / or Camera Derived Data During Minimally Invasive Robotic Surgery

    US20210000546A1