Medical camera system

Abstract

The invention relates to a multi-camera system including an image synthesis camera device and a multi-filter camera device. The image synthesis camera apparatus includes a plurality of visible light cameras, and is configured to determine a synthesized image of a scene by using a plurality of distance sensors covering a plurality of distance sections of a 360-degree view of the scene, thereby, a camera model of one of the plurality of visible light cameras is calibrated using the distance information provided by the plurality of distance sensors. The multi-filter camera device includes a multi-filter device configured to determine filtered light from collected light, where the filtered light includes temporally interleaved light of at least two wavelength bands such that the multi-filter camera device supports different wavelength bands in a more compact structure.

Description

Technical Field

[0001] The invention relates to a medical camera system, an image synthesis camera device and a corresponding method for determining a composite image of a scene, and a multi-filter camera device and a corresponding method. Background Art

[0002] Medical camera systems are used in medical environments such as operating rooms or surgical suites for monitoring (i.e., instrument usage, tray monitoring, equipment setup / posture, people and their movements), tracking (i.e., single tracking, event / workflow / people tracking, video feeds), and / or recording (i.e., documentation) tasks.

[0003] However, at this time, different medical camera systems can be used for several different camera applications, which cannot be integrated into one single medical camera system. In addition, the known systems have poor performance and are generally very complex.

[0004] The object of the present invention is to provide a medical camera system, in particular a medical camera subsystem, ie a camera device, with improved performance and / or reduced complexity.

[0005] The present invention can be used in conjunction with the Brainlab navigation system, e.g. or Brainlab Digital OR Systems, e.g. or Buzz All products of Brainlab AG.

[0006] The following discloses various aspects, examples and exemplary steps of the present invention and its embodiments.Different exemplary features of the present invention may be combined according to the invention whenever this is technically advantageous and feasible. Summary of the Invention

[0007] Example Short Description of the Invention

[0008] The present invention relates to a multi-camera system comprising an image synthesis camera device and a multi-filter camera device.

[0009] An image synthesis camera device for determining a composite image of a scene includes the following components: a plurality of visible light cameras covering a 360-degree view of the scene, wherein each of the plurality of visible light cameras is configured to determine an image of the scene; a plurality of range sensors covering a plurality of distance bins of the 360-degree view of the scene, wherein each of the plurality of range sensors is configured to determine distance information from the corresponding range sensor to a nearest object for at least one of the plurality of range bins; and a processing device configured to calibrate a camera model of one of the plurality of visible light cameras using the corresponding distance information determined for each of the plurality of range bins, and to determine a composite image of the scene using the calibrated camera model for each of the plurality of range bins and images of the scene from each of the plurality of visible light cameras. Thus, an image synthesis camera device for determining a composite image of a scene is provided, having improved power consumption and better stitching results for real-time applications.

[0010] The multi-filter camera device includes the following components: a lens configured to collect light reflected from a scene; a multi-filter device configured to determine filtered light from the collected light, wherein the filtered light includes temporally interleaved light of at least two wavelength bands; an image sensor configured to determine image data from the filtered light, wherein the image data includes temporally interleaved image data corresponding to at least two wavelength bands; a processing device configured to process the image data, wherein the processing device includes at least one video pipeline, wherein each of the at least one video pipeline is configured to process image data corresponding to at least one of the at least two wavelength bands, wherein the processing device includes a sequencer device configured to temporally manage the image data and the at least one video pipeline. Therefore, the multi-filter camera device supporting different wavelength bands has a more compact structure.

[0011] Example General Description of the Invention

[0012] The invention is defined by the subject matter of the independent claims. Additional features of the invention are presented in the dependent claims.

[0013] In this section, a description of general features of the invention is given, for example, by reference to possible embodiments of the invention.

[0014] According to one aspect of the present invention, an image synthesis camera device for determining a composite image of a scene includes: a plurality of visible light cameras covering a 360-degree view of the scene, wherein each of the plurality of visible light cameras is configured to determine an image of the scene; a plurality of range sensors covering a plurality of distance bins of the 360-degree view of the scene, wherein each of the plurality of range sensors is configured to determine distance information from the corresponding range sensor to a nearest object for at least one of the plurality of range bins; and a processing device configured to calibrate a camera model of one of the plurality of visible light cameras using the corresponding distance information determined for each of the plurality of range bins, and to determine a composite image of the scene using the calibrated camera model for each of the plurality of range bins and an image of the scene from each of the plurality of visible light cameras.

[0015] Consumer 360-degree cameras are typically optimized for a range of distances (usually far), though some cameras can be calibrated for either far or near ranges. Optical flow methods are often used to optimize the stitching of video / image content. The downside of optical flow solutions is that the computational power required for processing is very high, leading to higher hardware costs and energy / heat dissipation into the environment. High latency, such as several seconds, has been measured, making them unsuitable for some applications planned for operating rooms. Furthermore, they remain a poor solution for larger depths of field.

[0016] As used herein, the term "scene" includes a three-dimensional view of an area, in particular a room, further in particular an operating room. The scene is also referred to as a region of interest, which is monitored by the camera system.

[0017] As used herein, the term "composite image" refers to an image composed of, or in other words, combined, multiple images captured from different image sources (in particular, a camera system comprising multiple cameras). This is also known as stitching. Preferably, the composite image covers a 360-degree view of the scene.

[0018] Preferably, determining the composite image comprises the steps of pre-computing a representation of the camera system, preferably comprising a plurality of camera unit spheres and a composite unit sphere representing the visible light cameras, and creating the composite image from individual camera images of the plurality of visible light cameras, and applying a final distortion correction or an optimized parallax correction to the composite image.

[0019] Further preferably, pre-computing the representation of the camera system comprises calibrating the representation of the camera system and performing a direct alignment or a feature-based alignment.

[0020] Further preferably, creating the composite image from individual camera images of the plurality of cameras comprises finding seam lines among camera images that are viewing similar parts of the scene and blending the content of each of the camera images that share the seam line.

[0021] Further preferably, determining the composite image comprises transforming from a single camera view to a sphere using a mathematical lens model and / or a matrix of points to model the camera using intrinsic and extrinsic parameters. Further preferably, determining the composite image comprises using a distance model in the matrix and / or interpolation (particularly linear interpolation) between points (for x / y). This should reduce or allow interpolation of the matrix.

[0022] Further preferably, determining the composite image includes using forward projection. This involves taking each pixel from each visible light camera, transforming the coordinates into a position on a sphere or cylinder, potentially weighting it, and combining it with pixels from the other visible light camera in the overlapping area. In other words, forward projection is used to project points from the cameras onto the sphere or cylinder.

[0023] Further preferably, determining the composite image comprises using back projection, which comprises, for each pixel in the resulting sphere or circle, transforming it into coordinates of one or more visible light cameras and obtaining a weighted sum from these visible light camera pixels as a target pixel.

[0024] The proposed solution determines distance information in several distance bins of a 360-degree view, which is used to calibrate a camera model to determine a composite image of the scene for each of the plurality of distance bins in real time.

[0025] The calibration of the camera model, in particular the matrix transformation and the interpolation methods, can be implemented in hardware such as an FPGA, ASIC or ASSP to allow the calculations to be performed in real time.

[0026] In other words, instead of using complex stitching and optimization methods to determine a composite image, a camera model is calibrated for each of a plurality of distance bins based on the distance information of the respective distance sensors, wherein the camera model is configured to be calibrated in such a manner that zero parallax can be achieved between images of the plurality of visible light cameras within a specific distance to an object of interest based on the distance information.

[0027] Preferably, the image synthesis camera device is comprised in a medical camera system.The medical camera system is also preferably used for data aggregation in an operating room, in particular during a medical procedure.

[0028] Preferably, the camera model is configured to be calibrated such that zero parallax can be achieved at a specific distance between the visible light camera and the object when determining a composite image of the scene.

[0029] Determining the composite image includes stitching the image of the scene from each visible light camera into a seamless 360 degree image of the scene.

[0030] The 360-degree view is used for an immersive experience of the scene and seamless data coverage. In addition, mechanical movement (e.g., pan / tilt / zoom of the camera) can be avoided when the 360-degree field covers all parts of the scene at all distances with as little distortion (i.e., gaps and overlaps) as possible.

[0031] Preferably, the plurality of visible light cameras and / or the plurality of distance sensors are arranged in a ring to cover a 360-degree view of the scene.

[0032] For example, the image synthesis camera device includes two to six visible light cameras and two to six distance sensors.

[0033] Preferably, the image synthesis camera device includes two visible light cameras, each of which is configured to cover a field of view of 160 to 230 degrees. Thus, the two visible light cameras cover a 360-degree view of the scene without significant overlap of the fields of view of the two visible light cameras. To provide improved scanning resolution along the 360-degree view of the scene, the image synthesis camera device includes six range sensors, each of which is configured to cover a field of view of 55 to 65 degrees. Thus, the six range sensors cover a 360-degree view of the scene without significant overlap of the fields of view of the six range sensors.

[0034] The more cameras and sensors used, the larger the diameter of the ring-shaped assembly that arranges all the cameras and sensors. A larger diameter leads to more problems with stitching based on the distance to the visible light camera (i.e., based on parallax caused by diameter and lens imperfections). In particular, when calibrating the camera model of a single visible light camera in a multiplicity of visible light cameras, extrinsic distance is a significant factor in creating parallax. Extrinsic distance represents the distance between visible light cameras in an image synthesis camera setup. However, as in real life, multiple visible light cameras cannot occupy the same physical space at the same time. Therefore, there is a trade-off in the number of visible light cameras and distance sensors: a larger number of distance sensors increases the scan resolution of the image synthesis camera setup but increases the computational workload. It also increases the diameter of the ring-shaped arrangement of visible light cameras and distance sensors, introducing more parallax. Parallax is the displacement in the apparent position of an object observed along different lines of sight. Since multiple visible light cameras cannot occupy the same physical space, parallax leads to errors in the composite image, such as missing information or the presence of double information. Preferably, the diameter is less than 50 mm.

[0035] Hence, an image synthesis camera device for determining a composite image of a scene is provided with improved power consumption and better stitching results for real-time usage.

[0036] In a preferred embodiment, calibrating the camera model comprises the steps of obtaining a plurality of camera models statically pre-calibrated for different distances, and selecting a selected camera model from the plurality of camera models for each of the plurality of distance bins based on the determined distance information of the corresponding distance bins.

[0037] Instead of recalibrating the camera model each time new distance information is provided, the camera model is statically pre-calibrated for different distances. For example, the camera model is pre-calibrated for distances of 10 cm increments. Thus, when the instance sensor provides updated distance information, a corresponding pre-calibrated camera model is selected from the plurality of pre-calibrated camera models.

[0038] Thus, the step of calibrating the camera model can be skipped and the computational effort of determining the composite image can be reduced.

[0039] Hence, an image synthesis camera device for determining a composite image of a scene is provided with improved power consumption and better stitching results for real-time usage.

[0040] In a preferred embodiment, calibrating the camera model comprises the following steps: obtaining a parameterizable calibration model and parameterizing the parameterizable calibration model for each distance bin of the plurality of distance bins based on the determined distance information of the corresponding distance bin.

[0041] Instead of completely recalibrating the camera model each time new distance information is provided, the camera model can be parameterized based on distance. In other words, the parameters of the parameterizable calibration model include distance information. Therefore, when the distance information changes, the entire camera model does not have to be changed.

[0042] Thus, the time for calibrating the camera model can be significantly reduced and the computational effort for determining the composite image can be reduced.

[0043] Hence, an image synthesis camera device for determining a composite image of a scene is provided with improved power consumption and better stitching results for real-time usage.

[0044] In a preferred embodiment, the camera model includes an intrinsic compensation model and an extrinsic compensation model.

[0045] Preferably, the camera model is configured to be calibrated in such a way that zero parallax can be achieved at a specific distance between the visible light camera and the object, which is also referred to as the alignment distance.

[0046] In other words, the camera model (also known as the three-sphere model) can be interpreted as a generalization of the known two-sphere method, in which the entire camera system is represented by a two-sphere model. The three-sphere model introduces a composite unit sphere that unifies the camera unit spheres, which can be represented by two-sphere models, for example. Therefore, the distance between multiple cameras (which is called the extrinsic distance) is represented by a unified offset. In contrast, the two-sphere method does not consider the distance between multiple visible light cameras and therefore does not include any unified offset, i.e., it assumes that the cameras share the same physical location.

[0047] Hence, an image synthesis camera device for determining a composite image of a scene is provided with improved power consumption and better stitching results for real-time usage.

[0048] In a preferred embodiment, at least one distance sensor is a low-resolution distance sensor.

[0049] At least one distance sensor is provided to determine distance information to the nearest object in a respective distance segment covered by the at least one distance sensor. Therefore, the distance sensor itself does not need to have a high resolution.

[0050] Hence, an image synthesis camera device for determining a composite image of a scene is provided with improved power consumption and better stitching results for real-time usage.

[0051] In a preferred embodiment, the low resolution distance sensor has a resolution of one of 4x4, 2x2 or 1x1.

[0052] As explained, the number of range sensors provided in an image synthesis camera device is a trade-off between scanning resolution, computational effort, and parallax introduction. Preferably, the number of range sensors should not exceed six. This allows for a relatively small diameter for the ring arrangement of range sensors and visible light cameras, resulting in relatively low parallax introduction.

[0053] Six low-resolution range sensors with a resolution of 1x1 are provided. Each range sensor must cover one of six range bins ranging from 55 to 65 degrees in order to cover a full 360-degree view of the scene. Using a resolution of 2x2 or 4x4 increases the scan resolution to 12 or 24 range bins. The computational workload of using 24 range bins is not 24 times greater than using only one range bin, but only about twice as great as using only one range bin. Therefore, using 12 to 24 range bins provides a relatively good trade-off between scan resolution and computational workload.

[0054] In low-resolution distance sensors, only the horizontal resolution is used to determine distance information. The vertical resolution is simply ignored. In other words, although a 2x2 low-resolution distance sensor provides four pieces of distance information, only two of them that are horizontal are used. Thus, although a 4x4 low-resolution distance sensor provides sixteen pieces of distance information, only four of them that are horizontal are used.

[0055] Hence, an image synthesis camera device for determining a composite image of a scene is provided with improved power consumption and better stitching results for real-time usage.

[0056] In a preferred embodiment, the number of the plurality of visible light cameras is smaller than the number of the at least one distance sensor.

[0057] Preferably, the number of the plurality of visible light cameras is two, because two visible light cameras can cover a 360-degree view, and only two stitching operations and optimizations must be performed to determine the composite image. On the other hand, the number of the at least one range sensor is preferably greater than one, and more preferably six, in order to increase the scanning resolution of the at least one range sensor covering the entire 360-degree view.

[0058] Thus, an optimal trade-off between scanning resolution and computational effort is provided.

[0059] Hence, an image synthesis camera device for determining a composite image of a scene is provided with improved power consumption and better stitching results for real-time usage.

[0060] In a preferred embodiment, a plurality of distance sensors are arranged to be interleaved with a plurality of visible light cameras.

[0061] In a preferred embodiment, the plurality of distance sensors includes one of a time-of-flight sensor and a LIDAR sensor.

[0062] Other examples of distance sensors include range cameras and surface scanners.

[0063] Preferably, the plurality of distance sensors provide a two-dimensional image tracking function.

[0064] In a preferred embodiment, the processing device is configured to calibrate the camera model and determine the composite image within a predetermined time period.

[0065] For example, distance information is obtained only every 100 ms. Thus, the computational workload is further reduced without losing too much performance in determining the composite image.

[0066] Hence, an image synthesis camera device for determining a composite image of a scene is provided with improved power consumption and better stitching results for real-time usage.

[0067] In a preferred embodiment, the plurality of visible light cameras are large field of view cameras.

[0068] In contrast, pinhole cameras are suboptimal for viewing angles greater than 120 degrees. Therefore, to cover a 360-degree horizontal field of view, a large number of pinhole cameras would be required.

[0069] In a preferred embodiment, each large field of view camera includes a field of view of at least 160 degrees.

[0070] Therefore, to cover a 360-degree field of view horizontally, only two large-field-of-view cameras are required.

[0071] In a preferred embodiment, the large field of view camera comprises a camera with a fisheye lens.

[0072] In a preferred embodiment, the processing device comprises a hardware processing device.

[0073] According to another aspect of the present invention, a medical method for determining a composite image of a scene includes the following steps: Each of a plurality of visible light cameras covering a 360-degree view of the scene obtains an image of the scene; Each of a plurality of distance sensors covering a plurality of distance bins of the 360-degree view of the scene obtains, for at least one of the plurality of distance bins, distance information from the corresponding distance sensor to a nearest object; Calibrates, by a processing device, a camera model of one of the plurality of visible light cameras for each of the plurality of distance bins using the determined corresponding distance information; Determines, by the processing device, a composite image of the scene using the calibrated camera model for each of the plurality of distance bins and the image of the scene from each of the plurality of visible light cameras.

[0074] Preferably, at least part of the method is a computer implemented method.

[0075] In a preferred embodiment, calibrating the camera model comprises the steps of obtaining a plurality of camera models statically pre-calibrated for different distances, and selecting a selected camera model from the plurality of camera models for each of the plurality of distance bins based on the determined distance information of the corresponding distance bins.

[0076] In a preferred embodiment, calibrating the camera model comprises the following steps: obtaining a parameterizable calibration model and parameterizing the parameterizable calibration model for each distance bin of a plurality of distance bins based on the determined distance information of the corresponding distance bin.

[0077] In a preferred embodiment, the camera model includes an intrinsic compensation model and an extrinsic compensation model.

[0078] In other words, the camera model includes the intrinsic and extrinsic parameters of each visible light camera.

[0079] Preferably, the camera model is calibrated such that the intrinsic parameters (ie, the intrinsic parameters of each visible light camera) as well as the extrinsic parameters (such as the rotation and translation of the visible light cameras with respect to each other) are known.

[0080] In a preferred embodiment, at least one distance sensor is a low-resolution distance sensor.

[0081] In a preferred embodiment, the low resolution distance sensor has a resolution of one of 4x4, 2x2 or 1x1.

[0082] In a preferred embodiment, the number of the plurality of visible light cameras is smaller than the number of the at least one distance sensor.

[0083] In a preferred embodiment, a plurality of distance sensors are arranged to be interleaved with a plurality of visible light cameras.

[0084] Thus, splicing in the same area is avoided. Thus, double splicing is avoided.

[0085] In a preferred embodiment, the plurality of distance sensors includes one of a time-of-flight sensor and a LIDAR sensor.

[0086] In a preferred embodiment, the method comprises calibrating, by the processing device, the camera model and determining the composite image at least every 100 ms.

[0087] In a preferred embodiment, the plurality of visible light cameras are large field of view cameras.

[0088] In a preferred embodiment, each large field of view camera includes a field of view of at least 160 degrees.

[0089] In a preferred embodiment, the large field of view camera comprises a camera with a fisheye lens.

[0090] In a preferred embodiment, the processing device comprises a hardware processing device.

[0091] According to another aspect of the present invention, a multi-filter camera device includes: a lens configured to collect light reflected from a scene; a multi-filter device configured to determine filtered light from the collected light, wherein the filtered light includes temporally interleaved light of at least two wavelength bands; an image sensor configured to determine image data from the filtered light, wherein the image data includes temporally interleaved image data corresponding to the at least two wavelength bands; and a processing device configured to process the image data, wherein the processing device includes at least one video pipeline, wherein each of the at least one video pipeline is configured to process image data corresponding to at least one of the at least two wavelength bands, and wherein the processing device includes a sequencer device configured to temporally manage the image data and the at least one video pipeline.

[0092] As used herein, the term "scene" includes a three-dimensional view of an area, particularly a room, further particularly an operating room. The scene is also referred to as a region of interest, which is monitored by the multi-filter camera device.

[0093] As used herein, the term "video pipeline" refers to a video processing device that processes an image based on provided image data. For example, an infrared video pipeline may provide an infrared image based on image data from an infrared wavelength band. For example, an RGB video pipeline may provide an RGB image based on image data from a visible light wavelength band.

[0094] Multi-filter camera systems are used in operating rooms and within products / systems for applications ranging from documentation and enhanced observation (endoscopes and microscopes) to (marker) tracking. For these applications, visible light and near-infrared (NIR) views are typically used independently, depending on the application. In the past, cameras were often separated for visible light and IR, as different filters, optics, and sensors were required. Typically, the requirements for frame rate, global shutter, and high resolution were quite high, resulting in each of the two cameras being quite large and inherently expensive. Consequently, multi-filter camera systems that allow for processing of different wavelength bands present a high cost issue.

[0095] Hence, a multi-filter camera device is provided which uses a reduced number of components, in particular only one image sensor and only one set of optics (ie lenses).

[0096] Preferably, managing image data and at least one video pipeline includes: if the at least one video pipeline includes at least two video pipelines, each video pipeline is configured to process image data corresponding to one of the at least two bands, then the image data is temporally distributed to the at least two video pipelines, or if the at least one video pipeline includes one video pipeline, then the working parameters of the video pipeline are temporally switched so that the video pipeline is configured to process image data corresponding to the at least two bands respectively.

[0097] Preferably, the multi-filter camera device is comprised in a medical camera system.The medical camera system is also preferably used for data aggregation in an operating room, in particular during a medical procedure.

[0098] Preferably, if there are at least two video pipelines, each of the at least two video pipelines is configured to process image data of a specific wavelength band. In other words, each of the at least two video pipelines is dedicated to image data of a specific wavelength band, or in particular, image data based on filtered light of a specific wavelength band.

[0099] Preferably, the image sensor comprises a color sensor.

[0100] Preferably, the image sensor is configured to determine image data having a frame rate that is at least twice a desired frame rate of the multi-filter camera device.

[0101] Preferably, the processing device, or in particular the sequencer device, is configured to synchronize the multi-filter device with the sequencer device. In other words, the multi-filter device is controlled to filter the collected light according to at least two wavelength bands in a manner synchronized with the sequencer's distribution of respective image data to the corresponding at least two video pipelines. For example, when the sequencer distributes respective image data determined from filtered light having a first wavelength band to a video pipeline configured to process such image data, the multi-filter device is controlled to filter the collected light having one of the at least two wavelength bands (e.g., the first wavelength band).

[0102] Furthermore, the processing device, or in particular the sequencer device, is preferably configured to synchronize the multi-filter device and the image sensor with the sequencer device. Thus, not only the multi-filter device is controlled to be synchronized with the sequencer device, but also the image sensor is controlled to be synchronized with the sequencer device, if such control is necessary for the image sensor to process filtered light of different wavelength bands.

[0103] Preferably, the at least two wavebands include a visible light waveband and at least one non-visible light waveband. The at least one non-visible waveband includes a near infrared waveband, an infrared waveband and / or an ultraviolet waveband.

[0104] Because the multi-filter camera device provides time-interleaved light of at least two bands, the multi-filter camera device can process light of different bands in real time without stopping the video pipeline to switch the filter to another band. For example, stopping the video pipeline and switching the filter to the infrared band and switching the filter back to the visible band may take up to two seconds, where the multi-filter camera device is essentially blind with respect to visible light. The multi-filter device allows for rapid switching between different bands, so that the technical boundary is more precisely located in the frame rate of the output video. Given a video output of 30 frames per second, where only one frame per second is provided with light in the infrared band, the multi-filter camera device is blind for only 1 frame (i.e., approximately 16ms).

[0105] Therefore, the multi-filter camera device allows real-time switching between different video modes, such as infrared tracking mode and RGB mode. In infrared tracking mode, the infrared video pipeline provides infrared images based on infrared band image data. In RGB mode, the RGB pipeline provides RGB images based on visible light band image data.

[0106] Therefore, a multi-filter camera device uses only one image sensor to acquire image data in different wavelength bands.

[0107] Therefore, a multi-filter camera device supporting different wavelength bands has a more compact structure.

[0108] In a preferred embodiment, the multi-filter device comprises a rotatable flywheel having at least two filters, wherein each of the at least two filters is configured to filter light in one of the at least two wavelength bands.

[0109] The rotatable flywheel is disposed between the input optics (ie, the lens) and the image sensor such that only one of the at least two filters filters the collected light and only one wavelength band of filtered light is simultaneously determined by the rotatable flywheel.

[0110] Preferably, the flywheel is in the form of a circle comprising a plurality of segments. Each segment of the flywheel comprises a filter.

[0111] As the flywheel rotates, different sections of the flywheel and such filters pass through on the way the beam of collected light reaches the image sensor.

[0112] Thus, an improved multi-filter camera device supporting different wavelength bands is provided.

[0113] In a preferred embodiment, the rotatable flywheel includes a plurality of filters configured to filter light in the same wavelength band of at least two wavelength bands.

[0114] A flywheel is also called a filter wheel.

[0115] For example, the rotatable flywheel includes only one filter for the near infrared band and multiple filters for the visible light band.

[0116] Using more segments and such filters on the flywheel allows the flywheel's rotational speed to be reduced. For example, the flywheel is equipped with four filters, two filters for each of the at least two wavelength bands, for a lower rotational speed of the flywheel.

[0117] Thus, an improved multi-filter camera device supporting different wavelength bands is provided.

[0118] In a preferred embodiment, at least two filters are arranged in staggered segments around the flywheel.

[0119] Interleaving at least two filters around the flywheel allows moving through the different filters by simply rotating the flywheel.

[0120] Furthermore, the staggering of the segments allows for controlling the rotational speed of the flywheel to control the temporal staggering of the filtered light.

[0121] Thus, an improved multi-filter camera device supporting different wavelength bands is provided.

[0122] In a preferred embodiment, the multi-filter camera device includes a number of filters configured to filter light in one of the at least two wavelength bands that is different from the number of filters configured to filter light in the other of the at least two wavelength bands.

[0123] In other words, a multi-filter camera device, and in particular a multi-filter device, includes an asymmetric number of filters for different wavelength bands. As a result, video determined by the video pipeline associated with filtered light from wavelength bands having a greater number of filters has a higher frame rate than video determined by the video pipeline associated with filtered light from wavelength bands having a smaller number of filters.

[0124] For example, a multi-filter device includes six interleaved filters, where four filters include a first wavelength band and two filters include a second wavelength band. Images (i.e., video) associated with the first wavelength result in a greater frame rate for images (i.e., video) associated with the second wavelength.

[0125] Preferably, there are more filters for visible light than for invisible light because higher frame rates are generally preferred in RGB video compared to infrared, near infrared, or ultraviolet video, which are most preferred for tracking.

[0126] It is further preferred that the multi-filter device comprises a plurality of non-visible light filters. This allows the multi-filter device to be more sensitive in specific different near-infrared, infrared or ultraviolet wavelength bands. This is suitable for example for fluoroscopy applications.

[0127] Thus, an improved multi-filter camera device supporting different wavelength bands is provided.

[0128] In a preferred embodiment, the camera device comprises a motor configured to rotate the flywheel.

[0129] The motor that rotates the flywheel allows the processing equipment to control the speed of rotation of the flywheel.

[0130] Thus, the processing device is able to synchronize the flywheel and such a multi-filter device with the sequencer device.

[0131] Thus, an improved multi-filter camera device supporting different wavelength bands is provided.

[0132] In a preferred embodiment, the sequencer device is configured to distribute the image data based on the position of the flywheel.

[0133] The flywheel position includes information about which part of the flywheel is currently positioned along the path of the collected light beam to the image sensor. This information is provided to a processing device, particularly a sequencer. The sequencer thus obtains real-time information about the structure of the filtered light, or in other words, which image data corresponds to which wavelength band at which time. The sequencer can thus associate image data for a specific wavelength band with the corresponding video pipeline in real time.

[0134] In other words, the position of the flywheel indicates the segment of the flywheel that corresponds to a filter of a particular wavelength band. For example, for each position of the flywheel, it is predetermined which filter of which wavelength band is currently exposed to the collected light. For example, for a four-segment flywheel with interleaved filters A and B, it is predetermined that the flywheel positions of 0° to 90° and 181° to 270° are associated with filter A, and the flywheel positions of 91° to 180° and 271° to 360° are associated with filter B.

[0135] Therefore, the position of the flywheel corresponds directly to the filter exposed to the collected light.

[0136] Thus, an improved multi-filter camera device supporting different wavelength bands is provided.

[0137] In a preferred embodiment, the camera device comprises a position sensor configured to determine the position of the flywheel.

[0138] In a preferred embodiment, the sequencer device is configured to determine the position of the flywheel based on motor feedback from the motor.

[0139] Preferably, the motor is a brushless motor and the motor feedback is back electromotive force.

[0140] Thus, an improved multi-filter camera device supporting different wavelength bands is provided.

[0141] In a preferred embodiment, the multi-filter device comprises a liquid crystal tunable filter, wherein the liquid crystal tunable filter is tunable to filter light in each of at least two wavelength bands.

[0142] In a preferred embodiment, at least one of the wavelength bands includes visible light.

[0143] The bandwidth of the visible light band is larger than that of the near-infrared light band and the infrared light band.

[0144] Preferably, for visible light based image data, the corresponding video pipeline includes a conventional debayering pipeline.

[0145] Thus, an improved multi-filter camera device supporting different wavelength bands is provided.

[0146] In a preferred embodiment, at least one of the wavelength bands includes near infrared light and / or infrared light.

[0147] The bandwidth of the near-infrared light band and the infrared light band is smaller than that of the visible light band.

[0148] Preferably, for image data based on near infrared and / or infrared light, the corresponding video pipeline includes a special de-Bayerization pipeline configured to provide clear monochrome images.

[0149] Thus, an improved multi-filter camera device supporting different wavelength bands is provided.

[0150] In a preferred embodiment, at least one of the wavelength bands includes near ultraviolet light and / or ultraviolet light.

[0151] In a preferred embodiment, the image sensor comprises a CMOS image sensor.

[0152] Preferably, the image sensor is a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor.

[0153] In a preferred embodiment, the multi-filter device is arranged before the image sensor.

[0154] The multi-filter device may be arranged after the lens. However, the multi-filter device may also be arranged before the lens, as long as the multi-filter device is arranged before the image sensor so that only filtered light reaches the image sensor.

[0155] Thus, an improved multi-filter camera device supporting different wavelength bands is provided.

[0156] In a preferred embodiment, the processing device is configured to determine a multi-band image based on the image data.

[0157] In other words, the processing device, in particular at least one video pipeline, determines a video, ie an image, based on the image data provided by the sequencer device.

[0158] Thus, the processing device uses the output of at least one video pipeline to determine a multi-band video, i.e., an image, by combining the output of at least one video pipeline. This allows for the creation of HDR-like multi-wavelength images, where images from all bands are blended together into a single image, using exposures from different bands. Consequently, calibration of the multi-filter camera device is essentially unnecessary, as the same lens and image sensor are used for each band.

[0159] Thus, the multi-filter camera device provides real-time multi-band video.

[0160] Thus, an improved multi-filter camera device supporting different wavelength bands is provided.

[0161] In a preferred embodiment, the light collected by the lens is provided by a light source synchronized with a corresponding filter of the at least two filters of the multi-filter device.

[0162] In other words, if the light source is capable of providing light of different wavelength bands, the processing device, in particular the sequencer device, is configured to control the light source to provide light of corresponding wavelength bands synchronized with the sequencer device.

[0163] Thus, an improved multi-filter camera device supporting different wavelength bands is provided.

[0164] In a preferred embodiment, the light source is a flash light source.

[0165] According to another aspect of the present invention, a multi-filter image processing method includes the following steps: collecting light reflected from a scene through a lens; determining filtered light from the collected light through a multi-filter device, wherein the filtered light includes temporally interleaved light from at least two wavelength bands; determining image data from the filtered light through an image sensor, wherein the image data includes temporally interleaved image data corresponding to the at least two wavelength bands; and processing the image data through a processing device, wherein the processing includes temporally managing the image data and at least one video pipeline of the processing device through a sequencer device, wherein the at least one video pipeline processes image data corresponding to at least one of the at least two wavelength bands.

[0166] Preferably, at least part of the method is a computer implemented method.

[0167] In a preferred embodiment, the multi-filter device comprises a rotatable flywheel having at least two filters, wherein each of the at least two filters is configured to filter light in one of the at least two wavelength bands.

[0168] In a preferred embodiment, the rotatable flywheel includes a plurality of filters configured to filter light in the same wavelength band of at least two wavelength bands.

[0169] In a preferred embodiment, at least two filters are arranged in staggered segments around the flywheel.

[0170] In a preferred embodiment, the camera device comprises a different number of filters configured to filter light in one of the at least two wavelength bands than a number of filters configured to filter light in the other of the at least two wavelength bands.

[0171] In a preferred embodiment, the method comprises rotating the flywheel by a motor of the camera device.

[0172] In a preferred embodiment, the method includes distributing, by a sequencer device, the image data based on the position of the flywheel.

[0173] In a preferred embodiment, the method includes determining the position of the flywheel by means of a positioning sensor of the camera device.

[0174] In a preferred embodiment, the method includes determining, by a sequencer device, the position of the flywheel based on motor feedback from the motor.

[0175] In a preferred embodiment, the multi-filter device comprises a liquid crystal tunable filter, wherein the liquid crystal tunable filter is tunable to filter light in each of at least two wavelength bands.

[0176] In a preferred embodiment, at least one of the wavelength bands includes visible light.

[0177] In a preferred embodiment, at least one of the wavelength bands includes near infrared light and / or infrared light.

[0178] In a preferred embodiment, at least one of the wavelength bands includes near ultraviolet light and / or ultraviolet light.

[0179] In a preferred embodiment, the image sensor comprises a CMOS image sensor.

[0180] In a preferred embodiment, the multi-filter device is arranged before the image sensor.

[0181] In a preferred embodiment, the method includes determining, by a processing device, a multi-band image based on the image data.

[0182] In a preferred embodiment, the light collected by the lens is provided by a light source synchronized with a corresponding filter of the at least two filters of the multi-filter device.

[0183] In a preferred embodiment, the light source is a flash light source.

[0184] According to another aspect, a multi-camera system includes a composite image camera apparatus as described herein and a multi-filter camera apparatus as described herein.

[0185] At least part of the present invention relates to a computer program that, when executed on at least one processor (e.g., a processor) of at least one computer (e.g., a computer), or when loaded into at least one memory (e.g., a memory) of at least one computer (e.g., a computer), causes the at least one computer to perform the method according to the first aspect. Alternatively or additionally, the present invention may relate to a (physical, e.g., electrical, e.g., technologically generated) signal wave, e.g., a digital signal wave, that carries information representing a program (e.g., the program described above), the program e.g., including code means for performing any or all of the steps of the method according to the first aspect. A computer program stored on a disk is a data file, and when the file is read and transmitted, it becomes a data stream, e.g., in the form of (physical, e.g., electrical, e.g., technologically generated) signals. The signals may be implemented as signal waves as described herein. For example, the signals (e.g., signal waves) may be configured to be transmitted via a computer network (e.g., a LAN, WLAN, WAN, e.g., the Internet). Therefore, the present invention according to the second aspect may alternatively or additionally relate to a data stream representing the program described above.

[0186] In a third aspect, the present invention relates to a non-transitory computer-readable program storage medium having stored thereon the program according to the fourth aspect.

[0187] For example, the present invention does not involve or specifically include or encompass invasive steps that would represent a substantial physical disturbance of the body, requiring specialized medical expertise to perform and posing substantial health risks even when performed with the required specialized care and expertise. For example, the present invention does not include steps for positioning a medical implant for fastening it to an anatomical structure, or steps for fastening a medical implant to an anatomical structure, or steps for preparing an anatomical structure for fastening a medical implant thereto. More specifically, the present invention does not involve or specifically include or encompass any surgical or therapeutic activity. Rather, the present invention relates to methods suitable for calibrating camera models and switching between different medical camera device modes. For this reason alone, surgical or therapeutic activity, in particular surgical or therapeutic steps, is not required or implied for implementation of the present invention.

[0188] definition

[0189] In this section, definitions of certain terms used in this disclosure are provided, which also form a part of this disclosure.

[0190] Computer-implemented methods

[0191] The method according to the present invention is, for example, a computer-implemented method. For example, all steps or only some steps (i.e., less than the total number of steps) of the method according to the present invention can be performed by a computer (e.g., at least one computer). An embodiment of a computer-implemented method is to use a computer to perform a data processing method. An embodiment of a computer-implemented method is a method involving the operation of a computer such that the computer is operated to perform one, more, or all steps of the method.

[0192] A computer, for example, includes at least one processor and at least one memory for (technically) processing data, for example, electronically and / or optically. The processor, for example, is made of a semiconductor substance or composition, for example, at least partially n-type doped semiconductors and / or p-type doped semiconductors, for example, at least one of type II, III, IV, V, or VI semiconductor materials, for example, (doped) silicon and / or gallium arsenide. The calculation or determination steps described are, for example, performed by a computer. A determination step or calculation step is, for example, a step of determining data within the framework of a technical method (for example, within the framework of a program). A computer is, for example, any type of data processing device, for example, an electronic data processing device. A computer can be a device commonly considered such, such as a desktop PC, laptop, netbook, etc., but can also be any programmable device, such as a mobile phone or embedded processor. A computer can, for example, include a system (network) of "sub-computers," each of which represents its own computer. The term "computer" includes cloud computers, such as cloud servers. The term "cloud computer" includes a cloud computer system, for example, including at least one cloud computer and, for example, a system of multiple operatively interconnected cloud computers (such as a server farm). Such cloud computers are preferably connected to a wide area network such as the World Wide Web (WWW) and are located in a so-called cloud of computers that are all connected to the World Wide Web. Such infrastructure is used for "cloud computing", which describes computing, software, data access and storage services that do not require the end user to know the physical location and / or configuration of the computers that deliver specific services. For example, the term "cloud" is used in this context as a metaphor for the Internet (World Wide Web). For example, the cloud provides computing infrastructure as a service (IaaS). Cloud computers can be used as virtual hosts for operating systems and / or data processing applications for performing the methods of the present invention. Cloud computers are, for example, provided by Amazon Web Services TMThe provided elastic computing cloud (EC2). The computer, for example, includes an interface for receiving or outputting data and / or performing analog-to-digital conversion. The data, for example, represents physical properties and / or is generated from technical signals. The technical signals are generated, for example, by means of (technical) detection equipment (for example, equipment for detecting a marking device) and / or (technical) analysis equipment (for example, equipment for performing a (medical) imaging method), wherein the technical signals are, for example, electrical signals or optical signals. The technical signals, for example, represent data received or output by the computer. The computer is preferably operatively coupled to a display device that allows information output by the computer to be displayed to, for example, a user. An example of a display device is a virtual reality device or an augmented reality device (also referred to as virtual reality glasses or augmented reality glasses), which can be used as "goggles" for navigation. A specific example of such augmented reality glasses is Google Glass (trademark of Google Inc.). The augmented reality device or virtual reality device can be used to input information into the computer through user interaction, and can also be used to display information output by the computer. Another example of a display device would be a standard computer monitor, including, for example, a liquid crystal display operatively coupled to a computer for receiving display control data from the computer to generate signals for displaying image information content on the display device. A specific embodiment of such a computer monitor is a digital light box. An example of such a digital light box is the product of Brainlab AG The monitor may also be a monitor of a portable (eg handheld) device such as a smartphone or a personal digital assistant or a digital media player.

[0193] The invention also relates to a program which, when run on a computer, causes the computer to perform one or more or all of the method steps described herein, and / or to a program storage medium on which the program is stored (in particular in a non-transitory form), and / or to a computer comprising such a program storage medium, and / or to a (physical, e.g. electrical, e.g. technically generated) signal wave, e.g. a digital signal wave, which carries information representing a program (e.g. the above-mentioned program), which program, for example, comprises code means for performing any one or all of the method steps described herein.

[0194] Within the framework of the present invention, a computer program element may be embodied by hardware and / or software (this includes firmware, resident software, microcode, etc.). Within the framework of the present invention, a computer program element may take the form of a computer program product, which may be implemented by a computer-usable (e.g., computer-readable) data storage medium comprising computer-usable (e.g., computer-readable) program instructions, "code," or "computer program" embodied in the data storage medium for use on or in conjunction with an instruction execution system. Such a system may be a computer; a computer may be a data processing device comprising means for executing a computer program element and / or program according to the present invention, such as a data processing device comprising a digital processor (central processing unit, or CPU) for executing a computer program element and optionally a volatile memory (e.g., random access memory, or RAM) for storing data used for and / or generated by executing the computer program element. Within the framework of the present invention, a computer-usable (e.g., computer-readable) data storage medium may be any data storage medium that can contain, store, convey, propagate, or transmit a program for use on or in conjunction with an instruction execution system, apparatus, or device. Computer-usable (e.g., computer-readable) data storage media can be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or communication media, such as the Internet. Computer-usable or computer-readable data storage media can even be, for example, paper or another suitable medium on which the program is printed, because the program can be electronically captured, for example, by optically scanning paper or other suitable media, and then compiled, interpreted, or otherwise processed in a suitable manner. The data storage medium is preferably a non-volatile data storage medium. The computer program product described herein and any software and / or hardware form various devices for performing the functions of the present invention in the example embodiments. The computer and / or data processing device can, for example, include a guidance information device, which includes a device for outputting guidance information. The guidance information can be output to the user visually, for example, by a visual indicator (e.g., a monitor and / or a light) and / or acoustically, and / or tactilely, by an acoustic indicator (e.g., a speaker and / or a digital voice output device) and / or tactilely, by a tactile indicator (e.g., a vibration element or a vibration element incorporated into an instrument). For the purposes of this document, a computer is a technical computer, e.g., comprising technical (e.g., tangible) components, such as mechanical and / or electronic components. Any device so mentioned in this document is a technical device, e.g., a tangible device.

[0195] Get data

[0196] For example, the expression "acquire data" includes (within the context of a computer-implemented method) scenarios in which data is determined by a computer-implemented method or program. For example, determining data includes measuring a physical quantity and converting the measured value into data, such as digital data, and / or calculating (and, for example, outputting) the data by a computer, for example within the context of the method according to the present invention. For example, the meaning of "acquire data" also encompasses scenarios in which data is received or obtained by a computer-implemented method or program (e.g., input to the computer-implemented method or program), for example, from another program, a previous method step, or a data storage medium, for example, to be further processed by the computer-implemented method or program. The generation of the data to be acquired may, but need not, be part of the method according to the present invention. Thus, the expression "acquire data" may also, for example, mean waiting for data to be received and / or receiving data. The received data may, for example, be input via an interface. The expression "acquire data" may also mean that a computer-implemented method or program performs steps to (actively) receive or obtain data from a data source (e.g., a data storage medium (e.g., ROM, RAM, database, hard drive, etc.)) or via an interface (e.g., from another computer or network). The data acquired by the disclosed methods or devices, respectively, can be acquired from a database located in a data storage device, which is operatively connected to a computer for data transfer between the database and the computer, e.g., from the database to the computer. The computer acquires the data for use as input in the step of determining the data. The determined data can be output again to the same or another database for storage for later use. The database, or a database used to implement the disclosed methods, can be located on a network data storage device or network server (e.g., a cloud data storage device or cloud server) or a local data storage device (such as a mass storage device operatively connected to at least one computer performing the disclosed methods). The data can be made "ready for use" by performing an additional step prior to the acquisition step. According to this additional step, the data is generated for acquisition. The data is, for example, detected or captured (e.g., by an analysis device). Alternatively or additionally, according to the additional step, the data is input, for example, via an interface. The generated data can be input, for example, into a computer. According to the additional step (which precedes the acquisition step), the data can also be provided by performing an additional step of storing the data in a data storage medium (e.g., ROM, RAM, CD, and / or hard drive) so that it is ready for use within the framework of the method or program according to the present invention. Thus, the step of "obtaining data" may also involve instructing the device to obtain and / or provide the data to be acquired. Specifically, the step of acquiring does not involve an invasive step that would represent a substantial physical interference with the body, thereby requiring professional medical expertise to perform and posing a substantial health risk even when performed with the required professional care and expertise.In particular, the step of acquiring data (e.g., determining data) does not involve a surgical step, and in particular does not involve a step of treating the human or animal body using surgery or therapy. In order to distinguish between the different data used by the present method, the data are denoted (i.e., referred to) as "XY data" or the like, and are defined according to the information they describe, and are then preferably referred to as "XY information" or the like.

[0197] Mapping

[0198] The mapping describes a transformation (e.g., a linear transformation) of elements (e.g., pixels or voxels), e.g., positions of elements, of a first data set in a first coordinate system to elements (e.g., pixels or voxels), e.g., positions of elements, of a second data set in a second coordinate system (which may have a basis different from that of the first coordinate system). In one embodiment, the mapping is determined by comparing (e.g., matching) color values (e.g., grayscale values) of corresponding elements using an elastic or rigid fusion algorithm. The mapping is embodied, for example, by a transformation matrix (e.g., a matrix defining an affine transformation).

[0199] Elastic fusion, image fusion / deformation, rigidity

[0200] Image fusion can be elastic image fusion or rigid image fusion. In the case of rigid image fusion, the relative positions between pixels of the 2D image and / or voxels of the 3D image are fixed, while in the case of elastic image fusion, the relative positions are allowed to change.

[0201] In this application, the term "image deformation" is also used as an alternative to the term "elastic image fusion", but has the same meaning.

[0202] An elastic fusion transform (e.g., an elastic image fusion transform) is designed, for example, to achieve a seamless transition from one dataset (e.g., a first dataset, such as, for example, a first image) to another dataset (e.g., a second dataset, such as, for example, a second image). The transform is designed, for example, to deform one of the first and second datasets (images) in such a way that corresponding structures (e.g., corresponding image elements) are arranged at the same locations as in the other of the first and second images. The deformed (transformed) image transformed from one of the first and second images is, for example, as similar as possible to the other of the first and second images. Preferably, a (numerical) optimization algorithm is applied to find the transformation that results in the best similarity. The degree of similarity is preferably measured by a similarity metric (hereinafter also referred to as a "similarity metric"). The parameters of the optimization algorithm are, for example, vectors of the deformation field. These vectors are determined by the optimization algorithm in a manner that results in the best similarity. Therefore, the best similarity represents a condition, e.g., a constraint, of the optimization algorithm. The basis of the vectors is located, for example, at voxel locations in one of the first and second images to be transformed, and the tips of the vectors are located at corresponding voxel locations in the transformed image. Preferably, a plurality of these vectors is provided, for example, more than twenty, one hundred, one thousand, ten thousand, etc. Preferably, (other) constraints are imposed on the transformation (deformation), for example, to avoid pathological deformations (e.g., all voxels are shifted to the same position by the transformation). These constraints include, for example, the constraint that the transformation is regular, which means, for example, that the Jacobian calculated from the matrix of the deformation field (e.g., vector field) is greater than zero, and the constraint that the transformed (deformed) image is not self-intersecting and, for example, that the transformed (deformed) image does not include faults and / or fractures. Constraints include, for example, the constraint that if a regular grid is transformed simultaneously and in a corresponding manner with the image, the grid is not allowed to intersect at any position. For example, the optimization problem is solved iteratively, for example, using an optimization algorithm, which is, for example, a first-order optimization algorithm, such as a gradient descent algorithm. Other examples of optimization algorithms include optimization algorithms that do not use derivatives, such as the downhill simplex algorithm, or algorithms that use higher-order derivatives, such as Newton-type algorithms. The optimization algorithm preferably performs local optimization. If multiple local optima exist, a global algorithm or a general algorithm, such as simulated annealing, can be used. In case of linear optimization problems, for example the simplex method can be used.

[0203] During the optimization steps, voxels are shifted, for example, in one direction by an amount such that the similarity increases. This amount is preferably less than a predefined limit, for example, less than one tenth, one hundredth, or one thousandth of the image diameter, and is, for example, approximately equal to or less than the distance between adjacent voxels. Large deformations can be achieved, for example, due to a large number of (iterative) steps.

[0204] The determined elastic fusion transformation can, for example, be used to determine the similarity (or similarity measure, see above) between the first and second data sets (first and second images). To this end, the deviation between the elastic fusion transformation and the identity transformation is determined. The degree of deviation can be calculated, for example, by determining the difference between the determinant of the elastic fusion transformation and the identity transformation. The higher the deviation, the lower the similarity, and thus the degree of deviation can be used to determine a measure of similarity.

[0205] For example, the measure of similarity may be determined based on the determined correlation between the first data set and the second data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0206] In the following, the invention is described with reference to the accompanying drawings which give a background explanation and show specific embodiments of the invention. However, the scope of the invention is not limited to the specific features disclosed in the context of the accompanying drawings, in which

[0207] Figure 1 A medical camera system is shown;

[0208] Figure 2 An image synthesis device for determining a synthetic image of a scene according to a first embodiment is shown;

[0209] Figure 3 An image synthesis device for determining a synthetic image of a scene according to a second embodiment is shown;

[0210] Figure 4 An image synthesis method for determining a composite image of a scene is shown;

[0211] Figure 5 A multi-filter camera device is shown;

[0212] Figure 6 shows temporally staggered light in a multi-filter camera setup;

[0213] Figure 7 A multi-filter device with a flywheel is shown; and

[0214] Figure 8 A multi-filter image processing method for a multi-filter camera device is shown. DETAILED DESCRIPTION

[0215] Figure 1 The medical camera system 100 is shown as a multi-sensing system. The medical camera system 100 in this example has a range of approximately 1000 cm 3 The volume describes the size.

[0216] Medical camera systems are used in medical environments such as operating rooms or surgical suites for monitoring (i.e., instrument usage, tray monitoring, equipment setup / position, people and their movements), tracking (i.e., single tracking, event / workflow / people tracking, video feeds), and / or recording (i.e., documentation) tasks. Medical camera systems are primarily attached to the ceiling in a fixed position and orientation, but can also be mounted on a movable support structure, such as on a robotic device.

[0217] Medical camera system 100 is configured to acquire image data of a medical environment. Therefore, the medical camera system includes at least two camera subsystems. In this case, the at least two camera subsystems include a composite image camera device 110 for determining a composite image of a scene and a multi-filter camera device 120.

[0218] The synthetic image camera device 110 for determining a synthetic image of a scene is configured to acquire a 360-degree view and includes two visible light cameras 10, also referred to as RGB cameras, wherein the image data (e.g., video data) acquired by the RGB cameras are combined via stitching. The synthetic image camera device 110 for determining a synthetic image of a scene also includes two distance sensors 20. Figure 2 and Figure 3 The composite image camera device 110 for determining a composite image of a scene is described in further detail in .

[0219] The multi-filter camera device 120 is also referred to as a main camera, since this camera is primarily used for monitoring points of interest in an operating room. The multi-filter camera device 120 comprises switchable filters for different wavelength bands, in particular switchable filters for visible light, infrared, near infrared and / or ultraviolet. Thus, the multi-filter camera device 120 combines RGB image acquisition with 2D image tracking. In this example, the multi-filter camera device comprises a lens 41 for collecting visible light and invisible light. The invisible light is, for example, infrared light reflected by a tracked object, wherein the tracked object is illuminated by infrared light of an infrared source 50 of the multi-filter camera device 120. In Figures 5 to 7 The multi-filter camera device 120 is described in further detail in .

[0220] Medical camera system 100 includes a processing device 30 to implement the functionality of a composite image camera device 110 and a multi-filter camera device 120 for determining a composite image of a scene. Processing device 30 may also be supported externally, i.e., by a cloud, fog, and / or edge device 80.

[0221] In this example, the medical camera system 100 includes some optional additional components:

[0222] The inertial measurement unit IMU 70 (e.g., three-axis accelerometer, gyroscope, etc., in particular 3 gyroscopes + 3 accelerometers) is configured to determine the orientation of the medical camera system (e.g., to adjust the horizontal line of the first camera subsystem and / or the second camera subsystem).

[0223] The microphone 60 may be a microphone array or a directional microphone having up to 6 microphones.

[0224] A light projector for projecting lines and points onto the patient, for example based on anatomical objects or based on user input or remote user input (eg, telestration).

[0225] A third camera device, such as on a crane, lever arm, rails, or wireless means, is configured to be oriented toward a region of interest, particularly the same region of interest as the second camera system.

[0226] a pre-processor in a common housing for the first and second camera subsystems, the pre-processor configured to perform image processing on the raw data, such as edge detection, compression, segmentation, classification, optionally based on AI (background: streaming bandwidth is limited; on-device compression / pre-processing will only stream relevant data instead of raw image data, e.g., the processor selects a ROI and only forwards image data relevant to the ROI, see also Virtual Camera).

[0227] The external ROI camera is a separate highly zoomable and fixed focus camera on a flexible arm to focus on a specific area of interest, such as a surgical tray.

[0228] 360° view digital zoomer with different triggers to set zoom and position based on sound, human movement or meeting.

[0229] Voice or gesture controls, such as start / stop recording.

[0230] A user interface, such as a touch screen or touch field on a camera, is used to define where to look.

[0231] Therefore, the medical camera system 100 provides multi-sensing applications in the medical field.

[0232] Figure 2 Shown according to Figure 1The first embodiment of the present invention is consistent with the synthetic camera device 110. The synthetic camera device 210 includes four visible light cameras 10 and four distance sensors 20, which provide image data to a processing device 30. The four visible light cameras 10 and four distance sensors 20 are staggered in a circle to cover a 360-degree view of the scene. In the 360-degree view, multiple objects O, marked by circles, are identified. Each of the four distance sensors 20 covers a 90-degree range segment S1-S4 and provides the processing device 30 with distance information regarding the nearest object among the multiple objects O within the corresponding range segment S1-S4. In other words, each of the four distance sensors 20 provides the processing device 30 with distance information D1-D4. The processing device 30 then uses the provided distance information D1-D4 to calibrate the camera models of the four visible light cameras 10 to reduce parallax at the corresponding distances provided by the distance information D1-D4 within the corresponding range segments S1-S4. It can be seen from the distance segment S2 that only the closest object O in the distance segments S1 to S4 is related to the distance sensor 20 .

[0233] This allows processing device 30 to more accurately calibrate the camera models of the four visible light cameras 10 according to the environment. This allows for an improved synthesis of images of a scene based on the images provided by each of the four visible light cameras 10.

[0234] Figure 3 A composite camera device 310 for determining a composite image of a scene according to a second embodiment is shown. Figure 1 The synthesis of the camera device 110 is consistent.

[0235] and Figure 2 Compared to the composite camera device 310, each of the four distance sensors 10 has a resolution of 2×2 and thus provides two distance information for two distance segments. Thus, the processing device 30 is provided with eight distance information D1 to D8 from eight distance segments S1 to S8, each covering 45 degrees of the 360-degree view.

[0236] Figure 3 It is also shown that if the object O is not detected in a distance section (eg, S4 and S6 ), the corresponding distance information includes the distance between the corresponding distance sensor 20 and an end of the room (eg, a floor or a wall).

[0237] Figure 4A medical method for determining a composite image of a scene is shown, comprising the following steps. A first step S10 includes obtaining (S10) an image of the scene by each of a plurality of visible light cameras 10 covering a 360-degree view of the scene. A second step S20 includes obtaining, by each of a plurality of distance sensors 20 covering a plurality of distance bins S1, S2 of the 360-degree view of the scene, distance information D1, D2 from the corresponding distance sensor 20 to a nearest object O for at least one of the plurality of distance bins S1, S2. A third step S30 includes calibrating, by a processing device, a camera model of one of the plurality of visible light cameras 10 for each of the plurality of distance bins S1, S2 using the determined corresponding distance information D1, D2. A fourth step S40 includes determining, by a processing device, a composite image of the scene using the calibrated camera model for each of the plurality of distance bins S1, S2 and an image of the scene from each of the plurality of visible light cameras.

[0238] Figure 5 Shown is a multi-filter camera device 120. The multi-filter camera device 120 comprises a processing device 30, input optics in the form of a lens 41, a multi-filter device 42 and an image sensor 43. The processing device 30 comprises a sequencer device 31, a first video pipeline 32 and a second video pipeline 33.

[0239] The lens 41 is configured to collect light L reflected from a scene. In this case, the scene comprises a three-dimensional view of an area, particularly an operating room. The scene, also referred to as a region of interest, is monitored by the camera system 100. The camera system 100 comprises a multi-filter camera device 120.

[0240] The multi-filter device 42 is configured to determine filtered light Lf from the collected light L. The filtered light Lf includes light of two temporally interleaved wavelength bands (a first wavelength band and a second wavelength band). In other words, the multi-filter device 42 includes a first filter and a second filter, wherein the first filter is configured to filter the collected light L with the first wavelength band, and the second filter is configured to filter the collected light L with the second wavelength band. The multi-filter device 42 is configured to filter the collected light L using one of the two filters at a time. Therefore, the filtered light Lf includes a beam of light having two temporally interleaved wavelength bands. In other words, the multi-filter device 42 filters the collected light L using a first filter of the first wavelength band for a first time window and a second filter of the second wavelength band for a second time window. Therefore, the multi-filter device 42 is configured to switch between the filters and, therefore, switch between filtering the collected light L with the first wavelength band and with the second wavelength band. The multi-filter device 42 provides the filtered light Lf to the image sensor 43 .

[0241] The image sensor 43 is configured to determine image data D from the filtered light Lf, wherein the image data D includes temporally interleaved image data corresponding to at least two wavelength bands. In other words, the image sensor 43 is configured to convert the filtered light Lf of any one of the at least two wavelength bands into the corresponding image data D. Thus, the image sensor 43 digitizes the filtered light Lf into image data D and provides the image data D to the processing device 30 for further processing.

[0242] The processing device 30 is configured to process image data D. The first video pipeline 32 is configured to process image data D corresponding to a first wavelength band. The second video pipeline 33 is configured to process image data D corresponding to a second wavelength band. The sequencer device 31 is configured to temporally distribute the image data D to the first video pipeline 32 and the second video pipeline 33. In other words, the sequencer device 31 is configured to provide the image data D corresponding to the first wavelength band to the first video pipeline 32 in a first time window, and to provide the image data D corresponding to the second wavelength band to the second video pipeline 33 in a second time window.

[0243] The first video pipeline 32 and the second video pipeline 33 determine a video of the scene, ie images over time.

[0244] The sequencer device 31 is configured to control the multi-filter device 42 to use the corresponding first filter or second filter in accordance with the distribution of the first image data and the second image data provided by the image sensor 43 to the first video pipeline 32 and the second video pipeline 33, respectively. Therefore, the sequencer device 31 uses the control data C to ensure that the multi-filter device uses filters with the correct wavelength band on the collected light L, so that the first video pipeline 32 and the second video pipeline 33 are only provided with image data D corresponding to the wavelength band that the corresponding video pipeline is configured to process.

[0245] This allows for the provision of real-time medical camera devices that support different bands.

[0246] Figure 6 Shown Figure 5 The multi-filter camera device 120. However, Figure 6 Also shown is a temporally staggered light having a first wavelength band Bl and a second wavelength band B2. Figure 6 It is thus shown that the image data corresponding to the first band B1 is supplied only to the first video pipeline 31 and the image data corresponding to the second band B2 is supplied only to the second video pipeline 32 .

[0247] The filtered light Lf is a light beam having temporally interleaved wavelength bands (i.e., a first wavelength band B1 and a second wavelength band B2). Therefore, when the corresponding image data is assigned to the first video pipeline 31 and the second video pipeline 32, the image data is provided to the first video pipeline 31 or the second video pipeline 32, but never simultaneously. This leaves a blind spot for each video pipeline 31, 32. In the example, the first wavelength band B1 relates to visible light, and the second wavelength band B2 relates to infrared light. The first wavelength band B1 is used to provide an RGB video of the scene to the user, while the second wavelength band B2 is used to track an object via infrared video (or other special light applications, such as fluoroscopy). Therefore, the multi-filter device 42 is controlled to filter the collected light L with the first wavelength band B1, which is significantly longer than the second wavelength band B2 used for filtering the collected light L. Therefore, the blind spot of the RGB video is relatively small, and the tracking function of the infrared video is still provided.

[0248] This allows for the provision of real-time medical camera devices that support different bands.

[0249] Figure 7 Shown Figure 5 The multi-filter camera device 120 is shown in FIG. However, the multi-filter device 42 is implemented by a rotatable flywheel 42a. The rotatable flywheel 42a has a circular shape and includes multiple segments surrounding the flywheel 42a. Each segment of the flywheel 42a is provided with a filter. In this case, the flywheel 42a includes four segments arranged alternately around the flywheel 42a and four filters, namely, twice the first filter F1 and twice the second filter F2.

[0250] The multi-filter device 42 includes a motor 42b configured to rotate the flywheel 42a. The sequencer device 31 is configured to control the motor 42b. Furthermore, the multi-filter device 42 includes a position sensor 42c. The position sensor 42c is configured to determine the position P of the flywheel 42a, or in other words, which filter, the first filter F1 or the second filter F2, is currently filtering the collected light L. The position P of the flywheel 42a also includes information about the position of each filter of the flywheel 42a relative to the collected light beam L. Therefore, the sequencer device 31 can use the position of the flywheel 42a to control the motor 42b and / or control the distribution of image data to the first video pipeline 32 and the second video pipeline 33.

[0251] Figure 8 A multi-filter image processing method for the multi-filter camera device 120 is shown.

[0252] A first step Z10 comprises collecting light Z10 reflected from a scene via a lens 41. A second step Z20 comprises determining filtered light Lf from the collected light via a multi-filter device 42, wherein the filtered light Lf comprises temporally interleaved light of at least two wavelength bands B1, B2. A third step Z30 comprises determining image data D from the filtered light Lf via an image sensor 43, wherein the image data D comprises temporally interleaved image data corresponding to the at least two wavelength bands B1, B2. A fourth step Z40 comprises processing the image data D via a processing device 30, wherein the processing comprises temporally managing the image data D via a sequencer device 31 and at least one video pipeline 32, 33 of the processing device 30, wherein the at least one video pipeline 32, 33 processes the image data D corresponding to at least one of the at least two wavelength bands B1, B2.

Claims

1. A composite camera apparatus for determining a composite image of a scene, comprising: a plurality of visible light cameras (10) covering a 360-degree view of the scene, wherein each visible light camera (10) of the plurality of visible light cameras (10) is configured to determine an image of the scene; a plurality of distance sensors (20) covering a plurality of distance segments (S1, S2) of a 360-degree view of the scene, wherein each distance sensor (20) of the plurality of distance sensors (20) is configured to: determine distance information (D1, D2) from the corresponding distance sensor (20) to a nearest object (O) for at least one distance segment of the plurality of distance segments (S1, S2); A processing device (30) configured to: calibrating a camera model of a visible light camera among the plurality of visible light cameras (10) using corresponding distance information (D1, D2) determined for each of the plurality of distance segments (S1, S2); and A composite image of the scene is determined using a calibrated camera model for each of the plurality of distance bins (S1, S2) and an image of the scene from each of the plurality of visible light cameras.

2. The device according to claim 1, in, Calibrating the camera model includes: Obtain multiple camera models that are statically pre-calibrated for different distances; For each distance bin of the plurality of distance bins (S1, S2), a selected camera model of the plurality of camera models is selected based on the determined distance information (D1, D2) of the corresponding distance bin (S1, S2).

3. The device according to claim 1, in, Calibrating the camera model includes: Obtaining a parameterizable calibration model; For each distance segment of the plurality of distance segments (S1, S2), a parameterizable calibration model is parameterized based on the determined distance information (D1, D2) of the respective distance segment (S1, S2).

4. The device according to any one of the preceding claims, in, The camera model includes an intrinsic compensation model and an extrinsic compensation model.

5. The device according to any one of the preceding claims, in, The at least one distance sensor (20) is a low-resolution distance sensor.

6. The device according to claim 5, in, The low-resolution distance sensor (20) has a resolution of one of 4x4, 2x2 or 1x1.

7. The device according to any one of the preceding claims, in, The number of the plurality of visible light cameras (10) is smaller than the number of the at least one distance sensor (20).

8. The device according to any one of the preceding claims, in, The plurality of distance sensors are arranged to be interleaved with the plurality of visible light cameras (10).

9. The device according to any one of the preceding claims, in, The plurality of distance sensors (20) include one of a time-of-flight sensor and a LIDAR sensor.

10. The device according to any one of the preceding claims, in, The processing device (30) is configured to calibrate the camera model and determine the composite image within a predetermined time period.

11. The device according to any one of the preceding claims, in, The plurality of visible light cameras (10) are cameras with a large field of view.

12. The device according to claim 11, in, Each of the large field of view cameras includes a field of view of at least 160 degrees.

13. The device according to claim 11 or 12, in, The large field of view camera includes a camera with a fisheye lens.

14. The device according to any one of the preceding claims, in, The processing device (30) comprises a hardware processing device.

15. A medical method for determining a composite image of a scene, comprising the steps of: Obtaining (S10) an image of the scene by each of a plurality of visible light cameras (10) covering a 360-degree view of the scene; Obtaining (S20) distance information (D1, D2) from each of a plurality of distance sensors (20) covering a plurality of distance segments (S1, S2) of a 360-degree view of the scene to a nearest object (O) for at least one of the plurality of distance segments (S1, S2); Calibrate (S30) a camera model of one visible light camera (10) among the plurality of visible light cameras (10) using the determined corresponding distance information (D1, D2) for each distance segment among the plurality of distance segments (S1, S2) by a processing device; and A composite image of the scene is determined (S40) by the processing device using the calibrated camera model for each of the plurality of distance bins (S1, S2) and the image of the scene from each of the plurality of visible light cameras.

16. A multi-filter camera device comprising: a lens (41) configured to collect light (L) reflected from the scene; a multi-filter device (42) configured to determine filtered light (Lf) from the collected light (L), wherein the filtered light (Lf) comprises temporally interleaved light of at least two wavelength bands (B1, B2); an image sensor (43) configured to determine image data (D) from the filtered light (Lf), wherein the image data (D) comprises temporally interleaved image data corresponding to the at least two wavelength bands (B1, B2); A processing device (30) configured to process the image data (D), wherein the processing device (30) includes at least one video pipeline (32, 33), wherein the at least one video pipeline (32, 33) is configured to process the image data (D) corresponding to at least one of the at least two bands (B1, B2), wherein the processing device (30) includes a sequencer device (31), wherein the sequencer device (31) is configured to manage the image data (D) and the at least two video pipelines (32, 33) in time.

17. The multi-filter camera apparatus according to claim 16, in, The multi-filter device (42) includes a rotatable flywheel (42a) having at least two filters (F1, F2), wherein each of the at least two filters (F1, F2) is configured to filter light of one of the at least two wavelength bands (B1, B2).

18. The multi-filter camera device according to claim 17, in, The rotatable flywheel (42a) includes a plurality of filters configured to filter light in the same wavelength band of the at least two wavelength bands (B1, B2).

19. The multi-filter camera device according to claim 17 or 18, in, The at least two filters (F1, F2) are arranged as staggered segments around the flywheel (42a).

20. The multi-filter camera apparatus according to any one of claims 17 to 19, in, The number of filters (F1, F2) configured to filter light in one of the at least two wavelength bands (B1, B2) is different from the number of filters (F1, F2) configured to filter light in the other wavelength band of the at least two wavelength bands (B1, B2).

21. The multi-filter camera apparatus according to any one of claims 17 to 20, comprising: A motor (42b) is configured to rotate the flywheel (42a).

22. The multi-filter camera apparatus according to any one of claims 17 to 21, in, The sequencer device (31) is configured to distribute the image data (D) based on the position (P) of the flywheel (42a).

23. The multi-filter camera apparatus of claim 22, comprising: A position sensor (42c) is configured to determine the position (P) of the flywheel (42a).

24. The multi-filter camera apparatus according to claim 22, in, The sequencer device (31) is configured to determine the position (P) of the flywheel (42a) from motor feedback of the motor (42b).

25. The multi-filter camera apparatus according to any one of claims 16 to 24, in, The multi-filter device (42) includes a liquid crystal tunable filter, wherein the liquid crystal tunable filter is tunable to filter light in each of the at least two wavelength bands (B1, B2).

26. The multi-filter camera apparatus according to any one of claims 16 to 25, in, At least one of the wavelength bands (B1, B2) includes visible light.

27. The multi-filter camera apparatus according to any one of claims 16 to 26, in, At least one of the wavelength bands (B1, B2) includes near infrared light and / or infrared light.

28. The multi-filter camera apparatus according to any one of claims 16 to 27, in, At least one of the wavelength bands (B1, B2) comprises near-ultraviolet light and / or ultraviolet light.

29. The multi-filter camera apparatus according to any one of claims 16 to 28, in, The image sensor (43) includes a CMOS image sensor.

30. The multi-filter camera apparatus according to any one of claims 16 to 29, in, The multi-filter device (42) is arranged before the image sensor (43).

31. The multi-filter camera apparatus according to any one of claims 16 to 30, in, The processing device (30) is configured to determine a multi-band image based on the image data (D).

32. The multi-filter camera apparatus according to any one of claims 16 to 31, in, The light collected by the lens (41) is provided by a light source synchronized with a corresponding one of the at least two filters of the multi-filter device (42).

33. The multi-filter camera apparatus according to claim 32, in, The light source is a flash light source.

34. A multi-filter image processing method, comprising: collecting (Z10) light reflected from the scene through a lens (41); determining (Z20) filtered light (Lf) from the collected light by means of a multi-filter device (42), wherein the filtered light (Lf) comprises temporally interleaved light of at least two wavelength bands (B1, B2); determining (Z30) image data (D) from the filtered light (Lf) by an image sensor (43), wherein the image data (D) comprises temporally interleaved image data corresponding to the at least two wavelength bands (B1, B2); The image data (D) is processed (Z40) by a processing device (30), wherein the processing comprises temporally managing the image data (D) and at least one video pipeline (32, 33) of the processing device (30) by a sequencer device (31), wherein the at least one video pipeline (32, 33) processes the image data (D) corresponding to at least one of the at least two bands (B1, B2).

35. A multi-camera system comprising: A composite image camera apparatus according to any one of claims 1 to 14; as well as A multi-filter camera apparatus according to any one of claims 16 to 33.

Images