Automatic registration of preoperative volume image data using search images

Abstract

Various examples relate to techniques for recording measurement data with depth resolution in association with a procedure performed on a patient, such that a transformation specification can be determined based on the techniques that mediates between images captured by way of a robotic visualization system (e.g., a surgical microscope) and preoperative volumetric image data.

Description

Technical Field

[0001] Various examples involve techniques for capturing depth-resolution measurement data of a target region, in which the transformation specification between image data captured by a robotic visualization system and preoperative volumetric image data can be determined using the depth-resolution measurement data. Background Technology

[0002] In order to enable navigational applications in microsurgery, such as the precise spatial overlay of preoperative volumetric image data (e.g., computed tomography (CT) or magnetic resonance imaging (MRI)) onto the patient's anatomy during surgery, it is necessary to register the preoperative image data relative to the patient's position during surgery. That is, to determine the transformation specifications that describe how to transform the preoperative image data so that it can be precisely overlaid onto the patient in space.

[0003] Determining the transformation specification requires corresponding points in the preoperative volumetric image data and corresponding points from the patient in the operating room. The transformation specification maps images captured using a robotic visualization system in the operating room onto the preoperative volumetric image data, and / or vice versa.

[0004] In some examples, physical markers are attached to the patient for this purpose; these markers are visible in both preoperative volumetric imaging data and the image data from the visualization system in the operating room. The disadvantage of this approach is that attaching the physical markers to the patient is error-prone and laborious. Furthermore, it is necessary to attach the physical markers to the patient or to them stationary relative to the patient in order to record preoperative volumetric imaging data and before the actual surgery.

[0005] In other examples, a label-free approach was used. In this case, the surface geometry (anatomical topography) of the patient's target area was visible in both the measurement data and the preoperative image data, which was captured in the operating room and had depth resolution.

[0006] For example, a handheld portable pointer instrument can be used to palpate key points of the skull geometry (base of the ear, cheek, nose, etc.), and the corresponding measurement points (which are then defined in three-dimensional space) obtained in this way can be transmitted to a navigation system. This results in sparse sampling of the skull surface registered relative to surfaces obtained from preoperative datasets. This technique requires additional hardware and takes time to capture measurement data by operating the pointer instrument.

[0007] To capture surfaces more broadly, rapidly, and without additional hardware, methods have been proposed to determine the patient's head surface from stereoscopic image data obtained from robotic visualization systems (e.g., surgical microscopes). See, for example, US7561733B2 or DE102014210051A1. However, these methods assume that the robotic visualization system has been aligned with the patient to image stereoscopic data of the target area for registration. This is not normally available, as this information is only obtained after patient registration. Therefore, manual alignment may be required, which is in turn laborious and error-prone. Summary of the Invention

[0008] There is a need for improved techniques that enable the determination of transformation norms between images captured using robotic visualization systems and patients' preoperative volumetric image data. In particular, there is a need for techniques that eliminate or reduce the drawbacks mentioned above and the limitations of previously known techniques.

[0009] This objective is achieved through the features of the independent claims of the patent. The features of the dependent claims define the embodiments.

[0010] The techniques described in this paper enable rapid and efficient patient registration. This means that the transformation specification can be reliably and automatically determined. Target regions can be automatically discovered and identified, and depth-resolution measurements can be captured for these regions to determine the transformation specification in this manner.

[0011] The technique described in this paper thus enables the registration of preoperative volumetric image data associated with images captured using a robotic visualization system in a reference coordinate system.

[0012] A computer-implemented method includes actuating a robotic visualization system to record at least one search image in such a manner. The at least one search image at least partially images a patient. Furthermore, the method includes determining the arrangement of a target region within the at least one search image. Additionally, the method includes actuating the robotic visualization system based on the arrangement of the target region within the at least one search image to record measurement data with depth resolution. In this case, the measurement data indicates the morphology of the patient's anatomical structures. The measurement data allows for the determination of a transformation specification between the image recorded using the robotic visualization system and the patient's preoperative volumetric image data.

[0013] A target region can represent an area of ​​the patient for which preoperative volumetric imaging data is available. The preoperative volumetric imaging data can therefore also indicate the morphology of the patient's anatomical structures within the target region. For example, a target region can represent the intervention area for the planned surgery. For instance, a target region could include the patient's head. A target region could include a portion of the patient's head.

[0014] The arrangement of the target area can indicate the location and / or orientation and / or shape and / or size of the target area.

[0015] By using a robot visualization system to record at least one search image, it becomes possible to search for and identify the target area.

[0016] This avoids the need for manually locating the target area using a robotic visualization system, for example. The arrangement of the identified target area within the at least one search image enables automatic alignment of the robotic visualization system, allowing measurement data to be captured as observable data of the morphology of the patient's anatomical structures within the target area.

[0017] For example, preoperative volumetric imaging data can be CT or MR images. They can also be positron emission tomography (PET) images.

[0018] For example, it is conceivable to use a robotic visualization system to capture individual search images. Specifically, for instance, overview images can be captured using a corresponding camera in an overview image imaging unit that employs lenses that provide no magnification or relatively low magnification. In this case, the search image can image the target area and other nearby areas. For example, during a typical surgery on the patient's head, the patient's head can be imaged downwards to the chest area or downwards to the torso or hip area.

[0019] In other examples, one could conceive of recording multiple search images, each imaging a different region. A grid of search images could be captured.

[0020] For example, a robot visualization system can be actuated to record at least one sequence of search images. Each of the at least one sequence of search images may include multiple corresponding search images.

[0021] This means that the search image can be captured, for example, in one stage or in two stages, in one or two (or more) sequences.

[0022] For example, the search regions associated with the corresponding search image sequences can be adjusted or refined from one search image sequence to another. For example, search images from different search image sequences can have different resolutions. For example, search images from different search image sequences can have increased magnification.

[0023] For example, the search images in a search image sequence can scan the corresponding search area. For this purpose, at least one motor of the robot visualization system can be actuated during the recording of the at least one search image sequence to reposition the robot visualization system multiple times.

[0024] For example, the corresponding optical system and camera of the imaging unit used to capture the search image can be attached to the microscope head of the surgical microscope, wherein the microscope head is spatially locatable by a support having one or more movable axes (e.g., translational and / or rotational).

[0025] A relatively large search area can be covered by scanning. However, a relatively large magnification can be used simultaneously to ensure that the target area can be reliably identified.

[0026] For example, a first search region may be assigned to a first search image sequence in the at least one search image sequence, and a second search region may be assigned to a second search image sequence in the at least one search image sequence.

[0027] The second search area can be included within the first search area. The second search area can also be partially different from the first search area.

[0028] The robot visualization system can then be actuated sequentially to first record the first search image sequence and then the second search image sequence.

[0029] The method may include evaluating one or more search images from a first search image sequence. Based on this evaluation, the value of at least one imaging parameter of the robot visualization system used to record a second search image sequence can then be determined.

[0030] As a general rule, imaging parameters can represent the spatial arrangement of imaging cells and / or the parameterization of the imaging chain (zoom, focus, lighting, exposure parameters, etc.) used to capture the corresponding image of a robot visualization system. Therefore, imaging parameters can be parameters that affect the appearance of the image and / or the field of view of the image.

[0031] Therefore, this technique allows for situational adjustments to the value of at least one imaging parameter. Depending on the specific circumstances related to the visualization of the corresponding patient, the value of the at least one imaging parameter can be appropriately adjusted so that the target region can be reliably located using search images from a second search image sequence.

[0032] For example, evaluating one or more search images in a first search image sequence may include determining the arrangement of markers fixed relative to the patient in one or more search images of the search image sequence. The value of at least one imaging parameter can then be determined based on the arrangement of the markers fixed relative to the patient.

[0033] For example, the arrangement of the second search region relative to the first search region can be determined by setting the field of view of the search image of the second search region. For example, the relative arrangement of the markers relative to the target region can, in principle, be previously known. For instance, the markers can be attached to the patient, offset relative to the target region, for example, using clamps or other fixation devices. This offset can be known. The second search region can then be arranged relative to the first search region or to the markers found in the search image of the first search region.

[0034] A further exemplary implementation of evaluating one or more search images of a first search image sequence may include, for example, determining values ​​for one or more illumination parameters used to record a second search image sequence. In this regard, it is conceivable to, for example, evaluate the brightness histogram of one or more search images of the first search image sequence, and then set the exposure duration and / or the brightness of the illumination used to record one or more search images of the second search image sequence such that the brightness histogram for one or more search images of the second search image sequence corresponds to a predefined target. In this way, the contrast used to image features of a specific anatomical structure or skin tissue in one or more search images of, for example, the second search image sequence, can be ensured for the discovery of details in the target region.

[0035] In other examples, it is also conceivable to use one or more search images from a first search image sequence as a basis for determining whether one or more search images from the first search image sequence can be reused for a second search image sequence.

[0036] For example, if the second search region of the second search image sequence is determined in such a way that it overlaps with the first search region in the first search image sequence, then the search images already recorded for the first search image sequence can be reused for the second search image sequence within the corresponding overlapping area. This reduces the time required to capture search images.

[0037] Generally, it is conceivable that at least one search image is part of both a first search image sequence and a second search image sequence.

[0038] For example, one could consider evaluating a second search image sequence to identify the patient's skin tissue in one or more search images within that sequence. The arrangement of the target region could then be determined based on the identified skin tissue.

[0039] Typically, a surgical drape is used to cover most of the patient, exposing only the target area and the immediately surrounding area. Therefore, the target area can be identified by examining the skin tissue.

[0040] The method may include receiving tracking data from a tracking system. The tracking system may be juxtaposed relative to a robot visualization system. This means that the robot visualization system and the tracking system may operate in a common reference coordinate system, or the tracking system may be used to determine the arrangement of the robot visualization system in the reference coordinate system (and therefore also the pose or field of view of the measurement data or images captured by the robot visualization system). The tracking system may also be configured to determine the arrangement (i.e., position and / or orientation) of other markers in the reference coordinate system. The reference coordinate system may be determined relative to the operating room.

[0041] Tracking data can describe, for example, the arrangement of markers fixed relative to the patient relative to a robotic visualization system. The markers can be implemented, for example, using passive machine-readable symbols or active machine-readable light sources.

[0042] Accordingly, the method may also include determining the value of at least one imaging parameter of the robot visualization system used to record the at least one sequence of search images based on the tracking data.

[0043] For example, the arrangement of corresponding search areas covered by at least one sequence of search images can be determined based on tracking data. For instance, if markers are identified using a tracking system, the relative positioning of the markers with respect to the target area may be previously known.

[0044] Typically, the value of at least one imaging parameter of the robot visualization system used to record the at least one search image sequence can therefore be determined based on prior knowledge. For example, the brightness of the illumination can be set as an exemplary imaging parameter. The exposure time can be set as an exemplary imaging parameter. The arrangement of the corresponding search area can be determined. The reuse of search images between different search image sequences can be determined.

[0045] Such prior knowledge can involve, for example, a (rough) arrangement of the patient relative to the robotic visualization system. Alternatively or additionally, such prior knowledge can also involve the arrangement of the target area relative to markers fixed relative to the patient. Specifically, the patient can be roughly pre-positioned relative to the robotic visualization system. For example, the patient can be positioned on a hospital bed, the arrangement of which relative to the robotic visualization system is known. In this way, in the first step, the robotic visualization system can be roughly aligned with the patient's (assumed) position, and search images of at least one search image sequence can subsequently be captured based on such prior knowledge. The same approach can be applied to markers fixed relative to the patient.

[0046] The method may further include evaluating already captured search images of the at least one search image sequence during recording. Recording of the at least one search image sequence can then be selectively terminated depending on the evaluation; that is, capturing other search images may be omitted. Alternatively or additionally, it is also conceivable to adjust the value of at least one imaging parameter of the robot visualization system used for recording the at least one search image sequence depending on the evaluation.

[0047] In other words, if multiple search images are captured, it can be continuously checked whether it is necessary to capture other search images (or, for example, whether the target area has already been identified and therefore no further search images need to be captured). Furthermore, it can be checked whether the value of at least one imaging parameter is appropriately set, or whether adjustments should be made, such as adjustments to brightness or exposure time.

[0048] Generally, the arrangement of a target region can be determined based on the structural identification of one or more predefined structures in the at least one search image. In this regard, for example, the structure of skin tissue can be identified in the at least one search image. Alternatively or additionally, the structure of features of specific anatomical structures (e.g., nose, ears, mouth, eyes, etc.) can be identified in the at least one search image. Suitable algorithms can be used. However, specific implementations of such algorithms for determining the arrangement of target regions are not essential to implementing the techniques described herein, and previously known implementations can be used.

[0049] To record at least one search image, for example, an overview image imaging unit of a robot visualization system and / or a microscope imaging unit of a robot visualization system can be actuated. In other words, this means that the at least one search image may, for example, include an overview image with relatively low magnification and / or may also include a microscope image with relatively high magnification.

[0050] Measurement data can be captured, for example, using measurement models selected from the following group: stereo imaging; time-of-flight measurement; structured illumination; and defocus depth estimation.

[0051] In stereoscopic imaging, information about the shape is obtained by using two cameras that image the same area in different poses.

[0052] In the case of time-of-flight measurement, information about the morphology can be obtained by monitoring the correspondence between short light pulses and time of flight.

[0053] In the case of structured lighting, a predefined pattern (e.g., a line pattern) is projected onto the surface to be measured. Information about the morphology can then be obtained using the distortion of the predefined pattern.

[0054] In the case of out-of-focus depth estimation, information about the morphology can be obtained by using autofocus locally.

[0055] As a general rule, it is conceivable that measurement data includes at least one search image. This means that it is not absolutely necessary to capture the measurement data to be captured completely separately from the search image.

[0056] This method can also include filtering the measurement data. For example, in this case, data elements that are not imaged of the patient's skin tissue can be removed from the measurement data. For example, areas imaged by surgical drapes, markers, or surgical instruments can be removed from the measurement data. This avoids misregistration of the measurement data relative to preoperative volumetric image data.

[0057] Before recording the at least one search image, the motors of the robotic visualization system can be actuated to arrange the robotic visualization system in a predefined reference arrangement. For example, the predefined reference arrangement can be defined relative to an operating table. The predefined reference arrangement can be defined, for example, in such a way that the area on the operating table where the patient's head will be positioned is within the field of view of the imaging unit of the robotic visualization system, which is used to first record the at least one search image.

[0058] It is conceivable that the method would also include setting at least one imaging parameter of the robot visualization system to a predefined value before recording the at least one search image. This could include, for example, setting the target magnification of the scaling factor of the corresponding imaging unit of the robot visualization system. A specific illumination brightness could be selected. It is conceivable that the exposure time could be set to a specific value.

[0059] The methods described herein can be executed automatically based on predefined control scripts. For example, a control script can be initiated with a single user command (e.g., "one-click"). In this way, the simple and rapid acquisition of measurement data can be achieved, which subsequently allows for the determination of transformation specifications; manual repositioning of the robot visualization system may be unnecessary.

[0060] For example, such a control script may include an analysis module for evaluating image data or measurement data. Furthermore, the control script may include at least one localization module for locating the robotic visualization system. This localization can be used, for example, by means of scanning to capture a sequence of search images covering a search area larger than the field of view of a single search image. The analysis module may evaluate the image data, for example, to locate skin tissue. The analysis module may evaluate the image data, for example, to identify markers fixed relative to the patient.

[0061] The analysis module may include, for example, one or more machine learning algorithms. For instance, an artificial neural network that identifies predefined structures or objects in an image may be used. For this purpose, a convolutional neural network, for example, may be used. Training may be made possible, for example, by manually annotating the corresponding training image data or training measurement data.

[0062] On the other hand, it's conceivable that the localization module doesn't include machine learning algorithms. Manually parameterized algorithms can be used. This can be particularly worthwhile because, in this way, for each input parameter, the new position of the localization module in the effective phase can be verified, thus preventing uncontrolled or dangerous movement of the robot's visualization system.

[0063] The method may also include generating a topography dataset based on the measurement data. This means that the measurement data can be evaluated, for example, using the analysis modules described above.

[0064] This method may also include performing registration between the topographic dataset and the preoperative volumetric image data. This means that corresponding points can be found in the topographic dataset and the preoperative volumetric image data. The transformation specification can then be determined based on the relative arrangement of these corresponding points with respect to each other. Typically, the transformation specification can be determined based on the registration.

[0065] The data processing unit is configured to actuate a robotic visualization system to record at least one search image in such a manner. This at least one search image at least partially images the patient. Furthermore, the data processing unit is configured to determine the arrangement of a target region within the at least one search image, and based on this arrangement of the target region in the at least one search image, actuate the robotic visualization system to record measurement data with depth resolution. These measurement data indicate the morphology of the patient's anatomical structures. This allows for the determination of a transformation specification between the images recorded using the robotic visualization system and the patient's preoperative volumetric image data.

[0066] A system includes a data processing unit and a robot visualization system.

[0067] A computer program, computer program product, or computer-readable storage medium includes program code. The program code can be loaded and executed by a processor. This causes the processor to perform a method. The method includes actuating a robotic visualization system to record at least one search image in such a way that the at least one search image at least partially images a patient. Furthermore, the method includes determining the arrangement of a target region within the at least one search image. Additionally, the method includes actuating the robotic visualization system based on the arrangement of the target region within the at least one search image to record measurement data with depth resolution. In this case, the measurement data indicates the morphology of the patient's anatomical structures. The measurement data makes it possible to determine a transformation specification between the image recorded using the robotic visualization system and the patient's preoperative volumetric image data.

[0068] Without departing from the scope of protection of this invention, the features set forth above and described below can be used not only in the explicitly stated corresponding combinations, but also in other combinations or individually. Attached Figure Description

[0069] Figure 1 An optical visualization system is illustrated schematically based on various examples.

[0070] Figure 2 The data processing unit is illustrated schematically based on various examples.

[0071] Figure 3 This is a flowchart of an exemplary method.

[0072] Figure 4 The images and target regions are shown separately based on various examples.

[0073] Figure 5 The search image sequences and target regions are shown based on various examples. Detailed Implementation

[0074] The features, characteristics, and advantages of the invention described above, as well as the ways in which it is implemented, will become clearer and more apparent in conjunction with the following description of exemplary embodiments, which are explained in more detail with reference to the accompanying drawings.

[0075] The invention will now be explained in more detail with reference to the accompanying drawings, based on preferred embodiments. In the drawings, the same reference numerals denote the same or similar elements. The drawings are schematic representations of different embodiments of the invention. Elements shown in the drawings are not necessarily shown to scale. Instead, the different elements shown in the drawings are presented in a manner that makes their function and general purpose readily understandable to those skilled in the art. Connections and linkages between functional units and elements shown in the drawings may also be implemented as indirect connections or linkages. Connections or linkages may be implemented in a wired or wireless manner. Functional units may be implemented as hardware, software, or a combination of hardware and software.

[0076] The following describes the techniques that enable the determination of transformation specifications to map or transform image data captured by a robotic visualization system (hereinafter referred to as a surgical microscope) to preoperative volumetric image data; the inverse mapping specification can also be covered by the transformation specification.

[0077] For this purpose, the target region is automatically identified or its arrangement is automatically determined based on one or more search images. Measurements with depth resolution can then be determined for the target region, and these measurements can be registered relative to preoperative volumetric image data to derive a transformation specification from that registration.

[0078] This transformation specification then enables various applications. For example, it can provide surgeon-aiding functions based on preoperative volumetric image data and the transformation specification. For instance, preoperative volumetric image data, or portions thereof, can be spatially realistically inserted into the field of view of a surgical microscope or into image data captured using a surgical microscope. Specific areas identified in the preoperative volumetric image data can be highlighted in image data captured using a surgical microscope. Navigational assistance functions guiding surgical interventions can be made possible.

[0079] Figure 1 A surgical microscope 801 for use in surgery is schematically shown. In the example shown, the surgical microscope 801 includes an eyepiece 803. The surgeon can view a magnified image of an object located in the field of view 804 of the surgical microscope 801 through the eyepiece 803. In the example shown, this is a patient 805 lying on a hospital bed.

[0080] As an alternative to or supplement to the optical eyepiece, a camera 809 that transmits images to a screen (digital surgical microscope) can also be provided.

[0081] Generally, a surgical microscope 801 may include one or more imaging units configured to record digital images. Examples of imaging units may be, for example, a microscope imaging unit and an overview image imaging unit. The microscope imaging unit may include one, two, or more cameras; for example, stereoscopic imaging can be achieved using multiple cameras.

[0082] The operating device 808 is also configured as a human-machine interface; for example, the operating device can be implemented as a handle or foot switch. Figure 1 In the embodiment shown, it is a handle. The operating device 808 allows movement of the eyepiece 803, which is secured to the crossbeam 850. A motor may be provided to automatically perform the movement based on control data according to the corresponding settings of the surgical microscope. The motor may also assist the movement initiated by the handle 808.

[0083] Furthermore, the control device 880 is configured for the surgical microscope 801 and controls the operation of the surgical microscope 801 as well as the display of images and additional information and data in the eyepiece 803. The control device 880 can perform interactions with the surgeon.

[0084] The surgical microscope 801 also includes one or more other sensors 860, such as an environmental camera, a time-of-flight (TOF) camera, an image recording device for recording images captured under structured illumination, etc.; for example, such sensors 860 can be used to capture measurement data with depth resolution. This means that the depth resolution can describe the distance between the corresponding sensor 860 and the patient 805.

[0085] Further embodiments of this sensor 860 involve an internal tracking system. For example, such an internal tracking system can be used to identify markers fixed relative to the patient and embodied in a machine-readable manner. These markers are sometimes also referred to as targets. Specifically, the relative arrangement of the markers with respect to the surgical microscope 801 can be determined.

[0086] Figure 1 An external tracking system 890 is also shown. This external tracking system is juxtaposed with the surgical microscope 801. Tracking data 890 can be transmitted from the tracking system 890 to a control device 880. This tracking data indicates, for example, the arrangement of the camera 809 or eyepiece 803, or generally indicates the field of view 804 of the imaging unit in space, such as relative to a marker fixed relative to the patient 805, or a spatially fixed reference coordinate system.

[0087] Figure 2 Aspects related to the data processing unit 910 are illustrated. For example, the data processing unit 910 can be implemented using a computer. The data processing unit 910 can implement, for example, a control device 880 for a surgical microscope 801 (see reference). Figure 1Alternatively, it can be embodied separately from the surgical microscope 801 (and then communicate with, for example, the control device 880).

[0088] The data processing unit 910 includes a computing unit 911, a memory 912, and a communication interface 913. The computing unit 911 can, for example, load program code from the memory 912 and execute the code. The computing unit 911 can communicate with other devices, nodes, or apparatuses via the communication interface 913. For example, tracking data can be received from a tracking system. Preoperative volumetric image data can be received, for example, from an image archiving system or some other database. User input can be received from the human-machine interface.

[0089] For example, if computing unit 911 executes program code loaded from memory 912, this can enable computing unit 911 to implement techniques as described herein, such as: recording search images or measurement data; evaluating search images or measurement data; determining the arrangement of target regions in the search image; actuating surgical microscope 801 to capture images, search images or measurement data and / or bring the specific location of field of view 804; providing control data to enable functions that assist the surgeon; for example, determining the transformation specification between image data captured by surgical microscope 801 and preoperative volumetric image data based on the correspondence registration between measurement data with depth resolution and preoperative volumetric image data. The following is in conjunction with... Figure 3 The method descriptions in the text can be provided and implemented by the computing unit 911 based on the corresponding program code, and include various functions.

[0090] Figure 3 This is a flowchart of an exemplary method. For example, Figure 3 The method in the middle can be made by a data processing unit (e.g., Figure 2 The data processing unit 910 executes the data. Specifically, when the computing unit loads and executes the program code, Figure 3 The method can be executed by a computing unit (such as a processor).

[0091] Figure 3 The method described can be used to determine the transformation norm between images recorded using a surgical microscope and preoperative volumetric image data (such as CT or MRI images). In other words, Figure 3 The method described herein is therefore used to calibrate the patient's positioning relative to the surgical microscope and relative to preoperative volumetric image data. Specifically, Figure 3 The method supports the automated capture of depth-resolution measurement data via a surgical microscope, which can then be registered relative to preoperative volumetric image data. The transformation specification can then be determined based on this registration; however, the transformation specification can also be determined in downstream methods, for example, through different data processing systems, such as the control equipment of a tracking system.

[0092] For example, one could think of execution based on automated control scripts. Figure 3 This means that various method steps can be processed automatically and sequentially by a computer program. The advantage of this is that users can initiate the corresponding control script with a single human-computer interaction, and no manual interaction is required during the processing of the corresponding method steps.

[0093] exist Figure 3 Dashed lines are used to indicate optional boxes.

[0094] First, within frame 3005, the motor of the surgical microscope can be actuated to align the surgical microscope with a predefined reference arrangement. For example, when a patient is positioned on the corresponding bed, this reference arrangement aligns the field of view of the surgical microscope with the patient's head area.

[0095] Subsequently, in box 3010, optionally, one or more imaging parameters can be initialized. This means that at least one imaging parameter of the surgical microscope can be set to a predefined value before recording the search image. For example, the brightness of the illumination can be initialized, or, conceivably, the magnification factor of the scaling can be set. A predefined focus can be selected. A specific field of view can be set.

[0096] Sometimes boxes 3010 and 3005 can be implemented together. For example, if the imaging parameters of the corresponding imaging unit of the positioning surgical microscope are set in box 3010.

[0097] Subsequently, frame 3011 involves recording at least one search image, at least partially imaged of the patient. For this purpose, a surgical microscope or, more precisely, a corresponding imaging unit or corresponding control device can be actuated. A command can be sent to capture the corresponding at least one search image.

[0098] For example, an overview image imaging unit and / or a microscope imaging unit of a surgical microscope can be actuated to record at least one search image. Therefore, this means that, for example, one or more overview images with low magnification and / or one or more microscope images with high magnification can be used to implement at least one search image.

[0099] For block 3011, there are various implementation variations. Before discussing these implementation variations in detail below, the following method will be explained in more detail first.

[0100] Box 3030 relates to determining the arrangement of a target region within the at least one search image. This means determining the location and / or orientation and / or extent of the target region within the at least one search image. An analysis module may be used. For this purpose, a machine learning algorithm may be used, for example. Such a machine learning algorithm may be trained based on manually annotated search images and target regions. For example, the machine learning algorithm may be implemented using a convolutional neural network.

[0101] The target region can represent, for example, the intervention area of ​​the planned surgery. The target region can also represent a region of the patient's body, for which preoperative volumetric imaging data was captured or imaged using preoperative volumetric imaging data.

[0102] For example, the arrangement of a target region can be determined based on structural recognition of one or more predefined structures in the at least one search image. For instance, features of skin tissue or anatomical structures can be identified. However, machine-readable markers can also be identified.

[0103] In box 3035, depth-resolution measurement data can then be captured based on this arrangement of the target area. For this purpose, a surgical microscope, such as a corresponding sensor, a corresponding imaging unit, or an assigned control device, can be actuated. The measurement data thus indicate the morphology of the patient's anatomical structures within the target area.

[0104] For example, measurement data can be captured using measurement modes selected from the group consisting of: stereo imaging; time-of-flight measurement; structured illumination; and depth-of-focus estimation.

[0105] In some examples, the measurement data may also include at least one of the at least one search image. Therefore, it is conceivable that a specific image is not only used as a search image in box 3011, but also considered in the context of the measurement data in box 3035, i.e., achieving a dual function.

[0106] In box 3036, measurement data can optionally be filtered. For example, noise suppression can be performed. Background can be removed. Data elements that are not imaged from the patient's skin tissue can be removed.

[0107] In this way, a transformation specification can be determined between images recorded using a surgical microscope and the patient's preoperative volumetric image data. This transformation specification can be determined in optional box 3040. Alternatively, the measurement data can be stored for later determination, or it can be transmitted to different data processing units to determine the transformation specification.

[0108] If the transformation specification is determined, the following steps can be taken: First, a topographic dataset can be generated based on measurement data. For example, this means determining the height contour of the patient's head or the typical patient's skin surface in the target region. Then, registration of this topographic dataset with preoperative volumetric image data can be performed. Specifically, the preoperative volumetric image data can also specifically image the topography of the patient's head's skin surface. Based on this registration, the transformation specification can then be determined, considering, for example, the degrees of freedom of translation and / or rotation. Distortion can also be considered.

[0109] Optionally, an application based on this transformation specification can be implemented in box 3045. For example, computer-assisted surgery can be performed against the backdrop of enhanced images captured using a surgical microscope and containing information determined based on preoperative volumetric image data. Navigation assistance functions can also be implemented.

[0110] The following describes details related to box 3011, specifically details related to recording the at least one search image. In a simple variant, a single search image can be recorded and the arrangement of the target region within that single search image can be determined. Figure 4 This scenario is illustrated in the text. Figure 4 Search image 111 is shown. Search image 111 images the target region 131. For example, search image 111 can be generated by a surgical microscope 801 (reference). Figure 1 The overview image is captured by the imaging unit.

[0111] In another variant, one or more sequences of search images can also be captured. For example, Figure 5 The scene shown depicts a total of 25 search images 111-113 (only the first three search images are labeled for clarity). For example, each of these search images in the search image sequence can have a particularly high resolution. High magnification can also be made possible in this way. This corresponds to the search area being scanned. Scanning can be achieved by actuating the motor of the surgical microscope, allowing the microscope to be repositioned multiple times, for example, between the various search images in the captured search image sequence.

[0112] In some examples, multiple search image sequences can also be captured sequentially. Figure 3This scenario is described in conjunction with box 3011. In this case, firstly, a first search image sequence assigning a first search area is recorded in box 3015; and then, a second search image sequence assigning a second search area is recorded in box 3025. In box 3020 between the two, one or more search images from the first search image sequence in box 3015 can be evaluated, and based on this evaluation, the value of at least one imaging parameter of the surgical microscope used for recording the second search image sequence in subsequent box 3025 is determined.

[0113] Such an assessment in box 3020 may include, for example, determining the arrangement of the patient relative to one or more search images in a first search image sequence from box 3015 with the markers fixed in place.

[0114] This allows for determining, for example, the arrangement of a second search region, wherein a second search image sequence is captured in box 3025 relative to a first search image sequence from box 3015. It also allows for determining whether search images from the first search image sequence are reused for the second search image sequence. Illumination parameters, for example, for recording the second search image sequence can be set. A magnification factor can be set for the search images of the second search image sequence.

[0115] If multiple search image sequences are used—for example, in Figure 3 The following explanation is based on boxes 3015 and 3025—in principle, at least one search image may be part of a sequence of multiple search images.

[0116] The following section, TAB.1, outlines the implementation methods. Figure 3 A description of the workflow of the methods described.

[0117]

[0118]

[0119]

[0120] Tab. 1: An exemplary implementation of a workflow that enables the determination of transformation specifications between images captured using a surgical microscope and preoperative volumetric image data. These steps do not necessarily need to be performed in such a strict order, but can also be combined or run in parallel to reuse the recorded data where appropriate.

[0121] In summary, the technologies described above enable the automatic identification of the areas to be scanned for patient registration, i.e., the target regions. The automated alignment of the robotic visualization system has been disclosed, enabling the capture of depth-resolution measurement data and images of the target regions.

[0122] It goes without saying that the features of the embodiments and aspects of the present invention described above can be combined with each other. In particular, without departing from the scope of the present invention, these features can be used not only in the described combinations, but also in other combinations or individually.

Claims

1. A computer-implemented method, comprising: controlling a robotic visualization system (801) to capture at least one search image (111-113) at least partially depicting a patient (805), determining a location of a target region (131) in the at least one search image (111-113), and controlling the robotic visualization system (801) to capture measurement data with depth resolution based on the location of the target region (131) in the at least one search image (111-113), wherein the measurement data is indicative of a topography of an anatomy of the patient (805) so as to enable determination of a transformation specification between images obtained by means of the robotic visualization system (801) and preoperative volumetric image data of the patient (805), controlling the robotic visualization system to capture at least one search image sequence, each of the at least one search image sequence comprising a plurality of corresponding search images (111-113), the method further comprising: controlling at least one motor of the robotic visualization system (801) during the capturing of the at least one search image sequence so as to repeatedly position the robotic visualization system (801) such that the plurality of search images (111-113) of a respective search image sequence scan a search region associated with the respective search image sequence, wherein a first search region is associated with a first search image sequence of the at least one search image sequence, wherein a second search region is associated with a second search image sequence of the at least one search image sequence, wherein the robotic visualization system (801) is first controlled to capture the first search image sequence and subsequently controlled to capture the second search image sequence, the method further comprising: evaluating one or more search images (111-113) of the first search image sequence and determining a value of at least one imaging parameter of the robotic visualization system (801) based on evaluating the one or more search images (111-113) of the first search image sequence, the at least one imaging parameter being used for capturing the second search image sequence.

2. The computer-implemented method of claim 1, wherein evaluating the one or more search images of the first search image sequence comprises determining a disposition of a marker fixedly mounted in a stationary position relative to the patient (805) in the one or more search images (111-113) of the first search image sequence, and wherein the value of the at least one imaging parameter is determined based on a position of the marker fixed relative to the patient.

3. The computer-implemented method of claim 1 or 2, wherein the at least one imaging parameter is selected from a group comprising: a disposition of the second search region relative to the first search region; a reuse of search images (111-113) of the first search image sequence for the second search image sequence; an illumination parameter used for capturing the second search image sequence.

4. The computer-implemented method of claim 1 or 2, wherein At least one search image (111-113) is part of both the first search image sequence and the second search image sequence.

5. The computer-implemented method of claim 1 or 2, the method further comprising: - evaluating one or more search images (111-113) of the second search image sequence to detect a skin tissue of the patient (805) in the one or more search images (111-113) of the second search image sequence, wherein the arrangement of the target region (131) is determined based on detecting the skin tissue.

6. The computer-implemented method of claim 1 or 2, the method further comprising: - receiving tracking data from a tracking system collocated with respect to the robotic visualization system (801), the tracking data describing an arrangement of a marker fixedly mounted with respect to the patient (805) with respect to the robotic visualization system (801), and - determining a value of at least one imaging parameter of the robotic visualization system (801) based on the tracking data, the at least one imaging parameter being used for capturing the at least one search image sequence.

7. The computer-implemented method of claim 1 or 2, wherein, The method further comprises: - determining a value of at least one imaging parameter of the robotic visualization system (801) based on prior knowledge about at least one of an arrangement of the patient (805) with respect to the robotic visualization system and an arrangement of the target region (131) with respect to a marker fixedly mounted to the patient (805), the at least one imaging parameter being used for capturing the at least one search image sequence.

8. The computer-implemented method of claim 1 or 2, wherein, The method further comprises: - evaluating, during capturing of the at least one search image sequence, a search image (111-113) of the at least one search image sequence that has been captured, and - depending on the evaluation, terminating the capturing of the at least one search image sequence and / or adjusting the value of the at least one imaging parameter of the robotic visualization system (801), the at least one imaging parameter being used for capturing the at least one search image sequence.

9. The computer-implemented method of claim 1 or 2, wherein, determining the arrangement of the target region (131) based on feature recognition of one or more predetermined features in the at least one search image (111-113), wherein the one or more predetermined features are selected from a group comprising: a skin tissue; an anatomical structure feature.

10. The computer-implemented method of claim 1 or 2, wherein controlling at least one of an overview image imaging unit of the robotic visualization system (801) or a microscope imaging unit of the robotic visualization system (801) for capturing the at least one search image (111-113).

11. The computer-implemented method of claim 1 or 2, wherein the measurement data is obtained with a measurement modality selected from a group comprising: stereoscopic imaging; time-of-flight measurement; structured illumination; and out-of-focus depth estimation.

12. The computer-implemented method of claim 1 or 2, wherein the measurement data comprises at least one of the at least one search image.

13. The computer-implemented method of claim 1 or 2, the method further comprising: - filtering (3036) the measurement data so as to remove data elements not representative of the skin tissue of the patient (805).

14. The computer-implemented method of claim 1 or 2, further comprising: - controlling (3005) a motor of the robotic visualization system (801) so as to arrange the robotic visualization system (801) into a predefined reference arrangement prior to capturing the at least one search image (111-113).

15. The computer-implemented method of claim 1 or 2, further comprising: - setting (3010) at least one imaging parameter of the robotic visualization system (801) to a predetermined value prior to capturing the at least one search image (111-113).

16. The computer-implemented method of claim 1 or 2, wherein the method is performed in an automated manner based on a predefined control script.

17. The computer-implemented method of claim 16, wherein the control script comprises an analysis module for evaluating image data or measurement data, wherein the control script comprises a positioning module for positioning the robotic visualization system (801), wherein the analysis module comprises one or more machine learning algorithms, wherein the positioning module does not comprise a machine learning algorithm.

18. The computer-implemented method of claim 1 or 2, further comprising: - generating a topography data set based on the measurement data, - performing a registration of the topography data set with the preoperative volume image data, and - determining the transformation specification based on the registration.

19. A computer program product configured to perform the following steps: - controlling a robotic visualization system (801) to capture at least one search image (111-113) at least partially describing a patient (805), - determining an arrangement of a target region (131) in the at least one search image, and - based on the position of the target region (131) in the at least one search image (111-113), controlling the robotic visualization system (801) to capture measurement data with depth resolution, wherein, the measurement data being indicative of a topography of an anatomical structure of the patient (805) so as to enable determining a transformation specification between images obtained by means of the robotic visualization system (801) and preoperative volume image data of the patient (805), wherein the robotic visualization system (801) is controlled to capture a sequence of at least one search image, each of the sequence of at least one search image comprising a plurality of corresponding search images (111-113), wherein the computer program product is further arranged to perform the following steps: - controlling at least one motor of the robotic visualization system (801) so as to repeatedly position the robotic visualization system (801) during capturing of the sequence of at least one search image such that the plurality of search images (111-113) of a respective sequence of search images scan a search region associated with the respective sequence of search images, wherein a first search region is associated with a first sequence of search images of the sequence of at least one search image, wherein a second search region is associated with a second sequence of search images of the sequence of at least one search image, wherein first the robotic visualization system (801) is controlled to capture the first search image sequence and subsequently the robotic visualization system is controlled to capture the second search image sequence, wherein the computer program product is further arranged to perform the following steps: evaluating one or more search images (111-113) of the first search image sequence and determining a value of at least one imaging parameter of the robotic visualization system (801) used for capturing the second search image sequence based on evaluating the one or more search images (111-113) of the first search image sequence.

20. The computer program product of claim 19, wherein, The computer program product is configured to perform the method of any one of claims 2 to 18.

21. A system comprising: - the computer program product of claim 19 or 20, and - a robotic visualization system (801).

Images