Automatic registration of preoperative volumetric image data using search images

Abstract

Various examples relate to techniques that make it possible to record measurement data with depth resolution regarding a surgery performed on a patient and, based on these data, define conversion rules that perform a conversion between images recorded by a robotic visualization system, e.g. a surgical microscope, and pre-operative volumetric image data.

Description

[Technical field]

[0001] Various examples relate to techniques for capturing depth-resolved measurement data of a target region, which can identify conversion specifications between image data captured by a robotic visualization system and pre-operative volumetric image data. [Background technology]

[0002] In navigation applications in microsurgery, for example, in order to be able to accurately overlay pre-operative volumetric image data (e.g., computed tomography images, i.e., CT, or magnetic resonance images, i.e., MRI) onto the patient's anatomical space during surgery, it is necessary to register the pre-operative image data with respect to the patient's position during surgery, i.e., a transformation specification is specified that describes how the pre-operative image data must be transformed so that it can be accurately spatially overlaid with the patient.

[0003] Identifying the transformation specification requires matching points in the pre-operative volumetric image data with points from the patient in the operating room. The transformation specification maps images captured by the robotic visualization system in the operating room to the pre-operative volumetric image data and / or vice versa.

[0004] In some instances, for this purpose, physical markers are attached to the patient, which are visible in both the preoperative volumetric image data and the image data of the visualization system in the operating room. A drawback of this approach is that the physical markers must be attached to the patient, which is error-prone and labor-intensive. Moreover, the physical markers need to be attached to the patient or in a fixed position relative to the patient both for the recording of the preoperative volumetric image data and before the actual operation.

[0005] In another example, a marker-less approach is taken, where the surface geometry (anatomical topography) of the patient's target area is visible in both the depth-resolved measurement data captured in the operating room and in the pre-operative image data.

[0006] For example, a handheld pointer instrument can be used to touch key points of the skull geometry (base of the ears, cheeks, nose, etc.) and the corresponding measurement points of the measurement data thus obtained (which are then defined in three-dimensional space) can be transferred to the navigation system. This allows a sparse sampling of the skull surface registered with respect to the surface obtained from the pre-operative data set. Such a technique requires additional hardware and a specific time course for capturing the measurement data by manipulating the pointer instrument.

[0007] In order to capture surfaces more extensively, more quickly and without the need for additional hardware, methods have been proposed to use stereoscopic image data of a robotic visualization system (e.g. a surgical microscope) to identify the surface of the patient's head therefrom. See, for example, US 7561733 B2 or DE 102014210051 A1. However, these methods assume that the robotic visualization system has already been aligned with the patient, whereby the stereoscopic image data images target areas that allow the alignment. However, this is not the case in the normal case, since this information is only available after the patient has been aligned. Therefore, manual alignment is required, which in turn is laborious and prone to errors. Summary of the Invention [Problem to be solved by the invention]

[0008] There is a need for improved techniques that allow for the specification of a transformation between images captured by a robotic visualization system and a patient's preoperative volumetric image data, and in particular, a need exists for techniques that eliminate or reduce the above-mentioned shortcomings and limitations of previously known techniques. [Means for solving the problem]

[0009] This object is achieved by means of the features of the independent patent claims. The features of the dependent patent claims define embodiments.

[0010] Rapid and efficient alignment of the patient is made possible by the techniques described herein, which means that the transformation specification can be reliably and automatically identified. In this way, the target region from which measurement data with depth resolution is acquired to identify the transformation specification can be automatically found and recognized.

[0011] In this manner, the techniques described herein allow for the registration of pre-operative volumetric image data within a reference coordinate system associated with images captured by a robotic visualization system.

[0012] The computer-implemented method includes operating the robotic visualization system to record at least one search image in this manner. The at least one search image at least partially images the patient. Additionally, the method also includes identifying a location of the target area in the at least one search image. Additionally, the method includes operating the robotic visualization system to record measurement data having depth resolution based on the location of the target area in the at least one search image. In this case, the measurement data is indicative of a topography of the patient's anatomy. The measurement data allows for identifying a transformation specification between the images recorded by the robotic visualization system and the patient's pre-operative image data.

[0013] The target region may refer to a region of the patient for which preoperative volumetric image data is available. The preoperative volumetric image data may therefore also represent the topography of the area of ​​the patient's anatomy, the target region. For example, the target region may represent the planned surgical intervention area. The target region may include, for example, the patient's head. The target region may include a portion of the patient's head.

[0014] The location of a target area can refer to the position and / or orientation and / or shape and / or size of the target area.

[0015] By recording at least one exploration image by the robotic visualization system it is therefore possible to explore and recognise the target area.

[0016] For example, manual locating of the target region, such as manual positioning of a robotic visualization system, can be avoided. Automatic alignment of the robotic visualization system, whereby measurement data can be captured as observables of the topography of the target region of the patient's anatomy, can be performed based on the identified location of the target region in at least one search image.

[0017] The preoperative volumetric image data can be, for example, CT image data or MR image data, but can also be Positron Emission Tomography image data.

[0018] For example, it is conceivable that the individual search images are captured by the robotic visualization system. In particular, for example, the overview image can be captured by a corresponding camera of the overview image capture unit using a lens without magnification or with a relatively low magnification. In such a case, the search image can image the target area and the surrounding area as well. For example, in a typical operation in the area of ​​the patient's head, the patient's head can be imaged up to the chest or up to the torso or waist.

[0019] In another example, multiple search images may be recorded, each image imaging a different area. A grid of search images may be captured.

[0020] For example, the robotic visualization system may be operated to record at least one search image sequence, each of which may include a plurality of corresponding search images.

[0021] This means that the search images can be taken as one or two (or more) sequences, for example in one or two stages.

[0022] For example, the search images associated with each respective search image sequence can be adjusted and specifically refined for each search image sequence, for example, the search images of the different search image sequences can have different resolutions, for example, the magnification of the search images of the different search image sequences can be increased.

[0023] For example, a corresponding search area can be scanned with multiple search images of a search image sequence, for which purpose the position of the robotic visualization system can be changed multiple times by actuating at least one motor of the robotic visualization system during recording of at least one search image sequence.

[0024] For example, the corresponding optical system and camera of the image capturing unit used to capture the exploration image can be mounted on a microscope head of a surgical microscope, and the microscope head can be positioned in space by a stand having, for example, one or more movable axes for translation and / or rotation.

[0025] A relatively large search area can be covered by the scan, while at the same time using a relatively high magnification, so that the target area can be recognized with a high degree of reliability.

[0026] For example, a first search area may be assigned to a first search image of the at least one search image sequence, and a second search area may be assigned to a second search image of the at least one search image sequence.

[0027] The second search area may be included in the first search area, or may be partially different from the first search area.

[0028] The robot visualization system may therefore first record a first scout image sequence and then a second scout image sequence.

[0029] The method can include evaluating one or more scout images of the first scout image sequence, such that based on the evaluation, values ​​of at least one image capture parameter of the robotic visualization system can be identified that are used to record a second scout image sequence.

[0030] In general, image capture parameters may refer to the spatial arrangement of the image capture units used to capture a corresponding image of the robotic visualization system and / or parameter settings of the image capture chain (zoom, focus, light, exposure parameters, etc.) Thus, image capture parameters may be parameters that affect the appearance of the image and / or the field of view of the image.

[0031] Such a technique therefore allows for a situation-specific adjustment of the value of the at least one imaging parameter, the value of which can be reliably found by the search images of the second search image sequence depending on the specific situation relevant for the visualization of the respective patient.

[0032] For example, evaluating one or more search images of the first search image sequence can include identifying a location of a marker fixed in position relative to the patient in the one or more search images of the search image sequence. A value of at least one image acquisition parameter can be identified based on the location of the marker fixed in position relative to the patient.

[0033] For example, the location of the second search area with respect to the first search area can be determined by setting the field of view of the search image of the second search area. For example, the relative location of the marker with respect to the target area is essentially known in advance. The marker can be attached to the patient with an offset with respect to the target area, for example by a clamp or other fixing means. This offset can be known. The second search area can then be located relative to the first search area or the marker found in the search image of the first search area.

[0034] Another exemplary implementation of the evaluation of one or more search images of the first search image sequence may include, for example, checking the value of one or more illumination parameters for recording the second search image sequence. In this regard, it may be conceivable to, for example, evaluate the brightness histogram of one or more search images of the first search image sequence, and then set the exposure time and / or the brightness of the illumination for recording one or more search images of the second search image sequence such that the brightness histogram of one or more search images of the second search image sequence corresponds to a predefined target. In this way, it may be ensured that the contrast used for imaging, for example, a specific anatomical feature or skin tissue in one or more search images of the second search image sequence images details for finding the target area.

[0035] In another example, evaluation of one or more search images of a first search image sequence may be used as a basis for determining whether one or more search images of the first search image sequence can be reused in a second search image sequence.

[0036] For example, if a second search area of ​​the second search image sequence is identified such that it has an overlap with a first search area of ​​the first search image sequence, then in the corresponding overlap area, a search image already recorded for the first search image sequence can be reused for the second search image sequence, thus reducing the time required to capture the search image.

[0037] Generally speaking, it is conceivable that at least one search image is part of a first search image sequence and also part of a second search image sequence.

[0038] For example, the second search image sequence could be evaluated to recognize skin tissue of the patient in one or more search images of the second search image sequence, and the location of the target region could then be identified based on the recognition of the skin tissue.

[0039] A patient is typically covered for the most part, for example by a surgical drape. Only the target area and its immediate surrounding area may be exposed. Thus, the target area can be identified by recognition of the skin tissue.

[0040] The method may include receiving tracking data from a tracking system. The tracking system may be collocated with the robotic visualization system. This means that the robotic visualization system and the tracking system may operate in a common reference coordinate system or the tracking system may be used to determine the location of the robotic visualization system (and therefore the pose or field of view of the measurement data or images captured by the robotic visualization system) in the reference coordinate system. The tracking system may also be configured to determine the location (i.e., position and / or orientation) of other markers in the reference coordinate system. The reference coordinate system may be determined with respect to the operating room.

[0041] The tracking data can describe the placement of markers fixed in position with respect to the patient with respect to the robotic visualization system. The markers can be implemented, for example, by passive machine-readable symbols or active machine-readable light sources.

[0042] Thus, the method may further include determining, based on the tracking data, a value of at least one image capture parameter of the robotic visualization system used to record the at least one exploratory image sequence.

[0043] For example, the location of the corresponding search area covered by the at least one search image sequence can be determined based on the tracking data, e.g., if a marker is recognized by the tracking system, the relative position of the marker with respect to the target area can be known in advance.

[0044] In general, it is therefore possible to determine, based on prior knowledge, the value of at least one image capture parameter of the robot visualization system used to record at least one search image sequence. For example, the brightness of the lighting can be set as an exemplary image capture parameter. The exposure time can be set as an explicit image capture parameter. The arrangement of the corresponding search areas can be determined. The reuse of search images between different search image sequences can be determined.

[0045] Such prior knowledge may for example be about the (rough) positioning of the patient with respect to the robotic visualization system. Alternatively or additionally, such prior knowledge may also be about the positioning of the target area relative to a marker fixed in position with respect to the patient. For example, specifically, the patient may be roughly pre-positioned with respect to the robotic visualization system. For example, the patient may be positioned on a patient couch, the positioning of which with respect to the robotic visualization system is known. In this way, in a first step, the robotic visualization system may be roughly aligned with the (assumed) position of the patient, after which the search images of at least one search image sequence may be captured based on the prior knowledge. The same may apply to the marker fixed in position with respect to the patient.

[0046] The method may further comprise evaluating already captured search images of the at least one search image sequence during the recording of the at least one search image sequence. During the evaluation, the recording of the at least one search image sequence may be selectively terminated, i.e. no further capture of search images may take place. Alternatively or additionally, it may be conceivable to adjust a value of at least one image capture parameter of the robotic visualization system used for recording the at least one search image sequence based on the evaluation.

[0047] In other words, if multiple search images are captured, it may be continuously checked whether further search images need to be captured (or whether, for example, the target area has already been recognized and further capture of search images is necessary). Furthermore, it may be possible to check whether the value of at least one image capture parameter has been properly set or whether an adjustment, such as brightness or exposure time, should be made.

[0048] In general, the location of the target region can be identified based on structure recognition of one or more predefined structures in at least one search image. In this regard, for example, structures of skin tissue can be recognized in at least one search image. Alternatively or additionally, structures of specific anatomical features (e.g., nose, ears, mouth, eyes, etc.) can be recognized in at least one search image. Any suitable algorithm can be used. The specific implementation of such an algorithm for identifying the location of the target region is not essential to the implementation of the technology described herein, but may rely on previously known implementations.

[0049] To record the at least one search image, for example an overview image capture unit of the robotic visualization system and / or a microscope image capture unit of the robotic visualization system can be activated, which means in other words that for example the at least one search image can comprise a relatively low magnification overview image and / or a relatively high magnification microscope image.

[0050] The measurement data may be captured, for example, by a measurement method selected from the following group: stereoscopic imaging, time-of-flight measurements, structured illumination, and unfocused depth estimation.

[0051] In stereo imaging, information about the topography is acquired by two cameras imaging the same area at different orientations.

[0052] In the case of time-of-flight measurements, information about the topography can be obtained by a short light pulse and monitoring the corresponding time-of-flight.

[0053] In structured illumination, a predefined pattern, for example a line pattern, is projected onto the surface to be measured, and the distortion of the predefined pattern can then be used to obtain information about the topography.

[0054] For unfocused depth estimation, information about the topography can be obtained through the local use of autofocus techniques.

[0055] Generally, it will be considered that the measurement data comprises at least one search image, which means that it is not absolutely necessary that the measurement data is captured completely separately from the search image.

[0056] The method may further comprise filtering the measurement data, for example in this case data elements that do not image the patient's skin tissue may be removed from the measurement data, for example areas that image a surgical drape or a marker or a surgical instrument may be removed from the measurement data, so that misregistration of the measurement data with respect to the preoperative volumetric image data is avoided.

[0057] Prior to recording the at least one scout image, motors of the robotic visualization system can be activated to position the robotic visualization system in a predefined reference position. For example, the predefined reference position can be defined with respect to the operating table. For example, the predefined reference position can be defined such that an area of ​​the operating table where the patient's head is positioned is within the field of view of an image capture unit of the robotic visualization system used to initially record the at least one scout image.

[0058] It would be envisaged that the method further comprises setting at least one image capture parameter of the robotic visualization system to a predefined value prior to recording of the at least one search image. This may for example comprise setting a target magnification for a zoom factor of a corresponding image capture unit of the robotic visualization system. A particular brightness of the illumination may be selected. It would be envisaged that the exposure time is set to a particular value.

[0059] The methods described herein can be executed automatically based on a predefined control script. For example, the control script can be started by a single user command (e.g., "one click"). In this way, a particularly simple and fast measurement data capture can be achieved, which allows the specification of the transformation and can make manual alignment of the robot visualization system unnecessary.

[0060] For example, such a control script may include an analysis module for evaluating image data or measurement data. Furthermore, the control script may include at least one positioning module for positioning the robotic visualization system. Such positioning may be used, for example, by scanning, to capture search images of a search image sequence covering a search area larger than the field of view of an individual search image. The analysis module may evaluate the image data, for example, to locate skin tissue. The analysis module may evaluate the image data, for example, to recognize markers fixed in position with respect to the patient.

[0061] The analysis module may for example comprise one or more machine learning algorithms. For example, an artificial neural network may be used that recognizes predefined structures or objects in the image. For example, a convolutional neural network may be used for this purpose. Corresponding training may for example be enabled by manual annotation of corresponding training image data or training measurement data.

[0062] On the other hand, it would be conceivable that the localization module does not include a machine learning algorithm: a manually parameterized algorithm could be used, which is particularly worth doing, since in this way, for each input parameter, the new position of the localization module in the validation phase can be validated to avoid uncontrolled or dangerous movements of the robot visualization system.

[0063] The method may further comprise generating a topographical data set based on the measurement data, which means that the measurement data may be evaluated, for example by an analysis module as described above.

[0064] The method may further include performing a registration of the topography dataset and the preoperative volumetric image data. This means that corresponding points in the topography dataset and the preoperative volumetric image data may be found. The transformation specification may then be determined based on the relative placement of these corresponding points with respect to each other. In general, the transformation specification may be determined based on the registration.

[0065] The data processing unit is configured to operate the robotic visualization system for thus recording at least one search data, the at least one search data at least partially imaging the patient. Furthermore, the data processing unit is configured to operate the robotic visualization system for determining a location of the target area in the at least one search image and for recording measurement data with depth resolution based on this location of the target area in the at least one search image. These measurement data are indicative of a topography of the patient's anatomical structure. This allows for determining a transformation specification between the images recorded by the robotic visualization system and the patient's preoperative volumetric image data.

[0066] The system includes a data processing unit and a robot visualization system.

[0067] The computer program or computer program product or computer readable storage medium includes a program code. The program code can be loaded and executed by a processor, which causes the processor to execute a method. The method thus includes operating the robotic visualization system to record at least one search image. The at least one search image at least partially images the patient. Furthermore, the method also includes identifying a location of the target area in the at least one search image. Furthermore, the method includes operating the robotic visualization system to record measurement data having depth resolution based on the location of the target area in the at least one search image. In this case, the measurement data is indicative of a topography of the patient's anatomy. The measurement data allows identifying a transformation specification between the image recorded by the robotic visualization system and the patient's preoperative volumetric image data.

[0068] The features mentioned above and those to be described below can be used not only in the respective combinations explicitly indicated, but also in other combinations or separately, without departing from the scope of protection of the invention. [Brief description of the drawings]

[0069] [Figure 1] 1A-1D are schematic diagrams illustrating various exemplary optical visualization systems; [Diagram 2] 1 illustrates a schematic representation of a data processing unit according to various examples; [Diagram 3] 1 is a flow chart of an exemplary method. [Figure 4] Various example individual search images and target regions are shown. [Diagram 5] 1 shows various example search image sequences and target images. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0070] The foregoing characteristics, features and advantages of the present invention and the manner in which they are realized will become more apparent and more clearly understood from the following description of illustrative embodiments which are set forth in more detail in conjunction with the drawings.

[0071] The present invention will be described in more detail below based on preferred embodiments with reference to the drawings. In the drawings, the same reference numerals refer to the same or similar elements. The drawings are schematic representations of various embodiments of the present invention. The elements shown in the drawings are not necessarily drawn to scale. Rather, the various elements shown in the drawings are drawn in such a way that their functions and general purpose are easily understood by those skilled in the art. The connections or couplings between the functional units and elements shown in the drawings can also be implemented as indirect connections or couplings. The connections or couplings can be implemented in a wired or wireless manner. The functional units can be implemented as hardware, software, or a combination of hardware and software.

[0072] Below, we describe a technique that makes it possible to specify a transformation specification for mapping or transforming image data captured by a robotic visualization system (hereinafter simply referred to as a surgical microscope) into preoperative volumetric image data, although an inverse mapping specification can also be included in the transformation specification.

[0073] For this purpose, a target region is automatically recognized or its location is automatically determined based on one or more search images, for which depth-resolved measurement data can then be determined and registered with respect to the preoperative volumetric image data, from which a transformation specification can be derived.

[0074] Various applications can then be enabled based on the transformation specification. For example, functions to assist the surgeon can be provided based on the preoperative volumetric image data and the transformation specification. For example, the preoperative volumetric image data or a part thereof can be inserted in a spatially correct manner into the field of view of a surgical microscope or into image data captured by the surgical microscope. Particular areas identified in the preoperative volumetric image data can be highlighted in the image data captured by the surgical microscope. Navigational assistance functions to guide the surgical intervention can be enabled.

[0075] 1 shows a schematic representation of a surgical microscope 801 for surgery. In the illustrated example, the surgical microscope 801 comprises an eyepiece 803. Through the eyepiece 803, the surgeon can see a magnified image of an object that is within the field of view 804 of the surgical microscope 801. In the illustrated example, this is a patient 805 lying on a patient couch.

[0076] Alternatively or in addition to an optical eyepiece, a camera 809 (digital surgical microscope) can be provided which transmits an image to a screen.

[0077] In general, the surgical microscope 801 may include one or more image capture units configured to record digital images. Examples of image capture units are e.g. a microscope image capture unit and an overview image capture unit. The microscope image capture unit may include one or more cameras, e.g. stereoscopic image capture may be possible with multiple cameras.

[0078] A manipulation device 808 is also provided as a human-machine interface, which can be implemented for example as a handle or a foot switch. In the embodiment shown in Fig. 1, this is a handle. The manipulation device 808 allows the movement of the eyepiece 803 fixed to the cross beam 850. A motor can be provided to perform this movement automatically based on control data, according to the corresponding settings of the surgical microscope. The motor can also assist the movements initiated by the handle 808.

[0079] Additionally, a control device 880 for the surgical microscope 801 is provided, controlling the operation of the surgical microscope 801 and the display of the image and additional information and data at the eyepiece 803. The control device 880 can perform interaction with the surgeon.

[0080] The surgical microscope 801 may also include one or more further sensors 860, such as an environmental camera, a time-of-flight sensor (TOF camera), an image recording device for recording images captured in the case of structured illumination, etc., and measurement data having, for example, a depth resolution may be captured by such a sensor 860. This means that the depth resolution may describe the distance between the corresponding sensor 860 and the patient 805.

[0081] Another implementation of such a sensor 860 is in relation to an internal tracking system. For example, a marker that is fixed relative to the patient and is implemented as machine-readable can be recognized by such an internal tracking system. The marker may also be referred to as a target. In particular, the relative position of the marker with respect to the surgical microscope 801 can be determined.

[0082] 1 also shows an external tracking system 890, which is placed alongside the surgical microscope 801. Tracking data 890 can be transferred from the tracking system 890 to the control device 880, and indicates, for example, the positioning of the field of view 804 of the camera 809 or eyepiece 803, or of the image capture unit in general, relative to a marker that is fixedly positioned in space, for example relative to the patient 805, or spatially fixed in a reference coordinate system.

[0083] 2 shows the relevant parts of the data processing unit 910. The data processing unit 910 can be implemented, for example, by a computer. The data processing unit 910 can be implemented, for example, in the control device 880 (see FIG. 1) of the surgical microscope 801, or can be implemented separately from the surgical microscope 801 (and for example in communication with the control device 880).

[0084] The data processing unit 910 includes a computing unit 911, a memory 912, and a communication interface 913. The computing unit 911 can, for example, load program code from the memory 912 and execute said code. The computing unit 911 can communicate with other devices, nodes, or apparatuses via the communication interface 913. For example, tracking data can be received from a tracking system. Pre-operative volumetric image data can be received, for example, from an image archive system or any other database. User input can be received from a human-machine interface.

[0085] The computing unit 911 executes the program code loaded from the memory 912, which may, for example, cause the computing unit 911 to perform the techniques described herein, such as recording a search image or measurement data, evaluating the search image or measurement data, determining the location of a target area in the search image, operating the surgical microscope 801 to capture an image, a search image, or measurement data, and / or at a specific position in the field of view 804, providing control data to enable assistance functions for the surgeon, determining a conversion specification between the image data captured by the surgical microscope 801 and the pre-operative volume image data, for example based on a corresponding registration between the measurement data with depth resolution and the pre-operative volume image data. Various functionalities that the computing unit 911 may provide and perform based on the corresponding program code are described below with respect to the method of FIG. 3.

[0086] Figure 3 is a flow chart of one exemplary method. For example, the method of Figure 3 may be performed by a data processing unit, such as the data processing unit 910 of Figure 2. In particular, the method of Figure 3 may be performed by a computing unit, such as a processor, when it loads and executes the program code.

[0087] The method of Fig. 3 serves to allow the specification of a transformation between the image recorded by the surgical microscope and the preoperative volume image data, for example CT image data or MRI image data. In other words, the method of Fig. 3 therefore serves to calibrate the position of the patient with respect to the surgical microscope and the preoperative volume image data. In particular, the method of Fig. 3 supports the automatic capture of measurement data with depth resolution by the surgical microscope, whereby these measurement data can be registered with respect to the preoperative volume image data. The transformation specification can then be specified on the basis of this registration, but the transformation specification can also be specified by downstream methods, for example by different data processing systems, for example by the control equipment of the tracking system.

[0088] It is conceivable, for example, for the method according to Fig. 3 to be executed on the basis of an automatic control script, which means that the various method steps can be processed automatically and successively by a computer program, with the advantage that, for example, a user can start the corresponding control script with only one human-machine interaction, and no manual interaction is required during the processing of the corresponding method steps.

[0089] Optional boxes are represented by dashed lines in FIG.

[0090] First, in box 3005, the motors of the surgical microscope can be activated to place the surgical microscope in a predefined reference position. For example, in this reference position, the field of view of the surgical microscope can be aligned with the head of a patient when the patient is lying on a couch for this purpose.

[0091] Afterwards, in box 3010, an initialization of one or more parameters can optionally take place. This means that at least one image capture parameter of the surgical microscope can be set to a predefined value before recording the search image. For example, the brightness of the illumination can be initialized or the zoom factor can be set. A predefined focus can be selected. A specific field of view can be set.

[0092] Optionally, box 3010 and box 3005 can be performed together. For example, for an image capture parameter, box 3010 sets the value of the corresponding image capture unit of the surgical microscope.

[0093] Box 3011 then comprises recording at least one search image that at least partially images the image. For this purpose, the surgical microscope, or more precisely, a corresponding image capture unit or a corresponding control device, can be activated. A command can be sent to capture the corresponding at least one search image.

[0094] For example, the overview image capture unit of the surgical microscope and / or the microscope image capture unit of the surgical microscope can be activated to record at least one search image, which therefore means that the at least one search image can be realised for example by means of one or more overview images at low magnification and / or one or more microscope images at high magnification.

[0095] There are various variations on box 3011. Before these variations are described in detail below, the method thereafter will first be described in more detail.

[0096] Box 3030 includes determining the location of the target region in the at least one search image. This means determining the position and / or orientation and / or extent of the target region in the at least one search image. An analysis module can be used. For this purpose, for example, a machine learning algorithm can be used. Such a machine learning algorithm can be trained based on training search images and manual annotation of the target region. The machine learning algorithm can be implemented, for example, by a convolutional neural network.

[0097] The target region may refer, for example, to a planned surgical intervention area. The target region may refer to an area of ​​the patient's body for which preoperative volumetric image data was captured or imaged by the preoperative volumetric image data.

[0098] For example, the location of the target region can be identified based on structural recognition of one or more predefined structures in at least one search image, for example skin tissue or anatomical features, but also machine-readable markers can be recognized.

[0099] Based on this arrangement of the target area, measurement data with depth resolution can then be captured in box 3035. For this purpose, the surgical microscope, for example a corresponding sensor, a corresponding image acquisition unit or an assigned control device can be activated. The measurement data thus represents the topography of the patient's anatomy in the area of ​​the target area.

[0100] For example, the measurement data may be captured by a measurement method selected from the group including stereoscopic imaging, time-of-flight measurements, structured illumination, and unfocused depth estimation.

[0101] In some examples, it is also possible that the measurement data comprises at least one of the at least one search image, which means that a particular image can therefore be used as a search image in box 3011, but also considered with respect to the measurement data in box 3035, i.e. performing a dual function.

[0102] In box 3036, the measurement data can optionally be filtered. For example, noise suppression can be performed. Background can be removed. Data elements that do not image the patient's skin tissue can be removed.

[0103] In this way it may be possible to specify a conversion specification between the images recorded by the surgical microscope and the pre-operative volumetric image data of the patient, which conversion specification may be specified in optional box 3040. It is also possible to store the measurement data for later specification or to communicate them to a different data processing unit, which specifies the conversion specification.

[0104] If a transformation specification is determined, this can be done as follows: first, a topography data set can be generated based on the measurement data. This can determine, for example, a height profile of the patient's skin surface of the patient's head, or generally in the target area. Then, a registration of this topography data set with the preoperative volumetric image data can be performed. In particular, the registration with the preoperative volumetric image data can in particular image the topography of the skin surface of the patient's head. Based on this registration, a transformation specification can be determined, for example taking into account translational and / or rotational degrees of freedom. Distortion can also be taken into account.

[0105] Optionally, applications based on this transformation specification can be enabled in box 3045. For example, computer-assisted surgery can be enabled for an augmented image, e.g., captured by a surgical microscope and having information determined based on pre-operative volumetric image data. Navigational assistance functions are enabled.

[0106] Details regarding box 3011, i.e. recording at least one search image, are given below. In one simple variant, it is possible to record one search image and identify the location of the target area in the one search image. Such a scenario is illustrated in Fig. 4. Fig. 4 shows a search image 111. The search image 111 images the target area 131. For example, the search image 111 can be captured by an overview image capture unit of a surgical microscope 801 (see Fig. 1).

[0107] In other variants, one or more search image sequences can also be captured. For example, Fig. 5 shows a scenario in which a total of 25 search images 111-113 are captured (only the first three search images are numbered for clarity). For example, each of these search images of the search image sequence can have a particularly high resolution. In this way, a high magnification can also be made possible. This corresponds to a scan of the corresponding search area. The scan can be realized by actuating a motor of the surgical microscope, whereby the position of the surgical microscope is changed, for example many times between the capture of the various search images of the search image sequence.

[0108] In some examples, multiple search image sequences can also be captured consecutively. Such a scenario is depicted in FIG. 3 with respect to box 3011. In this case, first a first search image sequence is recorded in box 3015, in which a first search area is assigned, and then a second search image sequence is recorded in box 3025, in which a second search area is assigned. During this, in box 3020, one or more search images of the first search image sequence in box 3015 can be evaluated and based on this evaluation, values ​​of at least one image acquisition parameter of the surgical microscope can be identified, which are then used to record a second search image sequence in box 3025.

[0109] This evaluation in box 3020 may include, for example, identifying the location of a marker fixed in position relative to the patient in one or more scout images of the first scout image sequence of box 3015.

[0110] This makes it possible, for example, to determine the positioning of the second search area for which the second search image sequence was captured in box 3025 with respect to the first search area of ​​the first search image sequence in box 3015. It is also possible to check whether the first search image sequence is reused for the second search image sequence. It makes it possible, for example, to set the lighting parameters for recording the second search image sequence. It makes it possible to set the magnification of the search images of the second search image sequence.

[0111] If multiple search image sequences are used, then in principle at least one search image can be part of multiple search image sequences, as illustrated in FIG. 3 with respect to boxes 3015 and 3025.

[0112] A description of a workflow that can implement the method of FIG. 3 is provided in Table 1 below.

[0113] [Table 1]

[0114] [Table 2]

[0115] [Table 3]

[0116] [Table 4]

[0117] [Table 5]

[0118] In summary, a technique has been described that allows for automatic recognition of the scanned area, i.e., the target region, for patient alignment. Automatic alignment of a robotic visualization system has been disclosed that is capable of capturing measurement data that has depth resolution and images the target region.

[0119] It goes without saying that the features of the above-described embodiments and aspects of the invention can be combined with each other, in particular the features are not limited to the combinations described but can also be used in other combinations or on their own without departing from the scope of the invention.

Claims

1. - activating the robotic visualization system (801) to record at least one exploration image (111-113) at least partially depicting the patient (805); - determining the location of a target area (131) in said at least one search image (111-113); - operating the robotic visualization system (801) to record depth-resolved measurement data based on the location of the target area (131) in the at least one scout image (111-113), the measurement data being indicative of a topography of the anatomical structures of the patient (805), thereby allowing the identification of a transformation specification between images recorded by the robotic visualization system and pre-operative volumetric image data of the patient; 23. A computer-implemented method comprising:

2. the robotic visualization system is operated to record at least one scout image sequence, each of the at least one scout image sequence including a plurality of corresponding scout images; - actuating at least one motor of the robot visualization system in order to change the position of the robot visualization system a number of times during the recording of the at least one search image sequence, thereby enabling a search area assigned to the respective search image sequence to be scanned by a plurality of search images of the respective search image sequence, 10. The computer-implemented method of claim 1.

3. for example, a first search area is assigned to a first search image sequence of the at least one search image sequence; a second search area is assigned to a second search image sequence of the at least one search image sequence; the robotic visualization system is first operated to record the first scout image sequence and thereafter to record the second scout image sequence; moreover, - evaluating one or more scout images of the first image sequence and determining, based on the evaluation of one or more scout images of the first scout image sequence, values ​​of at least one image capture parameter of the robot visualization system to be used for recording the second scout image sequence; The computer-implemented method of claim 2 , comprising:

4. evaluating the one or more search images of the first search image sequence includes identifying a location of a marker in the one or more search images of the first search image sequence, the marker being fixed in a fixed position relative to the patient; the value of the at least one imaging parameter is determined based on the location of the marker fixed in place with respect to the patient.

4. The computer-implemented method of claim 3.

5. the at least one image capture parameter is selected from the group comprising: a positioning of the second search area relative to the first search area, reusing search images of the first search image sequence for the second search image sequence, and lighting parameters for recording the second search image sequence.

5. A computer-implemented method according to claim 3 or 4.

6. at least one search image is part of both the first search image sequence and the second search image sequence; A computer implemented method according to any one of claims 3 to 5.

7. - evaluating one or more search images of said second search image sequence in order to recognize skin tissue of said patient in said one or more search images of said second search image sequence. Further comprising: The location of the target area (131) is determined based on recognition of the skin tissue. A computer implemented method according to any one of claims 3 to 6.

8. receiving tracking data from a tracking system collocated with respect to the robotic visualization system, the tracking data describing a position of a marker fixed in position with respect to the patient relative to the robotic visualization system; - determining, based on said tracking data, values ​​of at least one image capture parameter of said robot visualization system used for recording said at least one exploratory image sequence; Further comprising: A computer implemented method according to any one of claims 2 to 7.

9. - determining values ​​of at least one image acquisition parameter of the robotic visualization system used to record the at least one exploratory image sequence based on prior knowledge of at least one of the positioning of the patient relative to the robotic visualization system and the positioning of the target area (131) relative to a marker fixed in place with respect to the patient. Further comprising: A computer implemented method according to any one of claims 2 to 8.

10. during said recording of said at least one search image sequence, evaluating already captured search images of said at least one search image sequence; - terminating the recording of the at least one scout image sequence and / or adjusting a value of at least one image capture parameter of the robot visualization system used to record the at least one scout image sequence depending on the evaluation; Further comprising: A computer implemented method according to any one of claims 2 to 9.

11. the location of the target region (131) is identified based on structure recognition of one or more predefined structures in the at least one search image; The one or more predefined structures are selected from the group including skin tissue, anatomical features; A computer implemented method according to any preceding claim.

12. at least one of an overview image capture unit of the robotic visualization system and a microscopic image capture unit of the robotic visualization system (801) is activated to record the at least one search image (111-113); A computer implemented method according to any one of claims 1 to 11.

13. The measurement data is captured by a measurement method selected from the following group: stereoscopic imaging, time-of-flight measurements, structured illumination, and unfocused depth estimation. A computer implemented method according to any preceding claim.

14. the measurement data includes at least one of the at least one search image; A computer implemented method according to any preceding claim.

15. - filtering (3036) the measurement data to remove data elements that do not image the patient's skin tissue; The computer-implemented method of any one of claims 1 to 14, further comprising:

16. - actuating (3005) the motors of said robotic visualization system (801) in order to position said robotic visualization system (801) in a predefined reference configuration prior to said recording of said at least one search image (111-113), The computer-implemented method of any one of claims 1 to 15, further comprising:

17. - setting (3010) at least one image capture parameter of said robot visualization system (801) to a predefined value prior to said recording of said at least one scout image (111-113), The computer-implemented method of any one of claims 1 to 16, further comprising:

18. The method is carried out automatically based on a predefined control script. A computer implemented method according to any preceding claim.

19. the control script includes an analysis module for evaluating image data or measurement data; the control script includes a positioning module for positioning the robotic visualization system; the analysis module includes one or more machine learning algorithms; The positioning module does not include a machine learning algorithm.

20. The computer-implemented method of claim 18.

20. generating topographical data based on said measurement data; - performing a registration between said topographical data set and said preoperative volumetric image data; - determining said transformation specification based on said alignment; The computer-implemented method of any one of claims 1 to 19, further comprising:

21. - activating the robotic visualization system to record at least one exploratory image at least partially depicting the patient; - determining the location of a target area (131) in said at least one search image; - operating the robotic visualization system to record measurement data having depth resolution based on the location of the target area (131) in the at least one scout image, the measurement data being indicative of a topography of the patient's anatomy, thereby allowing the specification of a transformation between images recorded by the robotic visualization system and pre-operative volumetric image data of the patient; A data processing unit (910) configured to execute:

22. A data processing unit (910) according to claim 21, configured to execute the method according to any one of claims 1 to 20.

23. - a data processing unit according to claim 21 or 22, - A robot visualization system; A system including:

Images