Method and system for providing navigation support to a person to navigate about a resected part

Abstract

The present invention relates to a method for providing navigation support to a person performing a resection on a patient (3) for navigation relative to the resection after the resection (5) has been extracted, the method comprising the following steps: recording a first surface data set of the resection using an ex vivo scanning device (16); determining a second surface data set of an extraction region (6) using a recording instrument (9, 11), the second surface data set covering at least a portion of a remaining tissue surface (7) of the extraction region (6) of the resection (5) in the patient (3); aligning the first surface data set of the resection (5) with the second surface data set of the extraction region (6) based on corresponding surface features of the remaining tissue surface (7) and the resection in the extraction region; and performing at least one support measure using the alignment, the support measure supporting navigation relative to the resection in the extraction region.

Description

Technical Field

[0001] The invention relates to a method and a support system for navigationally supporting a person performing a resection on a patient after the resection has been extracted, in order to navigate in the extraction area relative to the resection. The invention also relates to a computer program and an electronically readable data carrier. Background Art

[0002] A possibility for treating malignant tissue, such as a tumor, that occurs in a patient is to remove the malignant tissue. This process is also referred to as resection; the portion of tissue that should contain the malignant tissue as completely as possible is referred to as the resection portion. Here, the purpose is the so-called R0 resection, which means that it is possible to prove that there is no more malignant tissue within the resection margin, or that the defined minimum spacing to the malignant tissue is observed. For example, after a tumor resection, the surgeon wants to know as quickly and convincingly as possible whether the resection was successful in terms of oncology, so there should not be any tumor tissue residue in the patient. If there is no R0 resection, and if this information is presented quickly, subsequent resections can be performed again in a rapid manner and additional tissue can be removed.

[0003] In this case, for example, it is known that the excision in pathology is subjected to so-called rapid section diagnosis. Here, tissue sections are made and checked to see whether the spacing between the malignant tissue and the edge of the excision is observed. After about 10 to 15 minutes, the corresponding diagnosis can be reported back to the operating room, for example by telephone.

[0004] In order to improve the situation in this respect, digital histology devices for operating rooms have been proposed in the meantime, which are also called ex vivo scanning devices (Ex-Vivo-Scaneinrichtung). Here, for example, after tumor surgery, a resection removed from the patient can be scanned on the surface to provide a conclusion as to whether malignant tissue is present on the surface of the resection. Subsequently, a rapid slice diagnosis can then be performed by a pathologist. However, the diagnostic results of the ex vivo scanning device can be provided directly in the operating room. The doctor can therefore view the results of the digital histology device and roughly estimate the locations where subsequent resections are expected to be required. However, there is the problem that this corresponds only to a rough estimate and that erroneous and / or significantly excessive subsequent resections may occur.

[0005] So-called digital biopsy tools have also been proposed, which allow point-by-point documentation at the surgical site. Here, small surface segments (e.g. in the range of a few square millimeters) formed by the removal of the resection are scanned by optical methods such as microscopy and / or optical coherence tomography.

[0006] Such a digital biopsy tool is known, for example, from DE 10 2014 103 044 A1. There, a set of devices includes at least an endoscope for collecting cell-resolved image data of the surgical field. In addition, an imaging device, in particular a surgical microscope, can be provided for recording overview image data of the surgical field.

[0007] The problem with this digital biopsy tool is that it can only scan segments point by point, so it is not suitable for scanning large areas due to the high time cost. Here, scanning a large area should currently be understood as a size of at least 1 cm. 2 area.

[0008] US 2017 / 0105601 A1 relates to a method and system for calculating the volume of a resected tissue (i.e., a resected part) from an intraoperative image stream. In this case, two-dimensional or 2.5-dimensional intraoperative images, for example recorded using a laparoscope, are recorded from different angles relative to the resected part, wherein the resected tissue surface is segmented in each intraoperative image, and the segmented tissue surfaces are merged to obtain a three-dimensional point cloud representation of the resected part surface. This can then be used for volume determination. Summary of the invention

[0009] The object of the present invention is therefore to provide a person who performs the resection with an improved possibility for associating features of the resected part with the extraction region.

[0010] In order to solve the above technical problems, the present invention provides a method, a support system, a computer program and an electronically readable data carrier according to the present invention. The present invention also provides an advantageous design solution.

[0011] The method according to the invention for navigation support of a person performing a resection on a patient after the extraction of the resection for navigation in the extraction area relative to the resection has the following steps:

[0012] - recording a first surface data set of the resected part using an ex vivo scanning device,

[0013] - determining a second surface data set of the extraction region using the recording device, which second surface data set covers at least a part of the surface of the extraction region of the resected part in the patient,

[0014] - registering the first surface data set of the resected portion with the second surface data set of the extracted region based on the remaining tissue surface in the extracted region and the corresponding surface features of the resected portion, and

[0015] Using the registration, at least one supporting measure is performed which supports navigation in the extraction region relative to the resection part.

[0016] Therefore, the present invention proposes to establish a correspondence between the position on the resection part and the position on the remaining tissue surface in the extraction area in the patient's body formed by extracting the resection part on the surface, so that if the position of the relevant indicative feature on the surface of the resection part is known, the support measures, in particular by reproducing the display, are suitable for guiding the person performing the resection to the corresponding position on the surface in the extraction area, or making it possible to find the corresponding position faster. According to the basic idea of ​​the present invention, microscopic and macroscopic cell structures and / or tissue structures form recognizable surface features in surface data or general image data, wherein there is a strong similarity of these features at the resection part and at the surface area in the extraction area previously connected to the resection part, because they initially form a continuous or almost continuous structure before the resection is performed and are only cut apart from each other during the resection. On the other hand, if the correlation of the clearly identifiable surface features has a poor recognizability, the correspondence can also be established by observation during the resection process, which will be discussed in more detail. Examples of similarities that can be found include patterns of cell arrangement, tissue boundary layers, blood vessels, tissue density and cell types and, if necessary, infiltration patterns of tumor cells.

[0017] The corresponding features thus indicate corresponding, i.e. corresponding, positions on the corresponding surfaces. Here, a position on the surface of the resected part corresponds to a position on the surface of the extraction area formed by removing the resected part if the position on the surface of the resected part and the position on the surface of the extraction area are directly or almost directly connected before the resection. In other words, the registration method described herein serves to associate a position on the surface remaining in the patient's body and formed by the resection with a position on the surface of the resected part, where the position on the surface remaining in the patient's body and formed by the resection was previously as close as possible to the previously mentioned position (position on the surface of the resected part). The position on the surface of the resected part, for example the position of an indicative feature, is referred to as a R The corresponding position on the surface in the extraction area formed by the excision is referred to as b V Here, a can be defined two-dimensionally on a suitable coordinate system of the tissue surface of the resected part. R Alternatively, a can be defined three-dimensionally in a 3D coordinate system of the cutout or its surface. R For b V, this applies correspondingly to the coordinate system of the remaining tissue surface or the remaining tissue volume, wherein preferably a patient-related coordinate system is used here or at least a coordinate system registered with such a patient-related coordinate system, wherein the patient-related coordinate system is used, for example, by a navigation system which can in particular track instruments and / or devices. For example, the correspondence of positions can be defined mathematically in the following manner. B V represents the three-dimensional position in the continuous tissue before resection and the position of the remaining tissue after resection. V The cell / tissue element at that location was originally located at that location. R Correspondingly, it represents the three-dimensional position in the continuous tissue before resection, and the position of the resected part a R The cell / tissue element at position B is initially located at this position. The set {b} of all positions on the remaining tissue surface formed by resection is taken as position B in the initial three-dimensional continuous tissue volume. V With A R The minimum spacing position "b V " means "corresponding to a R ”.

[0018] The registration result thus makes it possible to associate the corresponding positions on the surface of the extracted area in the patient with the positions on the surface of the resected part. In this way, a link between digital histology and intraoperative navigation / imaging can be established. The person performing the resection is supported in the rapid and targeted correction of, for example, incomplete resections. The results of digital histology can be transmitted directly to the surgical site, thus making it possible to directly correct errors in the resection.

[0019] Preferably, the recording instrument can be tracked in a patient-related coordinate system by means of a navigation system, wherein the second surface data set is also recorded or determined or exists in the patient-related coordinate system. Here, the support measures can in particular include displaying the relative position of at least one indicative feature of the resected portion relative to the recording instrument and / or relative to another instrument tracked in the patient-related coordinate system by the navigation system and / or relative to the surface of the extraction area. Therefore, the navigation system working in the patient-related coordinate system can be used particularly advantageously. If, for example, the recording instrument is originally tracked by the navigation system, or at least the recording instrument can be converted into the patient-related coordinate system, and now there is a second surface data set in the same coordinate system, then the analysis results of the first surface data set, in particular the indicative features (i.e., the position-related information of the implant) can be located in the patient-related coordinate system by means of the registration results, thereby in particular using it for support measures there. Therefore, it is particularly conceivable to display other information present in the patient-related coordinate system, such as the relative position of the indicative features and the recording instrument and / or the instruments and tools used in addition. In addition to the support measures to be displayed, which make the person performing the resection intuitively aware of the position of the indicative features of the resection part in the patient-related coordinate system, other support measures are also conceivable within the scope of the present invention, for example, navigation-type target guidance outputs, etc. In particular, other output measures can be used as support measures to support the support measures to be displayed, for example in accordance with the type of acoustic navigation, in which, for example, the frequency of a continuous tone gradually increases as the instrument / tool ​​gets closer to the indicative feature, etc.

[0020] As already described above, indicative features can be determined in particular by analyzing the first surface data set and / or a further image data set of the resected portion registered with the first surface data set, wherein the further image data set of the resected portion is preferably also recorded using an ex vivo scanning device. For example, an ex vivo scanning device can be designed which records the resected portion in different modalities, for example by means of a camera and / or a microscope and / or optical coherence tomography and / or fluorescence imaging, wherein the modality is selected for the first surface data set which best corresponds to the modality of the second surface data set. Since the same ex vivo scanning device is used, the further image data set of the resected portion is automatically registered with the first surface data set of the resected portion.

[0021] At this point it should be mentioned that at least one of the at least one indicative feature can also be presented by the entire first surface dataset and / or the entire further image dataset, if necessary also in the form of annotations. For example, as a support measure it can be provided that the display of the possibly previously analyzed and / or annotated first surface dataset and / or further image dataset is reproduced on the display device together with the instrument / tool ​​or a corresponding marker.

[0022] In this context, it should also be noted that it can be particularly advantageously provided that at least one of the at least one indicative feature describes the presence of a malignant change in the resection at a defined approach distance to the resection surface, the presence of the malignant change being determined by at least partially automatically analyzing the first surface data set and / or a further image data set of the resection registered with the first surface data set. In particular in the case of tumor resections, positions of tumor cells and / or infiltration patterns, for example, which are located on or very close to the surface and have a safety distance that is less than the defined approach distance, can be used in this way as indicative features. Since resections are usually used to extract malignant tissue, such indicative features give clear indications of where a subsequent resection is necessary or where a subsequent resection makes sense. By using corresponding supporting measures of indicative features, these areas can be found significantly easier and more intuitively, since positions on the resection can be converted into positions described in a patient-related coordinate system with the aid of the registration result, and vice versa.

[0023] Advantageously, for displaying the relative positions, an augmented reality device, in particular augmented reality glasses, and / or a display device can be used for reproducing the first and / or second surface data set, wherein instruments and / or further instruments and / or markers indicating their relative positions are faded into the first and / or second surface data set, and / or for reproducing a current image of at least one of the recording instruments and / or at least one further instrument, in particular a laparoscope, wherein at least one indicative feature from the first surface data set is faded into the image. In this context, it may be particularly suitable to display the first and / or second surface data set as a surface image, which may be two-dimensional or, in the corresponding recording modality, three-dimensional. Furthermore, it is particularly preferred within the scope of the present invention that, when using an augmented reality device (AR device), an area in which a malignant change described by at least one of the at least one indicative feature is present is displayed in the extracted area and / or in the current image of the extracted area in a positionally faithful manner.

[0024] In other words, the present invention can realize the most different visualization deformation schemes as supporting measures so that support can be provided at the resection part with respect to indicative features. Therefore, it is conceivable, for example, to display a two-dimensional or three-dimensional surface map, in particular a tissue map, and / or other representations of the resection part, for example in the form of a first surface data set with annotations and / or another data set, which includes a stained area of ​​tissue with malignant changes, in particular tumor cells. Instruments, such as recording instruments and / or digital biopsy tools and / or navigation surgical instruments can be faded in to move virtually on the surface map based on the registration results. This means that, although the instrument is located at a certain position or near a certain position on the remaining tissue surface in the extraction area in real space, the instrument is virtually faded in at the corresponding position on the surface of the resection part, or the corresponding position / position area is marked in another way. Here, two-dimensional and three-dimensional display deformation schemes can also be conceived according to the presence form of the surface data set and other information.

[0025] Further visualization variants can, for example, provide that areas of malignant changes, i.e., areas with tumor cells, are again used as indicative features, but that surface areas on the surface of the residual tissue remaining after extraction of the resected part, which correspond to areas with tumor cells on the surface of the resected part, are displayed as augmented reality and / or on a virtual two-dimensional or three-dimensional surface model or surface map, preferably including a fade-in of a navigation instrument.

[0026] As already described, it can be provided that the first surface data set and / or the second surface data set and / or the further image data set are recorded using an optical imaging modality, in particular a camera and / or optical coherence tomography and / or fluorescence imaging and / or a confocal microscope and / or a laser microscope. In this case, in particular, the first surface data set and the second surface data set can be recorded using different modalities. For example, the surface of the resected part can be scanned over a large area using optical coherence tomography by an in vitro scanning device, while the surface of the remaining tissue in the extraction area formed by extracting the resected part can be recorded point by point, i.e. in very small segments, using an optical confocal microscope. Another conceivable design scheme provides that the surface of the resected part is recorded over a large area using a high-resolution optical microscope, while the surface of the remaining tissue is recorded using a low-resolution laparoscope or surgical microscope.

[0027] Furthermore, it is also conceivable to use a plurality of different modalities for the first and / or second surface data set, respectively. For example, it can be provided that the surface of the resected part is measured using two modalities that are registered with one another, for example a scanning laser microscope and an image from a panoramic camera integrated in the microscope. In this case, for example, a rough registration can be performed based on the first surface data set of the overview camera or a plurality of candidates for a rough registration can be determined, which can then be fine-tuned or verified in or with a microscopic method. The possibility of combining different modalities of this type within the framework of a registration process will also be described in detail below.

[0028] In an advantageous embodiment of the invention, it can be provided that a digital biopsy tool can be used as the recording device, which detects the biopsy segment by segment, in particular with a size of 0.01 to 10 mm. 2 The tissue surface in the extraction area is measured in segments of the resection, wherein the corresponding segments are positioned on the surface of the resected part in the framework of the registration. In this case, in addition to an ex vivo scanning device that allows large-area surface recording of the resected part, a digital biopsy tool is therefore used at the operating site, which is used for recording, for example, microscopically and / or "point by point" by means of optical coherence tomography, wherein, as described, the digital biopsy tool is particularly preferably tracked by means of a navigation system. It should also be noted at this point that the navigation system can generally be designed, for example, as an optical and / or electromagnetic navigation system, which is basically known in the prior art and does not need to be described in more detail here.

[0029] If a plurality of segments are to be recorded, it is particularly meaningful to track the digital biopsy tool by means of a navigation system or the navigation system. It is therefore generally appropriate that the second surface data set comprises measurements of a plurality of segments, wherein the position and orientation (posture) of the digital biopsy tool can be determined by means of the navigation system, and the segments are therefore known in their relative arrangement. With regard to the registration, it can be provided, for example, that first one segment is registered and then a check is made as to whether this also applies to the remaining segments; other registration methods can of course also be selected. For the plurality of segments in the second surface data set, the segments are preferably selected in a substantially uniformly distributed manner, so that ultimately there are digital measurement samples at different areas of the remaining tissue surface, and thus also for the resected part.

[0030] In an advantageous extension of the invention, it can be provided that at least one area is determined by analyzing the first surface data set, which area has surface features that are identifiable in the surface data of the digital biopsy tool, and this area is indicated to the personnel with respect to the resection part. Therefore, it can already be inferred by a pre-analysis process of the first surface data set in which (especially larger) areas there are many surface features that are particularly suitable for registration, so that the corresponding area on the resection part can be pointed out to the person performing the resection in a targeted manner. If this is a larger area, the person can at least roughly associate it with the resection part: approximately where in the extraction area the digital biopsy tool must be used. In this way, further support is provided for the person performing the operation.

[0031] In a very advantageous development of the invention, provision can be made for a quality level describing the reliability of the registration result to be determined within the framework of the registration, wherein if the quality level is below a threshold, a further segment is required to be recorded; and / or at least one verification measurement is performed with the digital biopsy tool to check the information conveyed by the support measure, in particular for checking whether a malignant change is present at the current position of the digital biopsy tool. An iterative process can thus be carried out, which means that, for example, starting with a certain number of segments to be measured (for example 1 to 5 segments), it can be checked after each registration process whether the desired registration quality has been achieved. Quality levels describing the quality of the registration and therefore the reliability of the registration have already been proposed in the prior art and can also be used within the scope of the invention, wherein it is particularly noted that such quality levels are usually components of algorithms or sub-algorithms that are already known per se. If the quality of the registration is not good enough, i.e., for example, the quality level is below a threshold value for the quality level, further measurements can be required with the digital biopsy tool, so that the registration quality is gradually improved in this way.

[0032] Particularly advantageously, the examination can also be performed with the aid of a digital biopsy tool. For example, if the person performing the excision is guided by means of a support measure to a position in the extraction area at which there is a concern that malignant changes will continue to occur on the surface, or generally at which a correspondence with the relevant surface features, in particular indicative features, of the excised part should be established, then the digital biopsy tool tracked by the navigation system can be guided to the corresponding position on the remaining tissue surface based on the support measure, and then it can be verified by measurements performed with the digital biopsy tool whether the indicated position and the correspondence of the surface features have been correctly determined during the registration process. Based on the recordings of the digital biopsy tool, the person can also directly determine whether malignant changes, such as infiltration of tumor cells, are also present on one side of the remaining tissue. In addition, of course, such repeated measurements can also be used to fine-tune the registration based on additional data of at least one segment at the position.

[0033] For example, in the case of malignant changes, a subsequent resection can optionally (outside the method described here) be performed in order to remove any malignant tissue changes that may be present.

[0034] In the scope of the present invention, it is also conceivable that the first and / or second surface data set is composed of sub-data sets recorded from different viewing directions directed to the cutout or extraction area, in particular sub-data sets of a stereo camera. For example, when using the method mentioned in US 2017 / 0105601A1 mentioned at the beginning, a three-dimensional surface model of the surface in the extraction area formed by the cutout or the extraction of the cutout can be generated, for example, by rotating the cutout in front of the stereo camera. Therefore, it can be provided in particular that a surface model describing the surface course is first determined based on the sub-data set, in particular as a polygonal model, which is associated with the image data of the sub-data set as a texture. This means that the camera image data constitutes a texture defined by the surface model, in particular a polygonal surface.

[0035] If the first surface data set consisting of sub-data sets recorded from different viewing directions directed at the resection is not recorded using an ex vivo scanning device, but is recorded using, for example, a laparoscope and / or another recording instrument, it can be provided that the first "first surface data set" is registered with the second "first surface data set" of the ex vivo scanning device, which can be easily achieved using common registration methods due to the recording of the same surface. Thus, for example, the registration of the large-area scan results of the ex vivo scanning device with the 3D surface model of the resection can be performed by image analysis with the aid of suitable analysis algorithms and possible interpolation / smoothing, which is performed, for example, based on macroscopic tissue structures, which are contained both in the scan data of the ex vivo scanning device and in the image data of the corresponding stereo camera of the recording instrument, in particular the laparoscope. Another possibility is to use markers on the resection in order to achieve such registration of multiple first surface data sets, one of which was not recorded using an ex vivo scanning device. However, it is preferred to directly use the ex vivo scanning device that has already been used.

[0036] An extension of the invention provides that when determining a second surface data set showing the entire surface of the extraction area formed by the excision, and when using a digital biopsy tool that measures the tissue surface in the extraction area segment by segment, the segments measured by the digital biopsy tool are registered to the second surface data set, which is carried out in particular with the supportive use of a first surface data set that is registered with the second surface data set. This means that in particular the measurement points of the tracked digital biopsy tool, i.e. the segments, are directly registered with the 3D surface model of the remaining tissue surface. The search for correspondence can be supported by geometrically matching the three-dimensional surface model of the excision part and the remaining tissue surface.

[0037] Furthermore, in a particularly advantageous design of the present invention, it can also be provided that, after recording the first surface data set, it can first be analyzed for whether malignant changes are present in the resected portion within a defined approach distance to the surface of the resected portion, wherein then, if no malignant changes are identified in the resected portion within a defined approach distance to the surface of the resected portion, the process is terminated, so that the recording of the second surface data set is not performed, and correspondingly no alignment is performed. Because if no malignant changes are present on the surface itself or up to a usually defined safety distance, a so-called R0 resection is presented, so that the resection can be considered to have been successfully completed. However, if relevant malignant changes are still identified in the surface area of ​​the resected portion, it can be displayed to the person performing the resection that an R0 resection is not presented and that a subsequent resection must be performed if necessary. The person performing the resection can then, for example, use the recording instrument or its recorded image data in a targeted manner to obtain the second surface data set.

[0038] Within the scope of the present invention, there are two basic methods for realizing the registration of the first surface data set and the second surface data set. Both methods will be described in more detail below and can preferably be used in combination.

[0039] Here, the first method is based on the following basic assumption: in the corresponding surface imaging, there is a similarity between the corresponding surface positions and therefore between the surface features of the resected part and the extracted area, i.e., the remaining tissue, which similarity can achieve pattern-based or image-based registration. Examples of such tissue patterns and tissue characteristics that can achieve a correlation between the surface features of the resected part and the remaining tissue by similarity include cell structure, cell type, cell arrangement, fibrous structure of the tissue, vascular structure of the tissue, spectral reflectivity (color) of the tissue, distribution of fluorescent dyes in the tissue, distribution of malignant changes (e.g., tumor cells) on the visible tissue surface, and action patterns of surgical instruments.

[0040] In a suitable specific design of the alignment process, it can be provided that, in particular when using surface data sets that respectively describe the entire surface, the alignment is performed successively at different resolution levels, which is performed in particular when the tissue structure of the surface is segmented into surface features; and / or, multiple first surface data sets and / or multiple second surface data sets are recorded using different modalities, and all surface data sets are used for the alignment, in particular, in sequence from coarser resolution to finer resolution.

[0041] Thus, in an exemplary embodiment, it is conceivable to first segment macroscopic tissue structures in the surface data set. Such macroscopic tissue structures can be, for example, fat regions and / or muscle fibers. This occurs both in terms of the resected parts and in terms of the extracted regions. Based on these segmentations, for example, a first rough registration can be performed, which can then be improved or fine-tuned in one or more further steps based on finer resolution or structures, in particular up to microscopic tissue structures.

[0042] In other cases, it may also be useful to carry out the registration continuously or even simultaneously at a number of size scales, since, in particular in the plane of very small, i.e. microscopic, surface features, the registration may also fail at some locations. This is because, during the excision, a smooth cut is usually not made between the excised part and the remaining tissue with a sharp scalpel, but rather the intermediate thin tissue layer may be cut away piecemeal, for example with a suction device. This may accordingly result in very small structures having no analogue on the respective other side.

[0043] For the registration, the resolutions of the first and second surface data sets to be registered can also be adapted to one another, in particular by digital scaling, or registration data sets of different resolutions can be generated based on the first and second surface data sets in general, and then the first and second surface data sets can be registered to one another respectively, successively or even simultaneously, by using registration data sets of the same resolution. However, if a plurality of first surface data sets and / or a plurality of second surface data sets are used, first or second surface data sets of different resolutions can basically already exist, so that, for example, a coarser-resolved surface data set can be considered first and then a finer-resolved surface data set. For example, it is conceivable that a high-resolution first surface data set is obtained by an ex vivo scanning device, whereas a coarsely-resolved second surface data set is obtained with respect to the extraction area, for example by a laparoscope camera, and a high-resolution segment is obtained using a biopsy tool, in which case a coarse registration can first be performed based on the coarser-resolved second surface data set, and then an attempt can be made to locate the segment, thereby improving the quality and accuracy of the registration.

[0044] As already mentioned, within the scope of the present invention, cell structure and / or cell type and / or cell arrangement and / or cell clusters and / or fibrous structure of the tissue and / or vascular structure of the tissue and / or spectral reflectivity of the tissue and / or the distribution of fluorescent dyes in the tissue and / or the distribution of malignant changes on the visible tissue surface and / or the trajectory of a surgical instrument, in particular a scalpel and / or a suction device and / or an energy instrument can be used as surface features for alignment.

[0045] In a particularly preferred extension of the invention, it can be provided that, in particular when using at least one filtering algorithm and / or smoothing algorithm and / or edge highlighting algorithm, surface features in different resolution areas of the surface data set are detected and / or combined into a fingerprint data set, which are compared for registration. In a preprocessing step, for example, tissue structures on different size scales can be extracted by filtering algorithms, smoothing algorithms and / or edge highlighting algorithms (edge ​​sharpening algorithms) and / or frequency analysis. Specific surface features are extracted from the preprocessed data, and in a further step, spatial registration can be performed based on these surface features. The entirety of the surface features or the spatial arrangement of the surface features, in particular also in a sub-area / segment of the surface, can be referred to as a fingerprint, so that it is therefore particularly preferred within the scope of the invention to determine the fingerprint data set, in particular on different size scales.

[0046] Then in order to carry out registration between the tissue surface of the excised part and the tissue surface of the remaining tissue, for example, the fingerprint data set of the first surface data set can be compared with the fingerprint of the second surface data set, wherein the spatial correlation can be determined based on the maximum consistency and the registration is determined thus. Here, in the scope of the present invention, when comparing the fingerprint data set, it can also be imagined that a plurality of corresponding relationship candidates with higher surface feature consistency are output. If this also occurs in particular on different size scales, then the registration result can be derived in a robust manner. The hash value of the tissue segment calculated by the correlation of the fingerprint or therefrom can save computing power relative to the direct comparison based on the image.

[0047] In particular, when using a digital biopsy tool, it can be provided that, for the purpose of registration, a fingerprint data set of the size of the segment and / or nine times the size of the segment is generated in a second surface data set of the surface of the extraction area formed by the excision, which is reproduced only segment by segment, and is compared with the fingerprint data set of the segment. Thus, for example, the fingerprint measured segment by segment on the remaining surface (especially using the digital biopsy tool) can be compared with the fingerprint of the determined excision part. In this case, it is appropriate to use similar size scales in multiple steps in order to first determine in which larger area the segment may be present, so that the segment can then be located more accurately there. By repeating the process of multiple segment-by-segment measurements on the remaining tissue surface in the extraction area, the correct registration result can then be derived by geometric matching of different correspondence candidates.

[0048] When using the fingerprint dataset, a certain degree of abstraction is performed, so that the processing is particularly suitable for different modalities. It can therefore be provided that when using the fingerprint dataset, the first and second surface datasets are recorded using different modalities. For example, the first surface dataset can be recorded using optical coherence tomography, and the second surface dataset can be recorded using optical confocal microscopy, etc.

[0049] According to another particularly advantageous extension of the present invention, an artificial intelligence registration algorithm, in particular a deep neural network (DNN), is used when registering the surface data set, and the artificial intelligence registration algorithm has been trained using the annotated surface data set as training data. Here, as described above, the artificial intelligence registration algorithm can, for example, generate a fingerprint data set and / or compare the fingerprint data set. In a design, the artificial intelligence registration algorithm, in particular the DNN, can be trained according to the annotated example data set, so that the association of the fragment of the tissue surface of the excised part and the extraction area is carried out according to the image information / surface features, and the registration is performed on this basis. Preferably, the artificial intelligence registration algorithm can learn which surface features are beneficial for registration by itself, which means that the registration algorithm learns to establish a suitable tissue fingerprint data set, and then uses it for registration. Here, in addition, the preprocessing steps already mentioned can be performed, that is, in particular the application of filtering algorithms, smoothing algorithms and / or edge highlighting algorithms and / or frequency analysis, wherein, however, in order to extract surface features, specifically, especially the fingerprint data set, a properly trained artificial intelligence registration algorithm is then used, and then a comparison can be made based on this.

[0050] In particular, in the case of using an artificial intelligence registration algorithm, the following sequence of steps can be given:

[0051] - preprocessing of the first and second surface data sets, in particular by using at least one filtering algorithm and / or at least one smoothing algorithm and / or at least one edge highlighting algorithm and / or a frequency analysis,

[0052] - using an artificial intelligence registration algorithm to extract a fingerprint data set from the pre-processed first and second surface data sets, and

[0053] - Comparing the fingerprint data sets to find corresponding fingerprint data sets, thereby finding corresponding surface features, thereby determining a registration result.

[0054] As already described, the basic assumption of the similarity of corresponding surface features can lead to errors, which means that the registration is performed (purely) on the corresponding surface features in the surface data set recorded after the separation of the resected part from the remaining tissue. The disadvantage of this processing method is that due to the use of surgical instruments, such as energy instruments for laparoscopic examinations and / or surgical instruments that absorb tissue, a clear tissue gap is usually formed between the tissue surface formed by the resection and the surface of the resected part. Although tissue properties with long effective distances (see the examples mentioned above) can still be associated, this becomes more difficult in larger and continuous tissue gaps and in tissue areas that have been burned over a large area by the energy instrument. In particular, it is known that the remaining tissue surface in the extraction area must then be burned again using the energy instrument. In order to stop the bleeding permanently. Difficulties arise here when using image-based methods, which can perform registration by similarity of surface structures. Therefore, the second basic method of the invention, which is described in more detail below, provides a registration method or a registration aid that can relate corresponding locations on the resected part and the remaining tissue to each other even though the two surfaces do not look very similar at the corresponding locations, either microscopically or macroscopically.

[0055] In order to achieve this, a particularly advantageous embodiment of the invention provides that, to support the registration, a correspondence data set containing correspondence pairs of surface features that correspond to one another is generated by:

[0056] - during the cutting of the cut-out part, continuously recording a monitoring image of the current working area by means of the said or another recording device,

[0057] - when the cutting is detected, detecting the surface features of the surfaces formed respectively on both sides of the cut by automatically analyzing the corresponding monitoring images,

[0058] - correlating surface features that geometrically correspond to each other with respect to the cutout to produce corresponding pairs, and

[0059] - storing each assignment pair in an assignment data record together with feature information describing the surface feature of the assignment pair, the feature information comprising in particular a partial image of at least one monitoring image showing at least the surface feature.

[0060] The monitoring images can preferably be individual frames of a monitoring video, which is recorded during the resection of the resection using the recording device or another recording device. The basic idea of ​​this particularly advantageous process is therefore to generate a correspondence data set between the resection and the remaining tissue and the respectively stored feature information from a monitoring video or monitoring image of the resection, which can be recorded, for example, by a laparoscope or an operating microscope. The correspondence data set is then used to register the position on the surface of the resection with the corresponding position on the surface of the remaining tissue in the extraction area formed by the extraction of the resection.

[0061] Preferably, the monitoring image can be recorded with a recording device or with a camera of the recording device, in particular a laparoscope and / or a surgical microscope and / or a lighting device. Preferably, the monitoring image can be recorded with a stereo camera, wherein the three-dimensional surface structure of the formed surface is determined and used for detecting and / or describing and / or determining the position of at least one surface feature. Although a monoscopic, i.e. two-dimensional video or a two-dimensional monitoring image is in principle sufficient for the described process, a stereo monitoring video or generally a stereo monitoring image can be advantageous for increased robustness.

[0062] It should also be noted at this point that the actual implementation of the excision is not part of the method described here. The analysis of the monitoring images recorded during the excision of the excision part can in principle be carried out during the excision, but preferably only after the excision is completed. This means that after the excision is completed, for example, a monitoring video and / or a series of individual monitoring images can be analyzed in a targeted manner to generate a correspondence data set. Different designs can be provided as surgical tools or surgical instruments for performing the excision, wherein the surgical tools or surgical instruments can in particular cut away and / or separate tissue and / or can cauterize tissue by applying energy and / or heat.

[0063] The correspondence data set can be used in different ways during the registration. On the one hand, it is conceivable, as described in more detail below, that at least one second surface data set and, if necessary, an additional first surface data set are already formed by the correspondence data set, or that the registration is established relatively directly. On the other hand, however, the correspondence data set can also be a registration aid in that it ultimately provides additional similarity criteria for seemingly different surface features, so that despite completely different appearance images in the imaging modality of the first and / or second surface data set, similarities can be determined based on the correspondences stored in the correspondence data set.

[0064] In a particularly advantageous extension of the invention, it can be provided that, when a new correspondence pair is generated, by detecting at least one surface feature and / or reference feature of another correspondence pair in a monitoring image showing the surface features of the new correspondence pair, a relative position relative to at least one other correspondence pair already contained in the correspondence data set and / or relative to a reference feature that is not part of the correspondence pair is determined and stored in the correspondence data set. In this way, the relative arrangement of the surface features of the respective correspondence pairs relative to each other is known, so that the known relative arrangement can also be used in the process of registration. In particular, the possibility of creating a representation of the corresponding surface of the remaining tissue in the resection part and the extraction area is provided, which will be described in more detail below. Here, the relational feature does not necessarily have to be a surface feature of another correspondence pair, but a relation relative to one or more reference features can also be established. The reason for this is that the number of robustly identifiable correspondence pairs of corresponding surface features is relatively small. Therefore, in order to define the relative position, "one-sided" surface features can also be identified, which can then define the relative position, at least as long as a bridge to the already defined correspondence pair can be established. The correspondence of the surface features on one side can then be geometrically interpolated accordingly on both sides, i.e. on the surface of the resected part and the remaining tissue. In addition, the reference feature can also be used to determine whether a working step / cut has already occurred. Thus, such a reference feature in principle also allows tracking the movement of the tissue.

[0065] Here, it should also be generally noted at this point that the design presented here is based on the assumption that almost all surgical steps that contribute to the separation of the resected part from the remaining tissue are performed under observation via the monitoring image. In practice, this condition is very well met, since the personnel performing the resection really want to see what they are doing during their sometimes critical actions; moreover, bleeding on the two surfaces formed should be avoided or prevented as much as possible.

[0066] As already described, it is important for detecting or identifying new surface feature correspondences that the geometric associability with respect to the incision or cut-off means that, in other words, certain geometric criteria must be met. Therefore, the corresponding surface features should be located on two different sides of the current or just past action area of ​​the surgical instrument. In addition, the corresponding surface features should also geometrically meet the following condition: they may have been separated by the instrument in the current or sufficiently new previous frame. A certain inaccuracy, in particular inaccuracy with respect to the registration only on this basis, is acceptable, because in subsequent excisions, a large amount of tissue will usually be cut off around the identified area. Further geometric criteria may require a similar spacing from the current incision edge and / or a symmetrical relative to the incision edge, where the surface of the excised part formed at the current incision edge and the surface of the remaining tissue are therefore delimited from each other.

[0067] Furthermore, it should be noted that, in particular when reference features are to be avoided as much as possible, when it is not the beginning of the excision, the new surface features of the new surface pair should preferably not be too far away from the previously identified surface features of the old corresponding relationship pair, in particular so that the previously identified surface features are still located in the field of view of the corresponding recording instrument, i.e. visible in the same monitoring image or at least can be linked thereto. Otherwise, as mentioned, reference features can be used.

[0068] It can be provided that the surface features on the resection side of all correspondence pairs with their corresponding arrangement form an additional first surface data set, and the surface features on the extraction area side of all correspondence pairs with their corresponding arrangement form a second surface data set (or one of at least one second surface data set), or that the surface data set is determined therefrom. By determining the relative positions of the correspondence pairs, a mapping of the corresponding surface can ultimately also be carried out during the formation of the corresponding surface. Accordingly, a surface data set can also be derived from this mapping, wherein at least the corresponding positions of the surface are also known point by point through the correspondences mapped by the correspondence pairs.

[0069] In this case, it is advantageously provided that a surface model of the surface of the extraction area formed by the excision and a surface model of the excision part are created based on the correspondence pairs and their respective relative positions, and that at least a part of the feature information and / or a part of at least one monitoring image showing the corresponding surface area are associated with the surface model in a position-accurate manner in order to determine the corresponding surface map, so that correspondence information that is associated with the position on the surface map is given by the correspondence pairs. The surface map thus formed can, for example, already constitute the further first and / or second surface data set. Accordingly, the correspondence information can be part of the registration result or, in particular, as will be explained below, in an extended manner, the registration result.

[0070] Thus, when analyzing the monitoring image, a textured surface map is generated or updated for a new correspondence pair, more precisely, a textured surface map of the resected part and the remaining tissue in the extraction area, respectively. In this case, roughly speaking, a surface pattern / surface image is extracted from the monitoring image and associated with the position of the corresponding surface features or the intermediate space between the surface features. Then, in connection with the description of the correspondence given at the beginning, the surface features of the correspondence pair have coordinates a on the surface map of the resected part and the remaining tissue. R or b V In a further development of this, it can also be provided that a plurality of parts of different monitoring images showing the corresponding surface area are averaged and used as texture. This means that a textured surface of a surface model / surface map can also be created using a plurality of monitoring images, in particular a plurality of frames of a monitoring video, for example by averaging a plurality of monitoring images. The textured surface map can be created two-dimensionally or, in particular, three-dimensionally when using a stereo camera.

[0071] In this embodiment, therefore, the correspondence pairs with the surface features, the textured surface map and possibly further information are stored together in a data memory.

[0072] Furthermore, in a particularly preferred embodiment of the invention, it can be provided that, in order to expand the correspondence information given by the correspondence pair, in the region of the surface map between the surface features of the correspondence pair, further corresponding positions of the surface map are determined by interpolation, in particular a correspondence map and / or a correspondence table suitable for the complete surface map is determined. With the processing method described so far, a limited number of robust surface correspondences can already be identified between the cutout and the extracted region. It is therefore appropriate to appropriately interpolate (or extrapolate) the correspondences so that as many positions as possible (preferably each position) on the surface of the cutout can be identified. R The corresponding position b on the surface of the remaining tissue in the extraction area V In this way, a complete registration map can be finally obtained as the correspondence information from the analysis process of the surveillance image.

[0073] Suitably, in order to locate the indicative feature of the resected portion on the surface of the extraction area, correspondence information is used, the position of the indicative feature being known on the surface of the resected portion. Here, firstly, the position is usually known in a first surface data set or the first surface data set and / or another image data set that does not correspond to the surface map of the resected portion. Then, the position in the surface map of the resected portion can be determined by registering the surface map of the resected portion with the first surface data set and / or another image data set that does not correspond to the surface map of the resected portion. The corresponding registration can obviously be performed more easily, on the one hand because the same surface is involved, and on the other hand because the surface features suitable for registration are already known as part of the correspondence pair (and possible reference features). In particular, the first surface data set or another image data set that does not correspond to the surface map can be recorded using an ex vivo scanning device. Accordingly, the indicative feature can be, in particular, an area where a malignant change of the tissue is too close to the surface of the resected portion.

[0074] In other words, for example, surface patches found in histology, for example as tumor infiltrations, can be registered image-based to the surface map of the resected part. For example, a microscope image of the surface can be correspondingly scaled down and then spatially associated with the surface map image-based. The result is a position a on the surface map R In this regard, the corresponding relationship information can be used to determine the R The corresponding extraction area position b V .

[0075] In principle, it is conceivable, in particular in the case of tracking of the or another recording device for recording the monitoring image, in particular in stereoscopic viewing, that the monitoring image and thus the surface image of the extraction region are already present in a patient-related coordinate system. If this is not the case, a registration (which can be performed significantly more robustly in this case) of the surface image of the extraction region with the or another second surface data set which does not correspond to the surface image of the extraction region can be performed. Thus, for example, it is possible to (if necessary limited to around the position b V An image-based registration is performed between the texture of the surface map of the remaining tissue (in the region of the extraction) and the current image (further in particular the monitoring image) of the and / or further and / or also further recording instrument, for example the current video image of the laparoscope. This makes it possible to display the surface portion of the remaining tissue corresponding to the surface portion of the tumor infiltration of the resected part, for example as an overlay, augmented reality or marking in the navigation system, as has already been described in detail with respect to the supporting measures. As an alternative to the image-based registration presented as a supporting measure, as has already been described, a registration can alternatively also be performed by means of the navigation system, wherein the entire surface map of the remaining tissue in the extraction region is registered to the geometry in the operation, i.e. in particular to a patient-related coordinate system.

[0076] Preferably, the detection of the implemented cut is performed at least partially based on the image in the monitoring image, in particular by tracking the already detected surface features and / or the surgical instruments visible in the monitoring image (in particular with regard to the surgical instruments in contact with the tissue) and / or based on additional information temporally associated with the monitoring image. Advantageously, it is thus determined using all available information whether the surgical instrument or surgical tool is currently separating or removing or processing tissue, or whether the surgical instrument is currently not having any effect on the tissue. This determination can be performed for each monitoring image, in particular for each frame in the monitoring video, so that the identification of new corresponding surface features is only performed if the surgical instrument has an effect on the tissue in the current monitoring image or in the monitoring image just recorded, which, for example, results in a further separation of the resected part from the tissue by cutting off or cutting. In particular, the surgical instruments visible in the monitoring image can thus be used as an indication of the significance of the generation of a new correspondence pair. In addition, the instruments can also be easily detected in the corresponding image processing.

[0077] Advantageously, additional information can be determined, which describes the movement of the surgical instrument determined by the navigation system or the navigation system, and / or the energy supply to the surgical instrument, in particular triggered by the actuation of a foot pedal, and / or the workflow steps of the surgical workflow. In particular, in this case it can be provided that the monitoring images, in particular the monitoring video, are annotated using an external information source or manually. For example, as is known in principle in the prior art, an optical navigation system can be used that tracks the progress of the resection. In addition, support devices have also been proposed that are able to identify the resection phases with the aid of a workflow analysis. In particular, such a workflow analysis itself can also analyze the monitoring images so that, for example, specific parts of the resection can be identified from the monitoring video of the operation.

[0078] It should also be noted with regard to the workflow steps that this additional information can also be used to make a basic preselection of the corresponding parts of the surveillance images or surveillance videos to be analyzed. This reduces the overhead since only the surveillance images showing actually relevant material then have to be analyzed.

[0079] Another relevant source of information for possible annotation of monitoring images or monitoring videos is an operating device, in particular a foot pedal, by which the surgical instrument can be supplied with electrical energy, since it is assumed that the energy supply is activated by the operator only when tissue treatment is to take place.

[0080] As already described, the surface features of the new correspondence pair are preferably selected such that in the corresponding imaging modality of the surface data set and / or at least one other image data set, the surface features can be detected with a reliability exceeding a threshold value. It is then particularly suitable that, starting from the surface map, which is also a surface data set, an image-based registration should be made to, in particular, another surface data set and / or another image data set of a recording instrument used in the extraction area, in particular an ex vivo scanning device and / or. The following fact is used here: in the frame of the resection, it is usually known which modality is used to work. Similarly, the physical imaging properties of these imaging modalities are also known, so that it is also known which imaging modality can be used to identify which surface features with what robustness, so that the surface features of the correspondence pair can be selected accordingly in a predictable manner.

[0081] In a particularly advantageous embodiment of the invention, provision can be made that, during the resection time period, surface features of the correspondence pair are tracked in the monitoring image as long as they are contained in the monitoring image, wherein for a determined subsequent change of the surface features caused by the or another surgical instrument, in particular an energy instrument, the feature information is adjusted to describe the changed surface features; and / or provision is made that for a unilateral surface change determined by the automatic analysis, in particular by image analysis of the change area in the monitoring image recorded before the change and / or based on information about the relative position, the changed surface features of the correspondence data set are found and the relevant feature information is adjusted to describe the changed surface features. Here, the term "unilateral" is understood to mean either only the surface of the resected part or only the surface in the extraction area, i.e. the surface of the remaining tissue. During the resection, a part of the tissue surface of the future resected part or the future remaining tissue may be processed again later, for example by cauterization with an energy instrument to stop bleeding. Such a position or change area can be identified in the following way: the surface features located around the instrument are either all located on the side of the resected part or all located on the side of the extraction area. In this case, the surface texture changes at the corresponding position, so that if there is a surface feature of the correspondence pair there, the corresponding feature information can be adjusted. With regard to the particularly preferred generation of the surface map, the texture in the change region derived from the monitoring image can then also be appropriately updated. In general, in this way, by noting the subsequent changes, it is achieved that the correspondence pair and the surface map are also particularly preferably kept up to date. In other words, therefore, especially whenever a unilateral subsequent treatment of the tissue occurs in the change region, the surface texture of the feature information or the surface map is adjusted and updated again. However, it should be noted that the corresponding location / position of the correspondence pair remains unchanged. Compared with the alignment of the first and second surface data sets based purely on images, there are also special advantages in this: in particular, similarities can be eliminated in the case of such subsequent unilateral changes to the tissue surface, and the image recorded after the excision is completed may fail in a purely image-based method.

[0082] In a general manner, a particularly advantageous embodiment of the method can also be designed as a method for navigation support of a person performing a resection on a patient after the extraction of the resection for navigating in the extraction area relative to the resection, characterized by the following steps:

[0083] - generating a correspondence data set comprising correspondence pairs of surface features that correspond to one another by:

[0084] - During the cutting of the cutout part, the monitoring image of the current working area is continuously recorded by the recording instrument,

[0085] - when the implementation of the cut is detected, the surface features of the surfaces formed respectively on both sides of the cut are detected by automatically analyzing the corresponding monitoring images,

[0086] - correlating surface features that correspond to each other geometrically with respect to the cutouts to produce corresponding pairs, and

[0087] - storing each correspondence pair in a correspondence data record together with feature information describing the surface feature of the correspondence pair, the feature information in particular comprising a sub-image of at least one monitoring image showing at least the surface feature,

[0088] -Wherein, in the case of generating a new correspondence pair, in a monitoring image showing surface features of the new correspondence pair, by detecting at least one of the surface features of the other correspondence pair and / or a reference feature, the relative position relative to at least one other correspondence pair already contained in the correspondence data set and / or relative to a reference feature that is not part of the correspondence pair is determined and stored in the correspondence data set, and a surface model of the surface of the extraction area formed by the resection and a surface model of the resection portion are created based on the correspondence pairs and their corresponding relative positions, and in order to determine the corresponding surface map, at least a part of the feature information and / or a part of at least one monitoring image showing the corresponding surface area is associated with the surface model in a positionally faithful manner, so that correspondence information that is correlated with the position on the surface map is given by the correspondence pair.

[0089] - recording a first surface data set of the resected part using an ex vivo scanning device,

[0090] registering the first surface data set of the resected portion with the surface map of the resected portion (which can be regarded as a further first surface data set) and thus with the surface map of the extracted region (which serves as a second surface data set) using the correspondence information, and

[0091] Using the registration, at least one supporting measure is performed which supports navigation in the extraction region relative to the resection part.

[0092] In addition to the method, the invention also relates to a support system for navigation support of a person performing a resection on a patient after the resection has been extracted, in order to navigate in the extraction region relative to the resection, the support system having a control device designed to carry out the method according to the invention. All the embodiments of the method according to the invention can be transferred analogously to the support system according to the invention, so that the advantages already mentioned can also be achieved by these embodiments.

[0093] The support system can have, in particular, various instruments, in vitro scanning devices, navigation systems, etc. The instruments can include, for example, recording instruments and / or surgical instruments. In addition, the support system can also include a display device, in particular for outputting diagrams within the framework of support measures, as well as an augmented reality device that can be conceived in addition or alternatively. In order to control all these conceivable components, the control device can be designed accordingly. In particular, the control device can have at least one processor and at least one storage device (data memory). Corresponding functional units can be provided in the control device in order to implement the steps of the method according to the invention.

[0094] For example, the control device can have at least one interface for receiving the first and / or second surface data set, a registration unit for registering the surface data set, and a support unit for performing at least one support measure, wherein, according to the description of the method according to the invention, further functional units may of course also be present for other design options.

[0095] The computer program according to the invention can be loaded directly into a storage unit of a control device of a support system, for example, and has program means for carrying out the steps of the method according to the invention when the computer program is executed in the control device of the support system. The computer program can be stored on an electronically readable data carrier according to the invention, which thus comprises electronically readable control information stored thereon, said electronically readable control information comprising at least one computer program according to the invention and being designed such that, when the data carrier is applied in the storage device of the support system, said electronically readable control information carries out the method according to the invention. The data carrier is preferably a non-transitory data carrier, for example a CD-ROM. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] Further advantages and details of the invention can be found in the following exemplary embodiments and in the accompanying drawings, in which:

[0097] Figure 1 An embodiment of a support system according to the invention is shown,

[0098] Figure 2 A schematic diagram showing the correspondence between the position on the resected portion and the position on the remaining tissue surface,

[0099] Figure 3 shows a flow chart of a first embodiment of the method according to the present invention,

[0100] Figure 4 shows a flow chart of a second embodiment of the method according to the present invention,

[0101] Figure 5shows a flow chart of a third embodiment of the method according to the present invention,

[0102] Figure 6 shows a flow chart for creating a correspondence data set based on a monitoring image,

[0103] Figure 7 shows a schematic monitoring image at the beginning of resection,

[0104] Figure 8 shows a schematic monitoring image at a later time point during the resection,

[0105] Fig. 9 shows a schematic monitoring image at an even later point in time during the resection,

[0106] Fig.10 A flow chart for using a correspondence data set is shown, and

[0107] Fig.11 The functional structure of the control device of the inspection system according to the present invention is shown. DETAILED DESCRIPTION

[0108] Below will be described in more detail an embodiment of the present invention, wherein as an example of resection, the removal of a tumor from an organ of a patient is considered. In such tumor resection, it is of particular concern whether a so-called R0 resection has been performed, and therefore whether there is still malignant tissue left in the patient, i.e., tissue that has been infiltrated by tumor cells. Therefore, it is suitable for the person performing the resection to not only be able to obtain histological results: whether there are still tumor cells or other malignant remnants on the surface of the resected part or near the surface, but also that this information can be spatially associated in the extraction space accordingly, i.e., in particular, on the surface of the remaining tissue formed by the resection. It should be noted at this point that, in the framework of the description here, the description of the presence of malignant changes in the resected part at a defined approach distance to the surface of the resected part should of course also include that the malignant changes are directly present on the surface of the resected part.

[0109] Figure 1 The schematic diagram of a support system 1 according to the invention is shown, which is suitable for carrying out a tumor resection and a method according to the invention.

[0110] A patient 3 is shown on an operating table 2 from which a resection 5 has been extracted from an organ 4. Accordingly, a tissue surface 7 formed by the resection remains in the extraction area 6. In the support system 1, a variety of different instruments can be provided for the person performing the resection, of which only a surgical instrument 8, a first recording instrument 9 in the form of a digital biopsy tool 10 and a second recording instrument 11 in the form of a laparoscope 12 with a stereo camera are shown purely by way of example. All instruments 8, 9 and 11, as well as possibly other instruments not shown here, have markers 13 for the current optical navigation system 14, which currently also shows a camera 15. For example, these cameras can be mounted on the ceiling. As an alternative or supplement to the optical navigation system 14, an electromagnetic navigation system can also be used. Importantly, the current position and orientation, i.e., posture, of all instruments used can be determined in a patient-related coordinate system by the navigation system 14. In particular, the recording position is also known, at which the image of the imaging device (e.g., camera and / or OCT sensor) of the recording instruments 9, 11 is formed.

[0111] In this case, the laparoscope 12 can have, for example, a stereo camera, with which monitoring images can also be recorded during the operation. The digital biopsy tool 10 can have, for example, a microscope and / or an OCT sensor in order to record, for example, biopsies with a size of 0.01 to 10 mm. 2 Smaller fragments within the range.

[0112] Furthermore, the support system 1 comprises an ex vivo scanning device 16, which can also be referred to as a digital histology device. The resection part 5 is already positioned in the ex vivo scanning device 16, so that the surface of the resection part 5 can be scanned over a large area there, for example, in order to identify surface areas in which malignant changes, in particular tumor cells, are still present.

[0113] Furthermore, the examination system 1 has a control device 17, which is preferably connected in communication with all other components, in particular the ex vivo scanning device 16, the instruments 8, 9 and 11, the navigation system 14 and other components. In particular, the control device 17 can also access a display device 18 and an augmented reality device 19, here augmented reality glasses 20, which can be controlled for support measures.

[0114] In particular, the control device 17 is designed to carry out the method according to the invention, which will be described in more detail in the following exemplary embodiments.

[0115] like Figure 2As more clearly indicated in the figure, the purpose of the method according to the invention is to establish a registration indicated by the arrow 21, which makes it possible to find again on the remaining tissue surface 7, for example, a surface area / position 22 on the resected part 5 identified as infiltrated by tumor cells in the surgical geometry, in particular in a patient-related coordinate system, so that the corresponding position 23 on the remaining tissue surface 7 in the extraction area 6 is automatically found, and the person performing the resection can find this position 23 as easily as possible by suitable support measures.

[0116] Figure 3 A flow chart of a first embodiment of the method according to the invention is shown. Here, in step S1, the surface of the resection 5 is scanned in an ex vivo scanning device, which means that a first surface data set of the resection 5 is recorded by the ex vivo scanning device. In step S2, this first surface data set of the resection 5 is automatically analyzed to check whether there are malignant changes in the resection 5 in a defined approach distance to the surface of the resection 5, in particular in a maximum safety distance from the surface of the resection 5. The positions where this occurs are marked as indicative features. As an alternative to the first surface data set or in addition to the first surface data set, other image data sets of the ex vivo scanning device can also be analyzed.

[0117] In step S3, it is checked whether a malignant change has been detected, in the present case by infiltration of tumor cells, wherein if this is not the case, the method is terminated in step S4. However, if indicative features are present, a plurality of segments of the remaining tissue surface 7, for example 1 to 5 segments, are recorded in step S5 using the digital biopsy tool 10 as a second surface data set of the extraction area 6. It is recommended to distribute the segments as evenly as possible; however, it is also possible to draw a rough conclusion from the analysis of the first surface data set: where there are a large number of well-recognizable surface features, to output a recommendation to the person performing the resection. The relative positions of the recorded segments of the remaining tissue surface 7 are known based on the navigation system 14.

[0118] In step S6, in this first embodiment, the first surface data set is now registered with the second surface data set by finding corresponding surface features of the surface of the resected portion 5 and the remaining tissue surface 7. Because the microscopic and macroscopic cell structures or tissue structures at the location of the resected portion and at the corresponding location of the remaining tissue surface 7 can have strong similarities, because they are originally connected together. Examples of such similarities are patterns of cell arrangement, tissue boundary layers, blood vessels, tissue density and cell types and, if necessary, infiltration patterns of tumor cells. Take advantage of the fact that candidates for corresponding surface features including geometric orientation are identified by algorithmically matching the image data of a small area of ​​the digital biopsy tool 10 measured segment by segment with the image data of a large area of ​​the ex vivo scanning device 16.

[0119] In this first embodiment, in the registration in step S6, an artificial intelligence registration algorithm is also used in particular. Here, in the first sub-step, preprocessing is performed, for example, by using at least one filtering algorithm and / or at least one smoothing algorithm and / or at least one edge highlighting algorithm and / or frequency analysis to extract tissue structures on different size scales. The preprocessed first and second surface data sets are then transmitted as input data to the artificial intelligence registration algorithm. Here, DNN can be used in particular. The DNN is trained according to the annotated example data set, and surface features that are conducive to registration are extracted from the preprocessed surface data set, and these surface features are composed of fingerprint data sets. The fingerprint data set describes the overall surface features and the spatial arrangement of the surface features in at least one sub-region of the surface described by the surface data set. For example, with respect to the second surface data set recorded using a digital biopsy tool, a fingerprint data set can be determined for each segment; of course, it is also conceivable that a subdivision is also performed within the segment and a smaller fingerprint data set is established for the sub-region of the corresponding surface.

[0120] The fingerprint data sets of the first surface data set and the second surface data set are then compared with one another, of course at the corresponding size scale, in order to find possible candidates for correspondence, in particular by applying a similarity criterion. In this case, the spatial association can be output based on the greatest consistency, but optionally also a plurality of correspondence candidates with a higher consistency of the surface features can be output. By repeating this process in a plurality of segment-by-segment measurements of the remaining tissue surface 7, a correct registration is then established by geometric matching of the different candidates.

[0121] Here, in particular also with regard to the further embodiments to be discussed, which have a second surface data set covering a larger part of the remaining tissue surface 7, it should be noted that implementation at different size scales is particularly advantageous, in particular, because the registration on the plane of the microscopic cell arrangement fails at many locations due to the removal of a thin tissue layer located between the surface of the cut-out part 5 and the remaining tissue surface 7. Accordingly, it is possible to work at multiple size scales. For example, fingerprint data sets can be used at a size scale of microscopic cell structures, such as 1 to 100 cells, at a size scale of several hundred cells, such as 100 to 1000 cells, and at a macroscopic size scale.

[0122] In particular, by using such a fingerprint data set, this type of image-based registration can also be robustly applied to first and second surface data sets of different modalities, for example using optical coherence tomography for the first surface data set and scanning the remaining tissue surface segment by segment with the aid of an optical confocal microscope. In a further example, the surface of the resected part 5 can be recorded over a large area with a high-resolution microscope, and the surface of the remaining tissue surface can be recorded by a laparoscope or surgical microscope with a lower resolution. In addition, it is also suitable in other types of registration to use an artificial intelligence registration algorithm that has been trained accordingly for a combination of modalities when using different modalities.

[0123] After step S6 is completed, a registration result is therefore present, which makes it possible to associate positions on the surface of the resection part 5 , whose position is known in the coordinate system of the first surface data set, with positions on the remaining tissue surface 7 .

[0124] Since these segments were recorded with the digital biopsy tool 10 tracked by the navigation system 14 , the locations of the positions are also known in the patient-related coordinate system of the navigation system 14 .

[0125] In step S7 , this is used for various possible supporting measures.

[0126] Thus, in step S7, as a support measure, a two-dimensional or three-dimensional tissue map or other representation of the resected part 5 can be displayed on the display device 18, wherein in particular the representation can be derived from the first surface data set and / or a further image data set of the ex vivo scanning device 16. In the representation showing the surface of the resected part 5, the indicative feature determined in step S2, i.e. the area of ​​malignant change, is highlighted in red. At the same time, the registration result of step S6 is used to display the relative position of the digital biopsy tool 10 and / or at least one further instrument, for example a surgical instrument 8, relative to the indicative feature and thus relative to the corresponding position on the remaining tissue surface 7. In other words, based on the registration result, it is determined which position on the remaining tissue surface 7 corresponds to the indicative feature; the relative position of the respective instrument relative to this position is determined by means of the navigation system 14, and based on this relative position, the relative posture of the instrument relative to the indicative feature is visualized using the representation of the resected part 5. In other words, although the instrument is located at a certain position or near a position on the remaining tissue surface in real space, the instrument is virtually faded in at a corresponding position on the surface of the resected portion 5, or the corresponding position / position area is marked in other ways.

[0127] In a further variant of this supporting measure, the surface area of ​​the remaining tissue surface 7 corresponding to the area with tumor cells on the surface of the resected part 5 can be displayed as augmented reality or on a virtual two-dimensional or three-dimensional surface model, preferably together with a fade-in of the navigated, i.e. tracked, instrument.

[0128] Optionally, a check is performed in step S8. Based on the support measures, the user can, for example, move the digital biopsy tool 10 to positions on the remaining tissue surface 7, which are displayed as areas that may have tumor cells. By measuring with the digital biopsy tool, the user can verify whether the displayed correspondence has been correctly determined and, if necessary, can directly determine from the image of the digital biopsy tool whether tumor cells have also infiltrated the remaining tissue side. Then, if necessary, a subsequent resection can be performed (outside the method described here).

[0129] In an optional step S8, the inspection measurements can be fed back into step S6 to improve the registration. It should also be noted at this point that steps S5 and S6 can also be repeated iteratively, for example when it is determined in step S6 that the quality of the registration result is too low. It is then conceivable to record surface data of a second surface data set using the digital biopsy tool 10 in a repetition of step S5 for at least one further segment.

[0130] Figure 4A second embodiment of the method according to the present invention is shown. In the second embodiment, the supplementary, intermediate first and second surface data sets are used as an intermediate step to more easily perform registration. Here, although steps S1 to S4 remain essentially unchanged, in particular, before positioning the resection 5 in the ex vivo scanning device, the measurement of the remaining tissue surface 7 in the resection 5 and the extraction area 6 is performed in step S9. For this purpose, for example, a laparoscope 12 or other other recording instruments 11, such as a surgical microscope, can be used. The resection 5 can be rotated in front of the recording instrument 11, for example. With regard to the remaining tissue surface 7, sub-data sets are also recorded in different directions of the imaging sensor of the recording instrument, particularly the stereo camera. Then, for example, by means of the method described in US 2017 / 0105601A1, a three-dimensional surface model can be generated for the resection 5 and the extraction area 6 respectively according to the sub-data set in step S10. If the image data of the sub-data set is now used as the texture of the surface model, the corresponding first and second surface data sets of the remaining tissue surface 7 in the resection 5 and the extraction area 6 are formed in step S10.

[0131] Since the same surface is shown, in step S11, the scan results of the large area of ​​the ex vivo scanning device 16 recorded in step S1 can be registered to the first surface data set of step S10 by image analysis with the aid of suitable algorithms and possible interpolation / smoothing, for example based on the macroscopic tissue structures contained in the scan data of the ex vivo scanning device 16 and in the image data of the stereo camera of the laparoscope 12. In step S11, markers (marking objects) arranged on the resection part 5 can also be used for simple registration.

[0132] Furthermore, a registration step S6 can also be performed in parallel with step S11 or before or after step S11, which registration step S6 registers the first and second surface data sets recorded in step S10 with one another, which have been recorded using the same modality. Nevertheless, it may be appropriate to perform step S6 only after step S11, since the first surface data set of the off-body scanning device 16 recorded in step S1 can then also contribute to improving the registration. Ultimately, the first surface data sets of two different modalities are then used in the registration step S6.

[0133] In step S12, the segments recorded in step S5 by the tracked digital biopsy tool 10 are also registered directly with the second surface data set determined in step S10, which can be realized particularly easily since the second surface data set is already present in the patient-related coordinate system of the navigation system 14, which also tracks the digital biopsy tool 10, due to the use of the recording device 11. It should be noted that if step S6 is only carried out afterwards, these measurements can also enter into the registration if necessary.

[0134] Then, according to Figure 4 In the second embodiment, step S7 may also provide support measures based on the visualization of the textured surface model determined in step S10 .

[0135] Figure 5 The third embodiment of Figure 4 The second embodiment of the invention differs in that the digital biopsy tool 10 is not used here. The first and second surface data sets determined in step S10 are directly rigidly or preferably deformably registered with each other. This is achieved based on the geometric properties of the corresponding texture of the three-dimensional surface model generated from the stereo camera data. As a result, the indicative features of infiltrating tumor cells identified by the digital histology device, i.e. the ex vivo scanning device 16, can be directly associated with the corresponding positions on the remaining tissue surface 7. However, it is still preferred to use the digital biopsy tool 10 within the scope of the present invention in order to provide a basis for the registration and to enable examination in the patient (see step S8).

[0136] In the corresponding step S6 of the embodiment under discussion, a correspondence data set can also be used, which contains correspondence pairs of corresponding surface features, in which the described similarity does not necessarily exist. This means that such a correspondence data set (exemplarily referring to Figure 6 The determination of such a correspondence dataset (explained in more detail) can also make it possible to associate such surface features with one another which no longer have a similarity which would allow an image-based association in the surface dataset recorded after the end of the resection, for example because too large a tissue layer was removed in the meantime and / or a subsequent treatment, in particular ablation, was performed on one side. The correspondence dataset can therefore provide an additional similarity criterion.

[0137] When determining the correspondence data set, use is made of the fact that almost all surgical steps that contribute to the separation of the resection part 5 from the remaining tissue are performed under observation, in particular with the aid of a stereo camera, such as a laparoscope 12, so that it is also assumed below that there is a monitoring video, the frames of which show individual monitoring images recorded successively in time. Since such monitoring videos are usually annotated manually and / or automatically, which in particular involves automatically determined workflow steps, it is easy to select the relevant part of the monitoring video that fully shows the resection of the resection part 5.

[0138] The basic idea is therefore to generate a correspondence data set between the surface of the resected part 5 and the remaining tissue surface 7 based on a surveillance video or surveillance image of the resection, for example recorded by a laparoscope 12 or alternatively by a surgical microscope. This correspondence data set can be used to subsequently align a position 22 on the surface of the resected part 5 with a corresponding position 23 on the remaining tissue surface 7.

[0139] The analysis of the monitoring images described here can be performed in real time during the resection, but preferably can also be performed subsequently based on the recorded monitoring video.

[0140] In step S13, the analysis starts by jumping to the frame representing the start time point of the annotated / identified excision in the surveillance video by counting the variable. In the following, the counting variable will be denoted by n.

[0141] In step S14, the nth frame of the monitoring video is loaded as the monitoring image accordingly. In the case of a monitoring video recorded with a stereo camera, in step S2, a stereoscopic reconstruction of the three-dimensional surface map of the current scene can also be performed using methods that are basically known from the prior art. Then, the subsequent steps are selectively performed on the monitoring image of the frame or on the textured three-dimensional surface map of the current scene.

[0142] In step S15, the current monitoring image is analyzed from a number of different aspects, in particular in order to determine whether an interaction of the surgical instrument 8 with the tissue has occurred at all. For this purpose, on the one hand, the surgical instrument 8 is detected and positioned in the monitoring image, wherein alternatively or additionally, position information of the navigation system 14 about the surgical instrument 8 is also used. Then, using the current information, it is determined whether the surgical instrument is currently separating or removing or processing tissue, or whether the surgical instrument is currently not affecting the tissue. For each frame within the relevant resection time period, this information is stored in association with the corresponding monitoring image. In particular, this information can be used to further reduce the number of frames or monitoring images to be analyzed more accurately, because new correspondence pairs of surface features only appear when the tissue is actually also affected by the surgical instrument.

[0143] In the steps following step S16 , each frame to be processed, ie, the monitoring image, is then continuously analyzed as described.

[0144] In step S16, if necessary, Figures 3 to 5After the described preprocessing, for the frame currently to be processed, an attempt is first made to find again the surface features of the correspondence pairs that have already been stored in the correspondence data set, so that the surface features are located in the monitoring image. The same applies to the reference features that are optionally used if necessary. Correspondingly, if the correspondence pairs have not yet been determined, they are accordingly omitted. In addition, an attempt is also made in step S16 to find new corresponding surface features. As already mentioned, this method step is only performed on the monitoring image if the surgical instrument has an impact on the tissue in the current frame or the frame just recorded, for which the impact, for example, leads to a further separation of the resected part 5 from the remaining tissue by cutting off or cutting the tissue or other changes in the tissue.

[0145] As candidates for corresponding surface features, it is preferred here to select by image analysis, on the one hand, those features that have not yet been fully analyzed in the previous monitoring image, but on the other hand, those features that can also be clearly identified at least in the imaging modality of the ex vivo scanning device 16. As a further imaging modality, the imaging modality of the recording instruments 9, 11 can be considered. For such surface features, it is geometrically checked whether an actually matching surface feature can be found on the "other side". In other words, the current incision edge or separation edge marks the boundary between the tissue that will form the surface of the resection part 5 and the tissue that will form the remaining tissue surface 7 of the extraction area 6. The corresponding surface feature should be located on two different sides of this separation edge and geometrically meet the following condition: the corresponding surface feature may have been separated by the surgical instrument 8 in the current or sufficiently recent previous frame. If this is not the beginning of the resection, the new surface feature should be adjacent to the surface feature that was identified earlier, which means that it will not be too far away. In such a check of the relative position to the surface features that already form part of a correspondence pair and to the relative position to possible reference features, if a new correspondence pair is actually found, this relative position can also be immediately stored, since the relative positions of the surface features (and possibly also the reference features) relative to one another are also used in further processing. In the context of the search for corresponding surface features, in particular in addition to geometric considerations, a similarity check as already described can also be performed, for example, fingerprint data sets can be created and compared.

[0146] If a new correspondence pair is found in step S16, it is stored as part of a correspondence data set, which includes feature information, here at least the part of the monitoring image showing the surface features, but also further feature information if necessary. In particular, when using the correspondence data set to support the registration according to step S6, a fingerprint data set of the surface features is provided as special further feature information.

[0147] Specifically, in this embodiment, each corresponding relationship pair found is used to expand the surface graph, which will be described in more detail below.

[0148] pass Figures 7 to 9 The diagram of this process for finding a correspondence pair of corresponding surface features is explained in more detail, wherein the surface features already discussed can of course also be used here. Figures 7 to 9 Surveillance images at different time points, ie, different frames of surveillance video, are schematically shown.

[0149] Figure 7 A monitoring image 24 at the start of a resection is schematically shown. The surgical instrument 8 at the surface of the organ 4 and the incision 25 which has already been started can be seen.

[0150] Figure 8 A subsequent frame is shown, that is, a monitoring image 26 recorded at a later point in time. A clear further separation along the incision edge 27, on which the surgical instrument 8 is acting, can be seen there. Surfaces are formed to the right and left of the incision edge 27, i.e., to the right a subsurface 28 of the future resected portion and to the left a subsurface 29 of the future remaining tissue surface 7. The triangle 30 marks the position of the surface features of the correspondence pair already detected in the earlier frame, while in this frame, i.e., in the monitoring image 26, the circle 31 shows the newly detected surface features of the new correspondence pair, the relative position of which with respect to the corresponding features marked by the triangle 30 can be clearly determined.

[0151] Fig. 9 A monitoring image 32 is shown at a relatively short time point after the time point 26. It can be seen that the surgical instrument 8 has just been moved out of the region of the incision edge 27, which indicates that no tissue influence occurs in the subsequent time period. However, due to the removal of the instrument 8 and the previously performed resection, further surface features marked by squares 33 can be seen near the surface features marked by circles 31, which form new correspondence pairs.

[0152] Back to Figure 6, step S17 involves generating or updating a textured surface map of the resulting surface of the cutout portion 5 and the extracted region 6. Due to the known relative positions of the surface features of the correspondence pair, a surface model is first created, if necessary three-dimensionally using a stereo camera, and a surface pattern or corresponding image data from at least one of the monitoring images 24, 26, 32 is associated with the surface model as a texture. The fragments from the monitoring images 24, 26, 32 used in each case at least show the corresponding surface features (as feature information), but preferably also the areas between the positions of the surface features are filled in accordingly. In this case, the texture can be determined in particular when using multiple frames, for example by averaging multiple monitoring images 24, 26, 32 showing the corresponding parts.

[0153] The current correspondence pair, the future cutout 5 and the textured surface image of the extraction region 6 , and possibly further information (further feature information, relative position, etc.) are stored in a data memory of the control device 17 .

[0154] In step S18, it is checked whether the end of the relevant part of the monitoring video has been reached. If not, then continue to jump to the next frame, i.e. n->n+1. If yes, then the method terminates.

[0155] At this point, it should also be noted that in step S16, it is also monitored whether a unilateral tissue change has occurred, for example, due to unilateral cauterization by means of an energy device, i.e. a tissue change that only or substantially only involves part 28 or part 29. In this case, the surface texture of the surface image is adjusted again at the corresponding location, and if necessary, other characteristic information is also adjusted. In this way, similarity relationships are formed, in particular in the form of correspondence pairs, which otherwise would no longer be recognizable by a purely image-based comparison of the final state.

[0156] Accordingly, according to Figures 3 to 5 In step S6 of the method, the correspondence data set can also be used to improve the alignment, in which method, based on the correspondence pairs with their feature information, in particular the fingerprint data set and possible relative positions, further similarity criteria are added, with the help of which surface features that no longer look similar due to excessively thick tissue layers cut off in the middle or subsequent one-sided processing can still be identified as corresponding.

[0157] However, if through Fig.10As shown in the flowchart in , a design of the method according to the invention is also conceivable, in which the surface image of the remaining tissue surface 7 of the extraction region 6 formed in step S17 ultimately serves as the second surface data set. If necessary, i.e. if the position of the organ 4 or the patient 3 has not changed significantly, this second surface data set is even already present in the patient-related coordinate system, since the remaining tissue surface 7 has not been removed relative to the resection part 5, and the patient-related coordinate system can also be used to track the recording instruments 9, 11 for recording the monitoring images 24, 26, 32 by the navigation system 14. However, it is of course also possible to record a further second surface data set, which, since it shows the same surface, simplifies the registration with the surface image of the extraction region 6, thereby establishing a relationship with the patient-related coordinate system.

[0158] exist Fig.10 In step S19, the surface map of the remaining tissue surface 7 in the resection 5 and the extraction region 6 created in step S17 is now first preprocessed. The information about the positions of the surface features of the correspondence pair in the surface map already represents the correspondence information, because these positions are associated with each other, but the correspondence information only exists point by point where the surface features determined to be suitable for the surface pair are present. These correspondences are now appropriately interpolated in step S19, so that the corresponding positions on the remaining tissue surface 7 can be associated with each position on the surface of the resection 5 or at least as many positions as possible. Therefore, a correspondence map or a correspondence table is finally formed, which is ideally valid for the entire surface map and therefore represents the completeness of the registration between the surface maps. Since the first surface data set recorded by the ex vivo scanning device 16 can be easily registered with the surface map of the resection 5 based on the same surface, the registration between the first surface data set and the surface map of the remaining tissue surface 7 of the extraction region 6, which can be regarded as a second surface data set, is formed by the association of this registration with the registration given by the correspondence information.

[0159] In other words, in step S20, at least a portion of the first surface data set recorded with the ex vivo scanning device 16, which shows malignant changes in the tissue near the surface, can be image-based registered with the surface map of the resected part 5. For example, a microscope image of the surface of the resected part 5 can be scaled down accordingly and then spatially registered to the surface map image-based. Advantageously, a complete registration of the surface data set with the surface map of the resected part 5 is performed. As a result of step S20, a position on the surface map of the resected part 5 can therefore be associated with an indicative feature in each case.

[0160] Then, in step S21, the corresponding relationship information is used to determine the RThe corresponding position b V .

[0161] If the surface image of the extraction region 6 is not yet available as a second surface data set in the patient-related coordinate system, then in step S22 it is possible to (at least at position b V Image-based registration is performed between the texture of the surface map of the remaining tissue surface 7 (surrounding) and the current video image of the laparoscope 12 or another second surface data set. Alternatively, the registration can also be performed by the navigation system 14, in which, as described, the entire surface map of the remaining tissue surface 7 is registered to the geometry during surgery.

[0162] Finally, in step S23 similar to step S7 , supporting measures are performed, such as displaying the surface portion of the remaining tissue surface 7 corresponding to the location of the tumor infiltration of the resected part 5 as an overlay, augmented reality or marking in the navigation system.

[0163] at last, Fig.11 Possible functional units of a control device 17 of a support system 1 according to the invention are shown. The control device 17 may firstly have an interface 34, via which the image data required for carrying out an embodiment of the method according to the invention can be received from the corresponding device / instrument (ex vivo scanning device 16, recording device 9, 11, etc.) for processing. Furthermore, according to an embodiment, the control device 17 has at least one registration unit 35 for carrying out steps S6, S11, S12, S16, S17, S19, S20, S21 and / or S22. According to an embodiment, a support unit 36 ​​may be provided for carrying out steps S7 or S23.

[0164] Depending on the specific configuration of the method according to the invention to be executed by the control device 17, further functional units may of course also be present, possibly subunits of the registration unit 35 and the support unit 36, in particular at least one image processing unit, at least one preprocessing unit, etc. The storage device or data memory 37 can be used to store intermediate results or registration results, etc.

[0165] Although the present invention has been explained and described in more detail through preferred embodiments, the present invention is not limited to the disclosed examples, and those skilled in the art may derive other variations therefrom without departing from the scope of protection of the present invention.

Claims

1. A support system (1) for providing navigation support to a person performing a resection on a patient (3) after extracting a resection portion (5) to navigate about the resection portion (5) in the extraction area, characterized in that - recording a first surface data set of the cut-away portion (5) using an ex vivo scanning device (16), - determining a second surface data set of the extraction region (6) using the recording device (9, 11), said second surface data set covering at least a part of the remaining tissue surface (7) of the extraction region (6) of the resected part (5) in the patient (3), - based on the remaining tissue surface (7) in the extraction region (6) and the corresponding surface features of the resection portion (5), registering the first surface data set of the resection portion (5) with the second surface data set of the extraction region (6), and - performing at least one supporting measure using the registration, which supports navigation in the extraction region (6) with respect to the resection (5).

2. The support system (1) according to claim 1, characterized in that The recording instrument (9, 11) is tracked in a patient-related coordinate system with the aid of a navigation system (14), wherein a second surface data set is also recorded or present in the patient-related coordinate system, wherein the support measure comprises displaying the relative position of at least one indicative feature of the resected portion (5) relative to the recording instrument (9, 11) and / or relative to another instrument tracked in the patient-related coordinate system by the navigation system (14) and / or relative to the surface of the extraction area (6).

3. The support system (1) according to claim 2, characterized in that In order to display the relative position, an augmented reality device (19) and / or a display device (18) is used to reproduce the first and / or second surface data set, wherein instruments and / or further instruments and / or markers indicating their relative position are faded into the first and / or second surface data record; and / or to reproduce an image currently recorded by at least one of the recording instruments (9, 11) and / or at least one further instrument, wherein at least one indicative feature from the first surface data set is faded into the image.

4. The support system (1) according to claim 2 or 3, characterized in that: At least one of the at least one indicative features describes the presence of a malignant change in the resected portion (5) at a defined approach distance to the surface of the resected portion (5), said presence being determined by at least partially automatically analyzing the first surface data set and / or a further image data set of the resected portion (5) registered with the first surface data set.

5. The support system (1) according to any one of claims 1 to 3, characterized in that The first surface data set and / or the second surface data set and / or the further image data set are recorded using an optical imaging modality.

6. The support system (1) according to any one of claims 1 to 3, characterized in that A digital biopsy tool (10) is used as the recording instrument (9, 11), which measures the tissue surface (7) in the extraction area (6) segment by segment, wherein the corresponding segment is positioned on the surface of the resection part (5) in a registered framework.

7. The support system (1) according to claim 6, characterized in that By analyzing the first surface data set, at least one region is determined, which has a surface feature identifiable in the surface data of the digital biopsy tool (10), and the region is indicated to a person with respect to the resection portion (5).

8. The support system (1) according to claim 6, characterized in that In the context of the registration, a quality level describing the reliability of the registration result is determined, wherein if the quality level falls below a threshold, it is required to record further segments and / or to perform at least one validation measurement with the digital biopsy tool (10) to check the information conveyed by the supporting measures.

9. The support system (1) according to any one of claims 1 to 3, characterized in that The first and / or second surface data set consists of sub-data sets recorded from different viewing directions directed toward the cut-away portion (5) or the extraction region (6).

10. The support system (1) according to claim 9, characterized in that Firstly, a surface model describing the surface course is determined from the partial data sets, which surface model is associated as a texture with the image data of the partial data sets.

11. The support system (1) according to claim 9, characterized in that When a second surface data set is determined and when a digital biopsy tool (10) is used, the fragments measured by the digital biopsy tool (10) are aligned to the second surface data set, the second surface data set showing the entire surface of the extraction area (6) formed by resection, and the digital biopsy tool measures the tissue surface (7) in the extraction area (6) fragment by fragment.

12. The support system (1) according to any one of claims 1 to 3, characterized in that The registration is performed successively at different resolution levels; and / or a plurality of first surface data sets and / or a plurality of second surface data sets are recorded using different modalities, and all surface data sets are used for the registration.

13. The support system (1) according to any one of claims 1 to 3, characterized in that As surface features used for alignment, cell structure and / or cell type and / or cell arrangement and / or cell clusters and / or fibrous structure of the tissue and / or vascular structure of the tissue and / or spectral reflectivity of the tissue and / or distribution of fluorescent dyes in the tissue and / or distribution of malignant changes on the visible tissue surface and / or trajectory of the surgical instrument (8) are used.

14. The support system (1) according to any one of claims 1 to 3, characterized in that Surface features in different resolution regions of the surface datasets are detected and / or combined into fingerprint datasets which are compared for registration.

15. The support system (1) according to claim 14, characterized in that When using the fingerprint dataset, the first and second surface datasets are recorded using different modalities.

16. The support system (1) according to any one of claims 1 to 3, characterized in that An artificial intelligence registration algorithm is used for the registration of the surface datasets, which has been trained using the annotated surface datasets as training data.

17. The support system (1) according to any one of claims 1 to 3, characterized in that To support the registration, a correspondence data set is generated which contains correspondence pairs of surface features that correspond to one another by: - during the cutting of the cut-out portion (5), continuously recording a monitoring image (24, 26, 32) of the current working area by means of the recording device (9, 11) or another recording device, - when the implementation of the cut is detected, detecting the surface features of the respectively formed surfaces located on both sides of the cut by automatically analyzing the corresponding monitoring images (24, 26, 32), - correlating surface features that geometrically correspond to each other with respect to the cutout to produce corresponding pairs, and - storing each correspondence pair together with feature information describing surface features of the correspondence pair in a correspondence data set.

18. The support system (1) according to claim 17, characterized in that When a new correspondence pair is generated, the relative position relative to at least one other correspondence pair already contained in the correspondence data set and / or relative to a reference feature that is not part of the correspondence pair is determined by detecting at least one surface feature and / or a reference feature of the other correspondence pair in a monitoring image (24, 26, 32) showing surface features of the new correspondence pair, and the relative position is stored in the correspondence data set.

19. The support system (1) according to claim 18, characterized in that A surface model of the surface of the extraction area (6) formed by the excision and a surface model of the excised portion (5) are created based on the correspondence pair and their corresponding relative positions. In order to determine the corresponding surface map, at least a part of the feature information and / or a part of at least one monitoring image showing the corresponding surface area are associated with the surface model in a position-true manner, thereby providing correspondence information that is mutually associated with the position on the surface map through the correspondence pair.

20. The support system (1) according to claim 19, characterized in that In order to expand the correspondence information provided by the correspondence pair, further corresponding positions of the surface map are determined by interpolation in the region of the surface map that lies between the surface features of the correspondence pair.

21. The support system (1) according to claim 19 or 20, characterized in that In order to locate the indicative features of the resected portion (5) on the remaining tissue surface (7) of the extraction area (6), the correspondence information is used, the position of the indicative features on the surface of the resected portion (5) being known.

22. The support system (1) according to claim 17, characterized in that The detection of the performed cut is performed at least partially on the basis of the image in the monitoring image (24, 26, 32) and / or on the basis of additional information which is temporally associated with the monitoring image (24, 26, 32).

23. The support system (1) according to claim 22, characterized in that The additional information is determined, which describes the movement of the surgical instrument (8) determined by the navigation system (14) or the navigation system (14), and / or the energy supply to the surgical instrument (8) triggered by operating the foot pedal, and / or a workflow step of the surgical workflow.

24. The support system (1) according to claim 17, characterized in that The surface feature of the new assignment pair is selected such that it can be detected with a reliability exceeding a threshold value in the corresponding imaging modality of the surface dataset and / or at least one further image dataset.

25. The support system (1) according to claim 17, characterized in that During the resection time period, as long as the surface features of the corresponding pair are contained in the monitoring image (24, 26, 32), the surface features of the corresponding pair are tracked in the monitoring image (24, 26, 32), wherein for determined subsequent changes in the surface features caused by the or another surgical instrument (8), the feature information is adjusted to describe the changed surface features; and / or for unilateral surface changes determined by automatic analysis, the changed surface features of the corresponding data set are found, and the relevant feature information is adjusted to describe the changed surface features.

26. A computer program product to be executed on a control device (17) of a support system (1) according to any one of claims 1 to 25.

27. An electronically readable data carrier on which a computer program product according to claim 26 is stored.

Images