Tool-independent assistance for surgery

Abstract

The present invention relates to a technique for controlling an assistant function for a surgical procedure on a patient. A machine learning algorithm is used to obtain a map (76) of a surgical area. This map can be used to control an assistant function in relation to the surgical procedure. The map indicates one or more areas of activity (101-103) that are associated with a higher probability of the presence of a surgical tool.

Description

[Technical Field]

[0001] Various examples of the present disclosure relate to techniques for controlling assistance functions associated with surgical intervention based on microscopic images from a surgical microscope. [Background technology]

[0002] The prior art discloses surgical microscopes that provide a user, typically a surgeon, with information on various items. For example, German Patent Application No. DE 10203215 A1 describes a surgical microscope that includes a camera that generates an electronic image signal. The image signal is displayed on an electronic eyepiece, which includes a corresponding display device for the electronic image data. Other items of information can also be output thereon. A surgical microscope is also known from German Patent Application No. DE 10 2014 113 935 A1.

[0003] A typical surgical microscope has many possible settings, which can often require significant expenditure to select a good setting during surgery.

[0004] The disclosure of the document EP 3593704 is known, among others, which describes an assisted endoscope whose actions are guided by image processing and a database of previous surgeries, where a manually created database is used, but such techniques are often limited in terms of flexibility and therefore often inaccurate. Summary of the Invention [Problem to be solved by the invention]

[0005] Therefore, there is a need for improved methods for controlling support functions associated with surgical interventions, and in particular for controlling support functions associated with controlling a surgical microscope used during a surgical intervention. [Means for solving the problem]

[0006] This object is achieved by the features of the independent patent claims. The features of the dependent patent claims define embodiments.

[0007] Below, a technique is described for providing assistance by means of computer-implemented algorithms during surgical interventions, in particular microsurgical interventions, in the head, eye area, heart area, spine (neurology), ear, nose and throat area or dental area.

[0008] For this purpose, the assistant functions are appropriately controlled. The assistant functions may automatically control one or more pieces of equipment used during the surgical intervention. For example, they may control a surgical microscope. The assistant functions may also provide for user interaction, for example by giving commands.

[0009] In various examples, for example, one or more actions of an assistive function can be initiated, the time at which the one or more actions are initiated can be identified, and the type of action can also be identified.

[0010] To control the assistance functions, the microscopic image of the surgical microscope is used to obtain a map of the intervention area imaged by the microscopic image.

[0011] The map, generally speaking, can show the temporal and / or spatial context of one or more areas of activity identified based on the microscopic images within the intervention area. Increased activity can be identified within an area of ​​activity in comparison with other areas within the intervention area. In other words, the map can have multiple areas, and some areas can be identified as active if their activity or dynamic characteristics are increased compared to other areas of the map. Increased dynamic characteristics can be caused in particular by the movement of a surgical instrument within the corresponding area of ​​activity.

[0012] Such techniques are based on the insight that changes over time in a microscopic image can be caused, inter alia, by the movement of a surgical instrument (also called a tool). Various examples described herein are based on the insight that it is not necessary to explicitly recognize the type of surgical instrument used to control assistance functions related to a surgical intervention. Rather, merely the presence or absence of a surgical instrument in different zones within the intervention area can be identified based on a comparison of the microscopic images, for example, according to the following principle: "Where something is moving, that is a possible location for a surgical instrument." In other words, the map can be tool-type independent, i.e., it does not change, for example, depending on the type of surgical instrument. Thus, a map identifying one or more active areas can be determined based on the time context of the microscopic image. Assistance functions can then be controlled based on this map. In particular, the position of a surgical microscope can be changed, for example. This can be done on the prerequisite that a surgical instrument has been recognized in the microscopic image.

[0013] According to one example, a method for controlling an assistance function for a surgical intervention on a patient includes applying a machine learning algorithm based on at least two microscope images taken by a surgical microscope. The at least two microscope images image an intervention area of ​​the surgical intervention. The at least two microscope images are taken at different time points, i.e., at two or more time points. Based on applying the machine learning algorithm, a map of the intervention area is obtained. The map shows one or more active areas within the intervention area. The one or more active areas are associated with a higher probability of the presence of a surgical tool. The method further includes using the map to control an assistance function related to the surgical intervention.

[0014] In general, it is also contemplated to identify one or more maps. In the following, the technique is described for one map for simplicity. However, the corresponding technique can be repeated for multiple maps. In general, it is also contemplated to consider multiple intervention areas, for example by using multiple maps, e.g., one for each intervention area.

[0015] The at least two microscopic images can be selected from a group including, for example, single images, stereo images, and images encoding depth information. Stereo images can encode depth information. Therefore, generally speaking, the at least two microscopic images can optionally encode further information, such as depth information. Any suitable imaging modality can be used for this purpose. For example, the corresponding information can be captured by time-of-flight imaging and / or structured illumination and / or stereo imaging. When structured illumination is used, an illumination pattern can be projected onto the intervention area, and the depth information can be ascertained based on the distortion of the illumination pattern due to the topography of the intervention area. For example, a corresponding topographic image can be generated based on such encoded depth information for each microscopic image in a preprocessing step, and this topographic image can then be used as input for a machine learning algorithm.

[0016] In various examples described herein, different types of machine learning algorithms can be used, in particular, deep neural networks, such as convolutional neural networks.

[0017] The time intervals between the different time points at which the microscopic images are taken can correspond to the time scale on which surgical instruments are typically moved by a surgeon, i.e. the time intervals can be in the range of seconds, i.e. for example in the range of 200 ms to 5 seconds.

[0018] An active area can thus refer to an area where the probability of a surgical tool being located is higher compared to other areas within the intervention area.

[0019] Such techniques are based on the insight that surgical instruments are moved over time, resulting in time-varying contrast in microscopic images. Areas of activity can be identified by considering the time-varying contrast. The background, i.e., the patient's anatomical features, may often be relatively static.

[0020] The map can generally represent specific features extracted from the imaging modality of the surgical microscope used to capture the at least two microscopic images of the semantic context represented by those images. The map can therefore extract features from the at least two microscopic images and, optionally, based on further input to a machine learning algorithm. The map can be correctively adjusted, for example, so that noise or disturbances contained in the at least two microscopic images are not used as features in the map. The contrast of a microscopic image typically depends on the imaging modality used. The map can extract contrast.

[0021] The map and the at least two microscope images can be defined in a common reference coordinate system, which allows an assignment to be made between a feature represented by the map, e.g., the location of an active area, and its corresponding location in the at least two microscope images. The microscope images represent the real field of view of the surgical microscope, which can include the intervention area.

[0022] Such techniques are based on the insight that it is beneficial to control an assist function based on the identification of one or more activity areas. It is often particularly beneficial when the presence, and especially the location, of a surgical instrument in the intervention area is used as the basis for controlling the assist function. Various examples are based on the insight that the presence of a surgical instrument in a particular activity area within the intervention area may be more important for the accurate control of the assist function than distinguishing the type of surgical instrument.

[0023] In this regard, in various examples, it may be possible to make the map independent of differences in the type of surgical tool, i.e., not distinguish between types of surgical tools (a tool-type-independent map). In this case, the type of surgical tool refers to, for example, the shape, appearance, and function performed by the surgical tool. The type of surgical tool may be, for example, a knife, scissors, clamp, scalpel, aspirator, tweezers, CUSA, bone punch, etc. However, the type of surgical tool is not intended to be surgeon-dependent. In other words, the type of surgical tool is not intended to be dependent on the specific role of the surgical tool in an actual instance of surgery. For example, a particular type of surgical tool, e.g., a knife, may be used in different ways by different surgeons, and the knife may be held and moved differently, i.e., its movement pattern may vary from surgeon to surgeon. However, such use of the surgical tool defines an activity area, which can be represented by the map in this regard.

[0024] This means, in other words, that although the active areas indicate the presence of surgical instruments, the map does not contain corresponding features that distinguish between different types of surgical instruments.

[0025] Robust control of assistive functions has been demonstrated, particularly without distinguishing between surgical instrument types. This may be due in particular to the fact that the appearance of surgical instruments can vary significantly, even among the same type of surgical instrument. For example, the color and shape of surgical instruments can vary significantly depending on the manufacturer. The reliability of assistive function control has been found to be adversely affected when accounting for differences in surgical instrument types, as this introduces new sources of error. Machine learning algorithms can be trained more robustly when differences in surgical instrument types are not required. Thus, in various examples, it is possible for assistive functions to be controlled independently of differences in surgical instrument types, i.e., not intended to be controlled in response to differences in surgical instrument types.

[0026] The activity area can be rendered in a number of ways within the map.

[0027] For example, it is conceivable that the map contains probability values ​​for the presence or absence of a surgical tool for multiple areas of the intervention region, which would correspond to a probability map. In this way, active areas can be indicated. In this case, one or more active areas can be localized to areas with higher probability values.

[0028] The presence of a surgical instrument can be checked, without determining its location. Depending on whether a surgical instrument is recognized, an assistance function can be selectively initiated, or different assistance functions can be initiated depending on whether a surgical instrument is recognized. This variant is based on the insight that surgical instruments can be reliably recognized, and a map indicating the area of ​​activity can be used for the spatial component of the assistance function, so localization is not necessary. This is based on the insight that if a surgical instrument is present in the microscopic image, it will generally initiate more activity (hence there is no need to perform a separate localization of the surgical instrument, e.g., based on optical flow, in addition to determining the map). This means that the recognition of the presence of a surgical instrument can be used to infer that an increase in activity in the area of ​​activity is indeed due to the surgical instrument (and not due to other disturbances).

[0029] The machine learning algorithm may have a recurrent layer to output a continuous probability value.

[0030] In this way, it may be possible to specify a probability distribution for the location of the active area within the intervention region. In this way, the assistance functions can be controlled in a more discriminatory way.

[0031] For example, techniques related to machine learning algorithms, such as those essentially known in connection with so-called "saliency" analysis, can be used for such probability values. See, for example, Reddy, Navyasri, et al. "Tidying deep saliency prediction architectures," arXiv preprint arXiv:2003.04942 (2020). For example, in this case, artificial deep neural networks can be used. These networks can have an encoding branch that converts the image information into a reduced-dimensionality feature vector or feature matrix. Optionally, decoding can then be performed and upsampled to the desired resolution. In this case, the latent features of the encoding can be defined in an application-specific manner. Instead of or in addition to using a probability distribution to indicate an active area, deterministic localization can also be performed. For example, binary differentiation can be performed regarding whether a particular localization is inside or outside an active area. The machine learning algorithm can include a classification layer for binary classification. In this regard, masks can be used for one or more active areas. The mask provides a binary discrimination between different locations within the intervention region, i.e. it can specify in each case whether a particular location within the intervention region is inside or outside the activity area.

[0032] The map may therefore, generally speaking, include active areas as corresponding features. In addition to map features specifying one or more active areas, the map may also include further features, such as the location of anatomical features of the patient, a reference position in the field of view of the surgical microscope, such as the center of the field of view, etc. However, instead of or in addition to such features unrelated to the active areas, the map may include one or more further features indicating contextual information for one or more active areas.

[0033] Various types of features are possible. Apart from one or more activity areas, the map may also include features indicating that an activity area group of one or more activity areas satisfies one or more spatial adjacency relationships. The map may, for example, indicate two or more activity areas. An activity area group may therefore include two or more activity areas. A machine learning algorithm may then check to see whether the activity areas of a group satisfy certain criteria regarding their relative spatial arrangement (spatial adjacency). For example, a check may be performed to see whether the distance between the activity areas of each group is less than a predetermined threshold, i.e., whether the activity areas are relatively close to each other (which may indicate that surgical instruments are positioned near each other). Another adjacency relationship may, for example, relate to the average distance between the activity areas closest to each other within the same group.

[0034] The additional identification of the activity areas of a group with respect to the map as satisfying a spatial adjacency relationship allows taking into account corresponding contextual information when controlling assistance functions based on the map. For example, in some variants, it may be beneficial to give more weight to spatial clusters of activity areas that are close to each other in controlling assistance functions than activity areas that are, for example, located at a long distance from other activity areas within the intervention area.

[0035] In addition to such adjacency relationships defined within the active areas themselves, adjacency relationships defined relative to the location of one or more active areas within the patient's anatomy can also be taken into account alternatively or additionally. The map can therefore include features indicative of the fulfillment of one or more anatomical adjacency relationships of active areas within one or more active areas. For example, active areas located relatively close to protected anatomical regions, such as sensory organs, can be identified. Active areas located particularly close to anatomical target regions, such as tumors to be removed, can also be identified.

[0036] It has been demonstrated that such anatomical context can be beneficial in reliably controlling assistive functions. For example, increased attention from the surgeon can be expected for activity areas close to sensory organs, etc. For example, activity areas located particularly close to sensitive anatomical features of the patient or close to the target area of ​​surgical intervention can be given more importance in controlling assistive functions. This is based on the insight that such activity areas typically require higher attention from the surgeon. Furthermore, in this way, an order of multiple activity areas can be defined, and assistive functions can be intentionally performed in relation to activity areas at higher hierarchical levels defined according to the order.

[0037] Another feature that the map may optionally include that provides contextual information about the activity area may relate to, for example, the number of surgical instruments in the activity area.

[0038] Multiple surgical instruments may be used on top of or next to each other, and the map could then show how many corresponding surgical instruments are included in the corresponding activity area. Machine learning algorithms could perform the corresponding classification / regression.

[0039] For example, an activity area having a relatively large number of surgical instruments disposed therein may require greater surgeon attention than, for example, an activity area having a relatively small number of surgical instruments disposed therein. The order of the activity areas may also be defined based on such an indication of the number of surgical instruments used per activity area. For example, it may be envisioned that an activity area having a large number of surgical instruments located therein may be determined to be at a higher hierarchical level in the order.

[0040] Instead of or in addition to the above-mentioned feature classes, the map can also indicate dynamic features, such as increased activity or dynamic features, or increased movement / activity in one or more activity areas, particularly in parts / elements of the image. For example, certain surgical instruments, such as clamps for fixing tissue, can be expected to be positioned in a relatively stationary state, whereas other surgical instruments, such as aspirators, are manually guided and therefore positioned in a relatively dynamic state. This can be expressed in dynamic features within different activity areas, quantified, for example, by the intensity of the optical flow between different microscopic images, and the type of surgical instrument is not trained and / or recognized by the model or algorithm.

[0041] Typically, surgical instruments with high dynamic characteristics in terms of positioning may be relatively important to the surgeon, especially for hand-guided surgical instruments. It may be envisaged that the sequence between active areas may also be determined by taking into account the dynamic characteristics of the surgical instruments within the active areas.

[0042] Generally speaking, the method could further include specifying an order for one or more activity areas.

[0043] The order can be determined based on one or more features, including contextual information about one or more activity areas, and the assistance function can then be controlled according to this order.

[0044] Therefore, the order can define a hierarchy between activity areas: activity areas at higher hierarchical levels may have a greater influence on the control of support functions than support areas at lower hierarchical levels.

[0045] Objective conflicts in particular can be resolved by such techniques. For example, if there are several different active areas arranged at a distance from one another, only one of these active areas can be positioned in the center of the field of view of the surgical microscope at any one time. Corresponding objective conflicts can then be resolved by sequentially positioning the "most important" active area in the center of the field of view. This allows particularly good control of the assistance functions, which are adapted to the requirements of the surgical intervention.

[0046] The order may be determined, for example, by a machine learning algorithm, however, it would also be conceivable that the order is determined by a downstream control algorithm.

[0047] Various examples have been described above relating to a map based on at least two microscopic images and including features associated with one or more areas of activity. A machine learning algorithm creates this map based on input.

[0048] This input is based on at least two microscopic images. In one variant, it could be envisaged that the machine learning algorithm receives the at least two microscopic images as direct input. For example, the at least two microscopic images could be transmitted to the machine learning algorithm as separate channels. However, alternatively or additionally, further or other information that the machine learning algorithm receives as input could also be determined based on the at least two microscopic images.

[0049] In various examples, pre-processing the at least two microscopic images to obtain input to the machine learning algorithm can be performed by a computer-implemented pre-processing algorithm.

[0050] For example, the depth information can be used to create a topographic image of the intervention area, for example, by a pre-processing algorithm. The depth information can be generated, for example, by stereo imaging, time-of-flight imaging, and / or structured illumination. Such identification can also be performed before the combined image is identified, as will be explained below.

[0051] It would also be possible to identify a combined image based on a combination of at least two microscope images, and for a machine learning algorithm to take this combined image as input.

[0052] The combined image can include optical flow values ​​as differences between at least two microscopic images. Spatial information is generally removed in the case of optical flow images. Optical flow can encode the movement of image points. Optical flow can describe a vector field that specifies the velocity of visible points in object space, for example, in the reference frame of the imaging optical unit, projected onto the image plane. A corresponding combination algorithm for determining an optical flow image can operate, for example, on an image point-by-image point or block-by-block basis. One exemplary combination algorithm for obtaining an optical flow image as a specific form of combined image is described, for example, in Sun, D., Yang, S., Liu, M.Y., & Kautz, J. (2018). Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume in Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 8934-8943). Another exemplary combination algorithm for obtaining an optical flow image as a specific form of a combined image is described, for example, in Hui, T.W., Tang, X., & Loy, C.C. (2018) in Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 8981-8989). Another exemplary combination algorithm for obtaining an optical flow image as a specific form of a combined image is described, for example, in IIg.E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., & Brox, T. (2017). Flownet2.0: Evolution of optical flow estimation with deep networks in Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2462-2470).

[0053] In general, in various examples described herein, a combination algorithm can be used to identify a combined image with contrasting optical flow values, i.e., to reveal optical flow, based on two or more microscopic images. In this case, such a combination algorithm can take into account that the background may also have some movement in addition to the surgical instrument. However, in this case, it is also possible for the machine learning algorithm that creates the map to be able to distinguish the background from the surgical instrument based on the optical flow image. This is because the optical flow values ​​for the surgical instrument are typically characteristically different from the optical flow values ​​for the background. For example, a continuous, homogeneous area in the optical flow vector field indicates a surgical instrument; the background does not move or moves in a different direction than the instrument (e.g., when only one of the two instruments is in contact with the background); there is a large difference in the magnitude of the movement for the instrument compared to the background, i.e., the background movement is often smaller; and the optical flow gradient for the surgical instrument is large.

[0054] In order for the machine learning algorithm identifying the map and the combination algorithm identifying the combined image with optical flow values ​​as contrast to work well together, end-to-end training of both algorithms can be envisaged.

[0055] In various examples described herein, it is possible that the at least two microscope images are taken with the surgical microscope in the same position relative to the intervention area, i.e., with the imaging optical unit of the surgical microscope at the same distance and in the same orientation relative to the intervention area. In one such example, the combined image can be obtained directly by forming a difference between the at least two microscope images.

[0056] However, in other examples, the at least two microscopic images could be taken with the surgical microscope at different orientations relative to the intervention area. In one such example, the at least two microscopic images can be registered with a common coordinate system and then transformed to the common coordinate system based on this registration. The corresponding transformed at least two microscopic images can then be combined with each other to obtain a superimposed microscopic image.

[0057] Above we have described various variants of the machine learning algorithm in which the input is based on at least two microscopic images. Optionally, one or more inputs that are not based on at least two microscopic images can also be used.

[0058] For example, the machine learning algorithm can take the context of the surgical intervention as another input: the corresponding context information can indicate, for example, the type of surgery, the progress of the surgery, the surgeon performing the surgery, etc. Such supplementary information can be useful for intentionally controlling assistance functions.

[0059] In the various variations described herein, at least one control algorithm may be used that takes as input the map provided as output by the machine learning algorithm.

[0060] The at least one control algorithm can then provide as output control data for assistance functions. The control data can be transferred, for example, to one or more actuators of individual instruments (e.g., a surgical microscope) used during the surgical intervention. The control data can also be transferred to a human-machine interface, for example, to output user guidance to a surgeon performing the surgical intervention.

[0061] Generally speaking, the control data may define at least one of the type of assistance function and the time at which the assistance function is to be performed.

[0062] For example, it is conceivable that in principle different actions of the assistance function can be performed, and that different actions can be selected depending on the content of the map. Alternatively or additionally, depending on the content of the map, certain actions can be performed earlier or later.

[0063] The control algorithm receives the map as an input. In various examples, the control algorithm could also receive one or more other inputs. For example, the control algorithm could receive the context of the surgical intervention as another input. Details regarding the context of the surgical intervention have already been described above in relation to another input for the machine learning algorithm that provides the map as an output, and corresponding details apply with respect to the control algorithm.

[0064] The control algorithm can take into account a predetermined set of rules. For example, the set of rules can be transferred to the control algorithm as separate inputs. The set of rules can also be determined in advance.

[0065] The rules can define boundary conditions for the selection of control data depending on one or more features of the map. In this way, for example, continuity of the provision of assistance functions can be ensured. For example, sudden changes in the actions provided by the assistance functions can be avoided. Individual user preferences can also be taken into account.

[0066] The rules can, for example, prescribe a temporal smoothing of the operation of the assistance function. For this purpose, a corresponding low-pass filtering of any of the operations can be performed on the output side. The temporal evolution of multiple maps obtained successively by the machine learning algorithm can also be taken into account on the input side. For example, a corresponding low-pass filtering can be applied to the corresponding time series of maps obtained successively by the machine learning algorithm.

[0067] An example rule of the rule group may be, for example, "The operation of the assistance function is performed only if multiple surgical instruments are visible in the microscope image" or "When two or more surgical instruments are visible, the surgical microscope is not focused on one surgical instrument group that is not spatially adjacent to at least one other surgical instrument group."

[0068] The rules may also specify how to determine the order of the activity areas, for example, depending on the characteristics related to the context information of the activity areas as described above. The assistance function can then be controlled according to such order.

[0069] Exemplary rules that can be considered depending on such an order are, for example, "if the tool tips of multiple surgical instruments are spatially close, give more importance to the corresponding active area for further dependent decisions" or "if two surgical instruments are within an active area and therefore spatially close, then the corresponding active area will be given more weight than other active areas."

[0070] In general, the various examples described herein can control a wide variety of assistance functions. For example, assistance functions related to a surgical microscope that captures at least two microscope images can be controlled based on which map is created. However, it is also conceivable that assistance functions unrelated to the surgical microscope can be controlled, for example, such assistance functions can relate to other individual instruments used in connection with a surgical intervention or that allow user guidance via a human-machine interface.

[0071] For example, it could be envisaged that the assistance function may be selected from a group including the following elements: adjusting the position of the surgical microscope, changing the magnification of the surgical microscope, focusing the surgical microscope, adjusting the illumination of the intervention area by the illumination of the surgical microscope.

[0072] For example, it could be envisaged that the assistance function changes the position of the surgical microscope depending on the distance between at least one of the one or more active areas and the center of the field of view of the surgical microscope, i.e. in other words, the recognized active area can be positioned in the center of the field of view of the surgical microscope (called "auto-centering").

[0073] For example, a technique has been described above for defining the order of active areas, e.g., based on features that describe the context of the active areas, i.e., based on spatial and / or anatomical proximity relationships between the active areas or with respect to the patient's anatomy. Corresponding active areas can then be selected based on this order, and the surgical microscope can be centered with respect to the active areas. Such a technique is based on the insight that it is often particularly important for a surgeon to center a particular active area in the field of view in order to proceed with a surgical intervention there in a targeted manner. Therefore, assistance functions can be provided specifically and intentionally.

[0074] The computer program or computer program product or computer-readable storage medium includes program code. The program code can be loaded into and executed by a processor. When executed by the processor, the program code causes the processor to perform a method for controlling an assist function for a surgical intervention on a patient. The method includes applying a machine learning algorithm to obtain a map of an intervention area. The application is based on at least two microscopic images, the at least two microscopic images being taken by a surgical microscope and imaging the intervention area of ​​the surgical intervention. The at least two microscopic images being taken at different times. The method further includes using the map to control an assist function related to the surgical intervention. In this case, the map indicates one or more active areas of the intervention area that are associated with a higher probability of a surgical tool being present.

[0075] A control device for controlling an assistance function for a surgical intervention on a patient is configured to obtain a map of the intervention area in this way by applying a machine learning algorithm based on at least one microscopic image. At least two microscopic images are taken by a surgical microscope and depict the intervention area. The at least two microscopic images are taken at different times. The control device is further configured to control an assistance function related to the surgical intervention using the map. The map indicates one or more active areas of the intervention area that are associated with a higher probability of a surgical tool being present.

[0076] The system may include, for example, a control device and a surgical microscope.

[0077] The surgical microscope may have an actuator for changing the position of the surgical microscope.

[0078] The assistance functions may then include positioning of the surgical microscope.

[0079] The features mentioned above and those to be described below can be used not only in the corresponding combinations explicitly described, but also in other combinations or alone, without also departing from the scope of protection of the invention. [Brief explanation of the drawings]

[0080] [Figure 1] 1A-1C are schematic diagrams illustrating various exemplary surgical microscopes. [Figure 2] The surgical target area shows diagrammatically the relative positioning of the surgical microscope with respect to the target area in various examples. [Figure 3] 1 illustrates a schematic representation of an apparatus that can be used in connection with various examples described herein to evaluate microscopic images and / or control support functions. [Figure 4] 1 illustrates an exemplary data processing scheme. [Figure 5] 10A-10C show various examples of activity areas in corresponding maps. [Figure 6]1 shows further details relating to a map representing the activity area. [Figure 7] 1 is a flowchart of an exemplary method. [Figure 8] 1 is a flowchart of an exemplary method. DETAILED DESCRIPTION OF THE INVENTION

[0081] The above-mentioned characteristics, features and advantages of the present invention and the manner in which they are achieved will become more apparent and will be more clearly understood from the following description of exemplary embodiments which are set forth in more detail in conjunction with the drawings.

[0082] The present invention will be described in more detail below based on preferred embodiments with reference to the drawings. In the drawings, the same reference numerals indicate the same or similar elements. The drawings are schematic representations of various embodiments of the present invention. The elements shown in the drawings are not necessarily drawn to scale. Rather, the various elements shown in the drawings are drawn so that their functions and general purposes can be more easily understood by those skilled in the art. The connections and couplings between the functional units and elements shown in the drawings can also be implemented as indirect connections or couplings. The connections or couplings can be implemented either wired or wirelessly. The functional units can also be implemented as hardware, software, or a combination of codeware and software.

[0083] Various examples of the present invention relate to techniques for controlling assist functions during the performance of a surgical intervention. In the various examples described herein, various types of assist functions can be controlled.

[0084] For example, it may control an assistant function that provides user guidance to a surgeon performing a surgical intervention. It may also be possible to control a surgical microscope or other individual instruments used in the surgical intervention (e.g., a microinspection tool or a confocal endomicroscope). The assistant function may also provide different types of actions.

[0085] Various examples of the present invention relate to specifying settings of a surgical microscope during surgery. In various examples described herein, different settings of the surgical microscope related to support functions can be specified. For example, it would be conceivable to specify the relative positioning, i.e., distance and / or orientation (posture), of the surgical microscope with respect to the patient undergoing surgery. Alternatively or additionally, the optical settings of the surgical microscope can also be specified, such as magnification (zoom), illumination intensity, and / or contrast. The operating mode can also be specified, such as the use of indirect or direct illumination, or illumination with light of a specific wavelength. The fluorescence mode can be activated. The video settings can be set.

[0086] The various examples are based on the insight that manually identifying and applying settings, such as manually adjusting the position of a surgical microscope, can increase the mental and emotional burden on the surgeon performing the surgery. For example, manual adjustment requires a "free hand," which forces the surgeon to either place the surgical instrument or wait for a downtime to change the surgical instrument. If the surgical instrument is placed during the operation phase, the manual adjustment would interrupt the surgery. If the surgeon waits for an appropriate opportunity to adjust the position, there is a risk that the surgeon will perform the surgery, at least in part, with a surgical microscope that is not optimally configured. Positioning the surgical microscope can often require extensive experience. Therefore, the various examples described herein enable automatic identification of the surgical microscope settings with respect to the corresponding assistance functions, thereby relieving the surgeon of the burden.

[0087] In general, the assistance functions can be controlled fully or semi-automatically, for example by automatically providing corresponding settings as described above in connection with the surgical microscope or by outputting corresponding user prompts.

[0088] In various examples described herein, the assistive functions are controlled based on the recognition of surgical instruments in the interventional field of the surgical intervention. The surgical instruments can be recognized, but not necessarily located. As the surgeon works with the surgical instruments, the assistive functions can be aligned with the focus of the surgeon's concentration as he works with the surgical instruments. Recognition of the surgical instruments ensures that increased activity is actually due to intentional manipulation by the surgeon, and not due to artifacts or disturbances. In this way, continuous human-machine interaction is possible.

[0089] Reference implementations for recognizing surgical instruments in the interventional field are conceivable. For example, one example is described in U.S. Pat. No. 10,769,443 B. For example, a reference implementation is based on attaching reference markings to surgical instruments. The reference markings can then be tracked and recognized. For example, electromagnetic or optical tracking is conceivable. One drawback of such marker-based reference implementations is the need to attach reference markings to the surgical instruments. In addition, the expenditure of additional hardware may be required to capture corresponding data that allows tracking the position based on the recognized reference markings. A different class of reference implementations utilizes markerless recognition of surgical instruments. For example, image processing algorithms can be used to recognize different types of surgical instruments in microscopic images from a surgical microscope. However, such approaches may be relatively poor in robustness due to the wide variety of colors and shapes of surgical instruments. Therefore, parameterizing or training a corresponding algorithm for recognizing surgical instruments is complex and often error-prone. The recognized parameter space is also broad due to the large number of types of surgical instruments.

[0090] In both the marker-based and marker-free surgical instrument recognition cases, a robust technical solution typically requires consideration of changes in image characteristics, such as depth of field, lighting, shading, reflection, contrast, and varying fluorescence. Such changes in image characteristics can arise from different imaging modalities that can be activated or deactivated, depending on the type of surgical microscope used. However, different image characteristics can also arise, for example, from background lighting that varies, for example, between operating rooms.

[0091] In the various techniques described herein, the assist functions can be robustly controlled taking into account the possible positions of the surgical instrument, and this can be done independently of the type of tool, overcoming the drawbacks mentioned above with respect to the reference implementation.

[0092] According to various examples, therefore, explicit recognition and / or localization of surgical instruments and classification of surgical instrument types may not be performed, in contrast to the aforementioned reference implementation. Instead of such explicit recognition of surgical instruments, an implicit approach using active areas associated with a higher probability of the presence of a surgical instrument is used. Active areas can be derived from a comparison of at least two microscopic images. In particular, active areas can correspond to regions of at least two microscopic images that exhibit increased dynamic features, e.g., increased activity or increased activity. Such techniques are based on the insight that associated surgical instruments are typically moved and therefore appear with more dynamic features in a sequence of microscopic images. In particular, dynamic features are increased relative to a relatively static background, e.g., tissue or fixed surgical instruments, e.g., clamps. This means that temporal context can be utilized in recognizing active areas. In other words, increased activity can be identified by identifying increased dynamic features or increased activity in active areas, which may exhibit an increased amount of dynamic features or movement compared to other areas, i.e., non-active areas. The amount of increased activity can include, for example, dynamic features or amounts of movement, such as speed or distance of movement, or optical flow, generally a temporal and / or spatial context, that can be imaged in the at least two microscopic images, that are described by the movement of the surgical instrument within the activity area. Furthermore, a higher probability of the presence of a surgical instrument within the activity area can be determined based on the increased activity, or in other words, based on the determination of increased activity.

[0093] According to various examples, such active areas can be stored as features in a map of the intervention region, which allows extraction of, among other things, specific characteristics of the imaging modality, such as contrast, illumination, shadow casting, etc. This can be done by considering the relative changes in at least two consecutive microscopic images. Static characteristics, including those of the imaging modality, can be recognized and ignored in the construction of the map, resulting in a particularly robust control of the assistance functions.

[0094] Figure 1 shows a schematic representation of a surgical microscope 801 for surgery. In the example shown, the surgical microscope 801 includes an eyepiece 803. Through the eyepiece 803, the surgeon can observe a magnified image of an object that is within the field of view 804 of the surgical microscope 801. In the example shown, this is a patient 805 lying on an operating table.

[0095] Instead of or in addition to optical eyepieces, cameras providing microscopic images (digital surgical microscopes) can also be used.

[0096] A surgical instrument 808 is also provided as a man-machine interface, which can be embodied as a handle or a foot switch, for example. In the embodiment shown in FIG. 1, this is a handle. The surgical instrument 808 allows the movement of the eyepiece 803 fixed to the cross beam 850. A motor can be provided for automatically executing the movement based on control data and in accordance with the corresponding settings of the surgical microscope. The motor can also assist the movement caused by the handle 808.

[0097] Furthermore, a control device 809 for the surgical microscope 801 is also provided, which controls the operation of the compound microscope and the display of the image and additional information and data in the eyepiece 803. The control device 809 can perform interaction with the surgeon. For example, the control device 809 can change the settings of the surgical microscope based on appropriate control data. For this purpose, one or more actuators can be controlled, for example, to move a crossbeam, to change an optical unit, etc. The settings can also include digital post-processing of sensor data. The settings can also relate to data acquisition parameters, for example, for the captured digital images. It would also be possible to switch between different image sources or imaging modes depending on the settings. Such settings can be performed at least in part automatically by assistance functions.

[0098] The surgical microscope 801 may also include one or more other sensors 860, such as a motion sensor, a thermal imaging camera, a microphone, a surround camera, etc. Such other sensors 860 may also perform differently depending on the settings of the surgical microscope. Such sensors 860 may provide context information describing the context of the surgical intervention.

[0099] In neurosurgery, surgical microscopes are used to visualize the intervention area. Such intervention areas are often characterized by low structures within narrow cavities. Depending on the type of surgery, the surgical microscope needs to be adjusted to a new viewing direction for the intervention area approximately every minute, for example, because new blind spots are created by changes in the position of instruments. With currently available commercially available surgical microscopes, this requires the surgeon to manually adjust the system's position, i.e., by grasping a handle attached to the microscope and guiding the system to a new position (microscope position and orientation). In various examples described herein, assistance functions can assist or even automate such position adjustments. However, in this case, the assistance functions described herein are not limited to the corresponding position adjustments of the surgical microscope. Alternatively or additionally, it would be conceivable to change other settings of the surgical microscope, such as setting the magnification, adjusting the focus, adjusting the illumination, etc. In some examples, the assistance functions do not relate to the settings of the surgical microscope, but rather to the settings of another instrument, such as an operating table on which the patient lies, a microinspection tool, or a confocal endoscopic microscope. The man-machine interface can also be controlled with respect to the assistance functions to provide user guidance.

[0100] 2 shows an exemplary positioning of the optical unit 806 of a surgical microscope 801 relative to an original position 53 within a patient's calvaria 54, all of which define an intervention area 50 (see also FIG. 1). Additionally, surgical instruments 51, 52 positioned within the intervention area 50 are also shown.

[0101] 3 shows a schematic diagram of a device 90 that can be used for data processing in various examples described herein. For example, the device 90 can be a PC or a cloud server. The device 90 can implement a control device for controlling assistance functions. The device 90 can be part of a control device 809 of a surgical microscope 801.

[0102] The device 90 comprises a processor unit 91 and a non-volatile memory 92. The processor unit 91 is able to load program code from the non-volatile memory 92 and execute said code, having the effect that the processor unit 91 performs techniques according to examples described herein, e.g., applying a machine learning algorithm to obtain a map of the intervention area, whereby an active area is marked, training the machine learning algorithm based on training data, applying a pre-processing algorithm to a series of microscopic images, whereby the output of the pre-processing algorithm is input to the machine learning algorithm, applying a control algorithm that obtains the map as input to provide control data for an assistance function, etc.

[0103] Details regarding an exemplary implementation of data processing, for example by device 90, are described below with respect to FIG.

[0104] 4 shows aspects relating to data processing for controlling assistance functions according to various examples of the present invention. The data processing according to FIG. 4 can be performed, for example, by device 90, in particular by processor unit 91, on the basis of program code read from non-volatile memory 92.

[0105] 4 shows that two microscopic images 71, 72 are obtained, which can image the intervention area 50 and can be taken by a surgical microscope 801. The two microscopic images 71, 72 are used as input by a pre-processing algorithm 81 (although this is generally optional, and pre-processing can also be performed).

[0106] Generally, in the various examples described herein, more than two microscope images 71, 72 may be used to identify the map.

[0107] The optional pre-processing algorithm 81 provides an output 75, which in turn becomes the input for the machine learning algorithm 82. For example, the pre-processing algorithm 81 could identify a combined image based on the combination of two microscopic images 71, 72. The combination can be performed, for example, by difference formation or summation. The combined image shows the dynamic features between the different microscopic images 71, 72. Other information can also be extracted from the microscopic images 71, 72, for example depth information for a topographical image.

[0108] The machine learning algorithm 82 takes as input the output 75 of the preprocessing algorithm 81, such as the combined image or topographic image described above. The machine learning algorithm can also take other inputs. Figure 4 shows that the machine learning algorithm 82 takes both microscopic images 71, 72 as other inputs (although if no preprocessing is performed, the machine learning algorithm 82 could also take only the microscopic images 71, 72).

[0109] Additionally, the machine learning algorithm 82 can receive as another input, see FIG. 1 , sensor data 78 from additional sensors of the surgical microscope, such as sensor 860. Additionally or alternatively, the machine learning algorithm 82 can also receive state data 79 that describes the context of the surgical intervention. For example, other microscope images, such as those taken by an endoscope, can also be received.

[0110] The machine learning algorithm 82 provides a map 76, which maps the intervention area 50 by using features of a predetermined specific feature class. In particular, one or more activity areas associated with a higher probability of the presence of a surgical tool are recorded in the map 76. The map can then be used to control an assistance function. For this purpose, in various examples, a control algorithm 83 can be applied to the map 76, as also shown in FIG. 4, and control data 77 can thus be obtained to appropriately configure the surgical microscope 801.

[0111] 4 shows that the control algorithm 83 can take state data 79 describing the context of the surgical intervention as another input instead of or in addition to the machine learning algorithm 82, thereby enabling context-dependent control of the assistance functions.

[0112] In general, the control algorithm 83 can take into account a set of predefined rules that define boundary conditions for the control data selection depending on one or more characteristics of the map 76. The rules can, for example, define the temporal smoothing of the operation of the assistance function. The rules can, for example, define the order of the active areas, which can in turn affect the control of the assistance function.

[0113] The data processing in Fig. 4 is essentially modular. For example, the pre-processing algorithm 81 is essentially optional. For example, it would be conceivable for the machine learning algorithm 82 to directly receive the microscopic images 71, 72 as input. The control algorithm 83 can be incorporated into the machine learning algorithm 82. In this regard, Fig. 4 merely shows one exemplary architecture for data processing.

[0114] Figure 5 shows an embodiment related to map 76. Three activity areas 101-103 are recorded in map 76 (outlines of activity areas 101-103 are shown in the example of Figure 5). One or more surgical instruments are located with a higher probability within each of the activity areas.

[0115] However, the map 76 does not distinguish between different types of surgical instruments. Therefore, the assistance functions are also not controlled by different types of surgical instruments. The map 76 therefore includes morphological features of the activity areas 101-103 that explain the presence of surgical instruments at different positions within the intervention area 50. On the other hand, the surgical instruments do not need to be localized based on object recognition (in other words, the map 76 can be created independently with respect to the cause of increased activity, thereby improving robustness).

[0116] In addition to the features of the activity areas 101-103, the map 76 in the illustrated example also includes other features, namely features in the form of markings 111-112, which are generally optional. These features describe the contextual information of the activity areas. The activity area 102 is marked with marking 111, and the activity area 103 is marked with both markings 111 and 112.

[0117] In this case, the markings 111 identify that the respective marked activity areas 102, 103 satisfy a spatial adjacency relationship. In the illustrated example, the two activity areas 102, 103 are located near each other, e.g., closer than a certain predetermined threshold, and therefore satisfy the corresponding spatial adjacency relationship. In contrast, the activity area 101 does not have any particularly close neighbors and therefore does not satisfy the corresponding spatial adjacency relationship.

[0118] The markings 112 identify that the corresponding active area 103 satisfies the corresponding anatomical neighbor markings. For example, it could be envisioned that the active area 103 is located particularly close to a particular anatomical structure, such as particularly close to the original position 53. For example, the anatomical feature can be located as an anatomical context by the pre-processing algorithm 81 and transferred as another input to the machine learning algorithm 82.

[0119] Generally, the markings 111-112 provide contextual information related to the active areas 101-103. Such contextual information can be taken into account when controlling assistance functions. In various examples described herein, it can be envisaged that the active areas 101-103 of the map 76 are sorted based on such contextual information and that a corresponding order is used to control the assistance functions. For example, rules of the control algorithm can specify how the order is determined, i.e., which contextual information is taken into account when determining the order, or how different features describing the contextual information of the map 76 are weighted.

[0120] Although spatial and anatomical adjacencies have been described above as examples of contextual information, in general, other or further types of contextual information can be taken into account. For example, the map 76 can include the number of surgical instruments in each of the active areas 101-103 and / or other features indicative of dynamic characteristics within each of the active areas 101-103. All of these and other types of contextual information can be taken into account when controlling assistance functions, for example, when specifying a sequence.

[0121] Figure 6 shows aspects relating to the activity areas 101-103. In particular, Figure 6 shows, for example, the activity area 101, how the activity areas are depicted in the map 76. In the example of Figure 6, a mask 151 is first used for the activity area 101. The mask 151 performs a binary division into an inside region and an outside region of the activity area 101, and the outline of the mask 151 is shown.

[0122] Alternatively or additionally to the use of such a mask 151, the map 76 can also include a probability value 152 for the presence or absence of a surgical tool, which is represented by a corresponding contour line in Figure 6. The active area can then be defined as that part of the intervention region 50 where the probability value for the presence of a surgical tool is higher than 50% (or any other threshold).

[0123] One exemplary method is shown in Figure 7. The method of Figure 7 illustrates various stages involved in the operation of the machine learning algorithm 82.

[0124] In box 3005, training of the machine learning algorithm 82 occurs. To this end, the ground truth is used to adjust the parameters of the machine learning algorithm 82 in a numerical iterative optimization process.

[0125] In box 3010, the currently trained machine learning algorithm is used for inference, i.e. the assistance function is controlled during the surgical intervention in the absence of a correct answer.

[0126] The training in box 3005 can be performed supervised, semi-supervised, or unsupervised: for example, the activity areas and optionally other features of the map can be manually annotated, thus obtaining a ground truth, which can then be used to adjust the parameters of the machine learning algorithm 82 based on the corresponding labels.

[0127] An exemplary method is shown in Figure 8. The method of Figure 8 may be performed by a data processing system. For example, the method of Figure 8 may be performed by an apparatus 90, such as a processor unit 91, based on program code read from a memory 92.

[0128] Figure 8 illustrates aspects related to the inference phase of box 3010 of the method of Figure 7. In Figure 8, dashed boxes are optional.

[0129] Image capture of two or more microscope images occurs in box 3050. Stereo microscope images can be captured.

[0130] For this purpose, for example, suitable control commands can be sent to a surgical microscope, which can then receive image data.

[0131] The two or more microscopic images can be taken at a predetermined orientation relative to the intervention area, for example at a fixed orientation.

[0132] In box 3055, pre-processing of the microscopic images from box 3050 is optionally performed. For example, optical flow can be extracted from the microscopic images obtained from box 3050. Generally speaking, the temporal context between the microscopic images can be identified.

[0133] A corresponding technique has already been described, for example, in connection with FIG.

[0134] In box 3060, a map is created that maps different feature types related to the intervention area. Machine learning algorithms are used for this purpose. One example would be a so-called "saliency" prediction algorithm. In particular, the activity areas are mapped as already described in connection with FIG. 5.

[0135] Generally speaking, the map can encode in a spatially related manner whether or not, and if appropriate what the probability is, that dynamic activity of the surgical tool is present at a particular spatial point within the intervention region, e.g., a particular xy pixel, or generally in a particular area. For example, a probability map can be output as described with respect to FIG. 6.

[0136] Optionally, contextual information about the activity areas can also be mapped, such as dynamic features within each activity area, the number of surgical instruments within each activity area, spatial adjacencies between activity areas, anatomical adjacencies of activity areas, etc.

[0137] Based on the map from box 3060, the assistance function can optionally be controlled in box 3065. For example, a decision can be made whether a particular action of the assistance function should be initiated. If a particular action is initiated, this can be implemented as a recommendation and / or request to the user, semi-automatic assistance, or fully automatic assistance.

[0138] In some examples, box 3060 can check whether one or more surgical instruments are recognized in the microscope image. This can be done based on an object recognition algorithm. Localization of the surgical instruments does not have to be performed. If one or more surgical instruments are recognized, then an assistance function can be initiated. For example, the object recognition algorithm can output "yes" if one or more surgical instruments are recognized in the microscope image. The object recognition algorithm can output "no" if no surgical instruments are recognized. In other words, this corresponds to a binary result. The number of recognized surgical instruments can also be output. The location does not have to be indicated.

[0139] In some instances, instead of or in addition to the initiation criterion "surgical instrument is recognized", it would be conceivable to make the initiation of the assistance function dependent on one or more other initiation criteria. One example would be a voice command, e.g. "centering", or the operation of a button, e.g. a foot switch.

[0140] Such an object recognition algorithm can be used in parallel in time, and for example independently, with respect to, for example, the algorithm that identifies the map in box 3060 (see algorithm 82 in FIG. 4).

[0141] Therefore, an object recognition algorithm can be used to validate the decision as to whether an increase in activity in an active area actually results from the surgeon's use of a surgical tool, or whether it is due to, for example, image artifacts or variable shadow casting or other disturbances. Only if a surgical tool is recognized can it be assumed that the active area is identified by the surgeon's activity. This can serve as a trigger criterion for an assistive function. For the assistive function, a control algorithm can be used that receives, for example, the map from box 3060 as input. A corresponding technique has been described, for example, in connection with control algorithm 83 of FIG. 4.

[0142] The assistance function can, for example, center the active area in the field of view of the microscope, for example, by controlling a robot arm for adjusting the position of the microscope.

[0143] Boxes 3050 to 3065 can be executed repeatedly to control the assistance functions in an updated manner each time.

[0144] Thus, in summary, a technique has been described that allows for implicit recognition of activities in an intervention area during a surgical intervention. The semantic meaning of the corresponding activities can be extracted, without the need to explicitly distinguish between different types of surgical instruments. A corresponding map can indicate the activity area. Assistance functions can then be controlled depending on the map. For example, a check can be performed to determine whether one or more actions of the assistance function should be initiated.

[0145] It has been described how microscopic images can be taken by a surgical microscope. The microscopic images can be taken at different times.

[0146] Based on these microscopic images, it is possible to identify a combined image that shows the optical flow as the amount of change between the two microscopic images. The microscopic images can also be sent directly to a machine learning algorithm. The microscopic images can also be fused with the combined image.

[0147] A control algorithm can be used to control the assistance function. Said algorithm can for example take into account one or more predetermined rules. An example of a rule is that a certain operation of the assistance function is performed only if two or more activity areas are recognized, optionally satisfying a certain spatial proximity relationship.

[0148] It goes without saying that the features of the above-described embodiments and aspects of the invention can be combined with one another, and in particular features can be used not only in the combinations described, but also in other combinations or on their own, without departing from the scope of the invention.

[0149] For example, a technique has been described above in which a map showing one or more activity areas is created for one intervention area. In general, it is possible to create multiple maps for multiple intervention areas, which appear next to each other in, for example, a microscopic image. To this end, the technique described herein can be applied in each case to each of the multiple maps.

[0150] Furthermore, techniques have been described above in which two or more microscopic images taken at different times are used. Essentially, a series of microscopic images taken within a time range can be used, some of which are taken at different times within that time range, and optionally some of which are taken at the same time within that time range. This means that for at least some of the times within the time range, there may be redundant information, for example as a result of multiple microscopic images being taken with different imaging optical units or using multiple imaging modalities. Multimodal imaging is possible. Such multimodal and / or redundantly taken microscopic images can then be fused.

Claims

1. A method for operating a control device, comprising: - said control device applies a machine learning algorithm (82) to obtain a map (76) of the intervention area (50) based on at least two microscopic images (71, 72) taken at different times and taken by a surgical microscope (801) and imaging said intervention area (50); - said control device controlling assistance functions related to said surgical intervention by using said map (76); Including, the map (76) indicates one or more areas (101, 102, 103) of activity within the intervention region (50) that have experienced increased activity and that are associated with an increased probability of the presence of a surgical instrument (51, 52) as a result of the increased activity; The method wherein the assistance function is selected from the group consisting of the following elements: a step in which the assistance function adjusts the position of the surgical microscope (801); a step in which the assistance function changes the magnification of the surgical microscope (801); a step in which the assistance function adjusts the focus of the surgical microscope (801); and a step in which the assistance function adjusts the illumination of the intervention area by the surgical microscope (801).

2. the map (76) is independent of different types of surgical instruments (51, 52); The assist function is controlled independently of the type of the surgical instrument (51, 52). The method of claim 1.

3. the map (76) includes probability values ​​(152) for the presence or absence of surgical instruments (51, 52) for a plurality of areas of the intervention region (50); The one or more activity areas (101, 102, 103) are located in the areas with higher probability values ​​(152).

3. The method according to claim 1 or 2.

4. the map (76) includes a mask (151) for the one or more activity areas (101, 102, 103); The method according to any one of claims 1 to 3.

5. The method according to any one of claims 1 to 4, wherein the map (76) comprises one or more features (111, 112) indicating contextual information about the one or more activity areas (101, 102, 103).

6. the one or more features (111) comprising context information of the one or more activity areas (101, 102, 103) indicate that activity areas of the one or more activity areas (101, 102, 103) satisfy one or more spatial adjacency relationships; and / or the one or more features (112) comprising context information of the one or more active areas (101, 102, 103) indicate that one or more anatomical adjacency relations of the one or more active areas (101, 102, 103) are satisfied; and / or the one or more features comprising context information of the one or more activity areas (101, 102, 103) are indicative of the number of surgical instruments (51, 52) in the one or more activity areas (101, 102, 103); and / or the one or more features comprising context information of the one or more activity areas (101, 102, 103) are indicative of dynamic characteristics in the one or more activity areas (101, 102, 103); The method of claim 5.

7. moreover, - the control device determining the order of the one or more activity areas (101, 102, 103) based on the one or more features comprising context information of the one or more activity areas (101, 102, 103); The support function is controlled according to the order.

7. The method according to claim 5 or 6.

8. the machine learning algorithm (82) is determined, for example, based on a combination of the at least two microscopic images (71, 72) and receives as input a combined image (75) encoding the optical flow and / or the at least two microscopic images (71, 72) and / or the context of the surgical intervention, The method according to any one of claims 1 to 7.

9. The step of the control device controlling the assistance function related to the surgical intervention by using the map (76) comprises: - the control device applies at least one control algorithm (83) which takes as input said map (76) to obtain control data (77) for said assistance function, The method according to any one of claims 1 to 8.

10. the control algorithm (83) takes as another input the context of the surgical intervention; and / or the at least one control algorithm takes into account a set of predetermined rules defining boundary conditions for selecting the control data depending on one or more characteristics of the map (76).

10. The method of claim 9.

11. moreover, - the control device recognizes the presence of a surgical instrument in the at least two microscopic images by means of an object recognition algorithm, the assistance function is initiated only if the presence of a surgical instrument is recognized in the at least two microscopic images; The method according to any one of claims 1 to 10.

12. the object recognition algorithm does not localize the surgical instrument in the at least two microscopic images. The method of claim 11.

13. A control device for controlling an assist function for surgical intervention on a patient, - applying a machine learning algorithm (82) to obtain a map (76) of the intervention area (50) based on at least two microscopic images (71, 72) taken at different times and taken by a surgical microscope (801) and imaging the intervention area (50) of said surgical intervention; - controlling assistance functions related to said surgical intervention by using said map (76) configured to run the map (76) indicates one or more areas (101, 102, 103) of activity within the intervention region (50) that are associated with increased activity and a higher probability of the presence of a surgical instrument (51, 52) as a result of the increased activity; The assistance function is a control device selected from the group consisting of the following elements: a step in which the assistance function adjusts the position of the surgical microscope (801), a step in which the assistance function changes the magnification of the surgical microscope (801), a step in which the assistance function adjusts the focus of the surgical microscope (801), and a step in which the assistance function adjusts the illumination of the intervention area by the surgical microscope (801).

14. - the control device according to claim 13; - said surgical microscope (801) equipped with an actuator for adjusting the position of said surgical microscope; In a system including The system wherein the support function includes adjusting the position of the surgical microscope.

Images