Order of priority determination of plurality of objects in support function for surgical microscope observation system

Abstract

To determine an order of priority of a plurality of objects in a support function for a surgical microscope observation system.SOLUTION: A technology is related to execution of a support function for a surgical microscope observation system. The support function, for example, of automatic centering or measurement of a surgical instrument is executed in association with a plurality of objects (231, 232, 233) shown in an image (220) photographed by the surgical microscope observation system. The order of priority information (241, 242, 243) related to the objects (231, 232, 233) are considered to perform categorization into related and unrelated objects, for example.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] Various examples of the present disclosure relate to techniques for performing support functions for a surgical microscopy system, and in particular to prioritizing different objects related to such support functions. [Background technology]

[0002] Medical surgical microscopy observation systems (also called robotic visualization systems or surgical visualization systems) having a robotic stand for positioning a microscope are known from the prior art, see for example US Pat. No. 5,399,633.

[0003] The robot stand here can be controlled manually. However, there are also known techniques in which the robot stand is controlled automatically, for example to allow automatic centering and / or automatic focusing on a particular object, such as a surgical instrument (also called surgical equipment or surgical tool or surgical instrument). A user command in this case triggers a positioning procedure, in which the robot stand and / or the objective optical unit of the microscope are driven. Such a technique is known, for example, from US Pat. No. 5,399,323.

[0004] It has been observed that, particularly in relatively complex surgical situations, such as those involving numerous surgical instruments and / or different types of surgical instruments, such techniques according to the prior art can lead to undesirable results, for example positioning may occur with the wrong surgical instrument, which was never intended by the surgeon.

[0005] To remedy such drawbacks, for example, US Pat. No. 5,399,633 discloses identifying important surgical instruments from among a group of visible surgical instruments. Such techniques also have certain drawbacks and limitations. For example, the technique described in this patent has been found to be unsuccessful with some types of surgical instruments. This means that good results are not achieved depending on the type of surgical instrument. Due to the large number of surgical instruments available, this can reduce the system's tolerance and lead to unexpected behavior. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] German Patent Publication No. 102022100626A1 [Patent Document 2] U.S. Patent No. 10,456,035B2 [Patent Document 3] U.S. Patent No. 10,769,443B [Patent Document 4] International Publication No. 2022 / 161930A1 Brochure Summary of the Invention [Problem to be solved by the invention]

[0007] There is therefore a need for improved techniques for assistive functions for surgical microscopy systems, and in particular for techniques that can reliably classify the priority of objects in an image to which an assistive function can be associated. Objects need to be classified as either important or auxiliary. [Means for solving the problem]

[0008] This object is achieved by means of the features of the independent patent claims. The features of the dependent claims define embodiments.

[0009] The various examples are based on the observation that techniques for identifying important surgical instruments as known in the prior art provide low robustness for various manifestations of surgical instruments, especially due to the presence of a large number of different types of surgical instruments in various surgical environments, e.g., in the range of three orders of magnitude. The various examples are based on the observation that it is difficult to reliably distinguish such a large number of possible classification outcomes within a classification of surgical instrument types using an algorithm. For example, when using a machine learning classification model, out-of-distribution situations can often occur where the occurrence of a particular type of surgical instrument was not considered during the training of the machine learning classification model. The model may then produce unexpected results.

[0010] In addition, the various examples are based on the observation that techniques for identifying important surgical instruments as known in the prior art overlook what may be important in some cases when a particular type of surgical instrument is present, but may be auxiliary in other cases. It has been found that consideration of context can help with priority shortcomings.

[0011] Aspects relating to a surgical microscopy system having a robotic stand and a microscope carried by the robotic stand are described below. In particular, techniques relating to assist functions for assisting a surgeon are described. The assist functions herein are performed with respect to at least one object shown in a corresponding image (e.g., captured by a microscope camera or an environmental camera). Examples of such assist functions are auto-positioning, auto-centering, auto-orienting, auto-zooming, or auto-focusing with respect to one or more objects, other examples include, for example, measuring one or more objects.

[0012] Techniques are disclosed for separating relevant and irrelevant objects from each other. Relevant objects are sometimes referred to as important objects.

[0013] More generally, techniques are described regarding how priority information for various objects can be identified. Assist functions can then be executed taking this priority information into account. For example, an assist function may consider only objects that are classified as relevant (i.e., high priority) based on the priority information. A corresponding hierarchical identification of priority information may also be used, whereby high priority objects are given more weight in the execution of the assist function.

[0014] The determination of priority information may generally correspond to a regression or classification task.

[0015] The priority information is determined based on the movement information of the various objects according to the various disclosed variants. The use of the movement information is particularly advantageous in relation to the reference implementation, in which object type classification (e.g. instance segmentation and classification of the associated instances) must be performed. A robust prioritization can be achieved especially for many different scenes that were previously unknown. For example, there is no need to parameterize a classification model for identifying the object type. For example, there is no need to train any corresponding machine learning classification model, which can obviate the need for a complex training campaign for collecting images of various types of objects. The priority information can be obtained without any classification, in particular without any instance segmentation of the objects.

[0016] Nevertheless, in some variants it is envisaged that further information may also be taken into account when determining the priority information, such as semantic context information about the imaged scene or whether a particular surgical instrument is being held in the left or right hand.

[0017] A computer-implemented method for controlling a surgical microscopy system is disclosed. The surgical microscopy system includes a stand. The surgical microscopy system also includes a microscope. The microscope is carried by the stand.

[0018] For example, the stand can be a robotic stand or a partial robotic stand.

[0019] The method includes driving cameras of a surgical microscopy system. For example, a microscope camera and an environmental camera can be driven. By driving the cameras, an image sequence is obtained. For example, the image sequence can thus correspond to a time sequence of images showing a scene.

[0020] The method also includes determining motion information for each of the two or more objects shown in the sequence of images. The motion information is determined based on the sequence of images.

[0021] For example, a heuristic or machine learning model may be used to identify the movement information, e.g., a threshold comparison may be made between one or more variables indicated by the movement information and one or more predefined thresholds.

[0022] For example, the movement information may be representative of an optical flow. Alternatively or additionally, the movement information may be representative of active regions, i.e. regions with relatively large changes in contrast between image sequences. It may be envisaged that the movement information is representative of a movement pattern of an object. For example, the movement information may be representative of the movement amplitude and / or the movement frequency of an object. Combinations of such contents of the above mentioned movement information are also envisaged.

[0023] It would be conceivable (but not essential) that the movement information is related to the positioning of the objects in the image sequence, for example it would be possible to first locate the objects in the image sequence and then determine corresponding movement information for each located object.

[0024] The method further includes determining priority information for two or more objects based on the movement information. For example, the priority information indicates which objects are important, i.e., relevant, and which objects are auxiliary, i.e., irrelevant. In addition to such binary categorical assignments of various objects as part of the priority information, multi-dimensional class assignments or regressions could also be envisioned. For example, a priority value could be output, e.g., ranging from 0 (=irrelevant) to 10 (=relevant).

[0025] Then, based on the priority information, the assist function is performed in relation to two or more objects, which may mean, for example, that an object with a higher (lower) priority is given more (less) weight as part of the assist function.

[0026] As part of the assistance function, for example, a robotic stand may be driven to perform automatic positioning with respect to reference points identified, for example, based on one or more objects. As part of the assistance function, the appearance of one or more objects in one or more images, for example, microscopic images, may be evaluated.

[0027] A data processing apparatus is disclosed, the data processing apparatus being configured to control a surgical microscopy system. The data processing apparatus includes a processor. The processor is configured to load and execute program code from a memory. Execution of the program code causes the processor to perform the above-mentioned method for controlling a surgical microscopy system.

[0028] A surgical microscopy system including such a data processing device is also disclosed.

[0029] The features mentioned above and those described below may be used not only in the corresponding combinations explicitly stated, but also in other combinations or alone, without departing from the scope of protection of the invention. [Brief description of the drawings]

[0030] [Figure 1] 1A-1C are schematic diagrams illustrating various exemplary surgical microscopy systems; [Diagram 2] 1 illustrates a schematic of different fields of view for a microscope and an environmental camera of an exemplary surgical microscopy system. [Diagram 3] 1 is a flow chart of an exemplary method. [Figure 4] 1 is a flow chart of an exemplary method. [Diagram 5] 1 shows an image captured by a camera in which several surgical instruments are visible. [Figure 6] Corresponding to FIG. 5, priority information for various surgical instruments is additionally displayed. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] The above-mentioned characteristics, features and advantages of the present invention and the manner in which they are realized will become more apparent and be more clearly understood in connection with the following description of exemplary embodiments which are set forth in more detail in conjunction with the drawings.

[0032] The present invention will be described in more detail below based on preferred embodiments with reference to the drawings, in which the same reference numerals refer to the same or similar elements. The drawings are schematic representations of various embodiments of the present invention. The illustrated elements are not necessarily drawn to scale. Rather, the various illustrated elements are drawn in such a way that their function and general purpose are understandable to those skilled in the art. The illustrated functional units and connections and couplings between the elements may also be implemented as indirect connections or couplings. The connections or couplings may be implemented in a wired or wireless manner. The functional units may be implemented as hardware, software, or a combination of hardware and software.

[0033] Techniques relating to the operation of a surgical microscopy system are described below. The techniques described allow for the performance of an assistive function, such as an automatic configuration of one or more components of the surgical microscopy system. Typically, the assistive function is performed with respect to at least one object of a plurality of objects visible in an image captured by the surgical microscopy system.

[0034] According to various examples, priority information is determined for the objects, where the priority information is determined based on the movement information of the objects.

[0035] 1 shows a schematic embodiment of an exemplary surgical microscopy system 80. The surgical microscopy system 80 is used for the generation of microscopic images of an observation area during a surgical intervention. For this purpose, a patient 79 is placed on an operating table 70. A position 78 is shown with a surgical instrument 78.

[0036] The surgical microscopy system 80 includes a robot stand 82, which carries a positionable head portion 81. The robot stand 82 may have different degrees of freedom for each variant. Robot stands 82 having six degrees of freedom for positioning the head portion 81 are known, namely translation along each of the x-, y-, and z-axes, and rotation about each of the z-, y-, and z-axes. The robot stand 82 may have a handle 82a, as shown in FIG. 1.

[0037] Although FIG. 1 shows a robotic stand 82, it would generally be possible to use a stand that is only partially robotic or manual.

[0038] The head part 81 comprises a microscope 84 with optical components 85, such as an illumination optical unit, an objective optical unit, a zoom optical unit, etc. The microscope 84 further comprises, in the example shown, a microscope camera 86 (here a two-channel stereo camera, but a single optical unit can also be envisaged), by means of which an image of the observation area can be taken and which can be reproduced, for example, on a screen 69. The microscope 84 is therefore also called a digital microscope. The field of view 123 of the microscope camera 86 is also shown.

[0039] In the example of Fig. 1, microscope 84 also includes an eyepiece 87 having an associated field of view 122. Eyepiece 87 is therefore optional. For example, the detection beam path may be split by a beam splitter, whereby an image may be captured by camera 86 and viewed through eyepiece 87. Not all variants of microscope 84 have an eyepiece. Purely digital microscopes 84 without an eyepiece are also possible.

[0040] In the example of FIG. 1, the head part 81 carried by the stand 82 of the surgical microscopy system 80 also includes an environmental camera 83. The environmental camera 83 is optional. Although the environmental camera 83 in the example of FIG. 1 is shown as being integrated into the microscope 84, it would be possible to arrange it separately from the microscope 84. For example, the environmental camera may be a CCD camera. The environmental camera may also have depth resolution. Instead of or in addition to the environmental camera, it would be envisaged that there are other auxiliary sensors, for example distance sensors (for example a time-of-flight camera or an ultrasonic sensor or a sensor using structured illumination). The field of view 121 of the environmental camera 83 is shown in FIG. 1.

[0041] Therefore, the surgeon has multiple options to view the observation area, namely, using the eyepiece 87, using the microscopic image captured by the camera 86, or using the overview image captured by the environmental camera 83. The surgeon can also view the observation area directly (without magnification).

[0042] Next, aspects relating to the fields of view 121, 122, and 123 will be described.

[0043] Fig. 2 shows various aspects of the field of view. Fig. 2 shows the field of view 121 of the environmental camera of the surgical microscopy observation system 80. Thus, the overview image shows a relatively wide area. In addition, the field of view 122 of the eyepiece and the field of view 123 of the microscope camera 86 are shown as examples. The fields of view 121, 122, 123 do not have to be arranged in a centered state with respect to each other.

[0044] Referring again to FIG. 1, various components of the surgical microscopy system 80 , such as the robotic stand 82 , the microscope 84 , or one or more other components, such as an environmental camera 83 , are controlled by a processor 61 of the data processing device 60 .

[0045] The processor 61 may be designed, for example, as a general central processor (CPU) and / or a field programmable logic module (FPGA) and / or an application specific integrated circuit (ASIC). The processor 61 is capable of loading program code from the memory 62 and executing it.

[0046] The processor 61 can communicate with various components of the surgical microscopy system 80 via the communication interface 64. For example, the processor 61 can drive the stand 82 to move the head portion 81, for example, translationally and / or rotationally, relative to the surgical table 70. The processor 71 can change the zoom and / or focus, for example, by driving the optical components 85 of the microscope 84. Images from the environmental camera, if any, can be retrieved and evaluated. In general, the images can be evaluated and an assistance function can be performed based on this evaluation.

[0047] The data processing device 60 further includes a user interface 63. The user interface 63 may be used to receive commands from a surgeon, or generally from a user of the surgical microscopy system 80. The user interface 63 may have a variety of configurations. For example, the user interface 63 may include one or more of the following components: a handle on the head portion 81, a foot switch, voice input, input via a graphical user interface, etc. It would also be possible for the user interface 63 to provide graphical interaction via menus and buttons on the monitor 69.

[0048] Below is described a technique as to how an assistive function can be provided by the surgical microscopy system 80. An assistive function can be requested, for example, by a user command.

[0049] In general, such a user command may take various forms. For example, the user command may specifically identify a particular object. For example, the user may specify by voice command "auto-center on surgical instrument 1". Such a user command therefore specifies exactly the object for which (here for "surgical instrument 1") the surgical microscopy system 80 is to be configured. However, in other examples, it could be envisaged that the user command is not specific to a particular object. For example, the user may specify "auto-center" by voice command or by pressing a button. The object for which auto-centering should be performed is therefore not specified (even if multiple objects that are candidates for auto-centering are visible). The user command is not explicit in this example. The user command does not distinguish between multiple visible surgical instruments. In the reference implementation, the user command may then be interpreted incorrectly, e.g. auto-centering may be performed on the wrong surgical instrument. User expectations (e.g. automatic centering on "surgical instrument 1") may then not correspond to the actual system behavior (e.g. automatic centering on the geometric center of all visible surgical instruments).

[0050] In the following, a technique is described that allows for a better adaptation of user expectations than in the case of the reference implementation with regard to assistance functions triggered by user commands. A deterministic system behavior is made possible, which provides reproducible and comprehensive results in different situations and / or despite different scenes. Such a technique is based on the observation that the system behavior of an automated control system must precisely adapt to the user's expectations, especially in stressful situations under time pressure, such as those typically occurring in a surgical environment.

[0051] Figure 3 is a flow chart of an exemplary method relating to a technique involving configuring a surgical microscopy system to image an object in response to a user command.

[0052] The method of Fig. 3 may be executed by a processor of a data processing device, for example by the processor 61 of the data processing device 60 of the example surgical microscopy system 80 of Fig. 1. The processor is for this purpose capable of loading program code from a memory and executing it.

[0053] In operation 3005, a user command is received. The user command may request an assistive function. For example, the user command may implicitly or explicitly request that the microscope camera be configured to image an object, such as a surgical instrument. The user command is received from a user interface. For example, the user command may request automatic alignment or automatic focusing. The user command may request a measurement of the surgical instrument.

[0054] A user command may not be specific to a particular object: a user command may not specify which of multiple visible objects it relates to.

[0055] In process 3010 (optional process) a microscope image is taken. For this purpose, a microscope camera of the microscope is activated, see microscope camera 87 of microscope 84 in the example of Fig. 1. As already mentioned with respect to Fig. 2, the microscope camera typically has a relatively small field of view, i.e. smaller than the field of view of the eyepiece or the environmental camera (if present).

[0056] In process 3015 (optional process), the object (e.g., a surgical instrument) identified by a user command is located in the microscope image from process 3010. If the object is already in the field of view of the microscope camera, the object is found in process 3015, i.e., it is visible in the microscope image, then process 3020 is performed, which may involve performing an assistive function based on the microscope image (e.g., auto-centering or auto-focusing, or measuring the object).

[0057] It may also happen that the object is not found in the microscope image in process 3015. This means that the object is not in the central area of ​​the scene imaged by the microscope camera. In such a case, the environment camera is activated to take an overview image in process 3025. This is done to check whether the object is in the peripheral area of ​​the scene imaged by the environment camera but not the microscope camera.

[0058] Process 3030 can then determine whether the object is visible in the overview image or not. Process 3030 can then determine whether the object is in the peripheral region, i.e. the region of the scene that is covered by the field of view of the environmental camera and not covered by the field of view of the microscope (in FIG. 2 this is the region outside field of view 123 and in field of view 121).

[0059] Generally, the object is thus located in the overview image. If the object cannot be found in the overview image, an error is output in operation 3035. Otherwise, operation 3040 is executed.

[0060] In process 3040, control commands are provided to the robotic stand to move the microscope so that the object is positioned within the field of view of the microscope camera, i.e., within the central region of the scene, meaning that in process 3040, a rough alignment is performed so that the surgical instrument is also visible in the microscope image taken in another iteration 3041 of process 3010.

[0061] In summary, the method of Fig. 3 thus allows first taking an overview image using the environmental camera to locate a surgical instrument for an assist function. In this case, this overview image from the environmental camera is first evaluated to determine whether it contains a surgical instrument. The robot stand is then driven to move the recognized surgical instrument into the field of view of the microscope camera. The actual assist function can then be performed based on the tabulation of one or more microscope images.

[0062] At times, it may be the case that the user command from operation 3005 does not precisely specify at least one object for which an assist function is to be performed in operation 3020. For example, four surgical instruments are visible, all of which are candidates for an assist function (e.g., auto-centering). Thus, it is initially unclear for which of the visible surgical instruments a search is to be performed in operation 3030 and then located in operation 3040. To solve such problems, the techniques described below, particularly with reference to FIG. 4, can be applied.

[0063] Fig. 4 is a flow chart of an exemplary method. The method of Fig. 4 relates to a technique for preparation for executing an assistance function. The assistance function may for example use the position and / or orientation (also called positioning, pose or absolute position) and / or other geometric characteristics of a surgical instrument, even if such a surgical instrument is multiple visible in a corresponding image, e.g. an overview image or a microscopic image. Alternatively or additionally, it could be envisaged that the assistance function takes into account movement information determined based on an image sequence.

[0064] The technique of FIG. 4 is directed to a method for avoiding any ambiguity due to the large number of corresponding visible surgical instruments.

[0065] The aspects of Figure 4 may be used, for example, in connection with locating a surgical instrument in processes 3030, 3040 of Figure 3. The aspects of Figure 4 may alternatively or additionally be used in connection with the support functions in process 3020 of Figure 3. However, Figure 4 may also be implemented alone, i.e., without reference to Figure 3.

[0066] The various variants of Figure 4 are described with respect to implementation of objects in the form of surgical instruments, however, the corresponding techniques can also be implemented for other kinds of objects.

[0067] Examples of surgical instruments are generally scalpels, forceps, scissors, needle holders, clamps, aspirators, trocars, coagulators, electrocoaters, retractors, drills, spreaders, osteotomes, sutures, knot pushers, periosteal elevators, hemostats, lancets, drainage, thread cutters, spatulas, and ultrasonic aspirators.

[0068] The method of Fig. 4 may be executed by a processor of a data processing device, for example by the processor 61 of the data processing device 60 of the example surgical microscopy system 80 of Fig. 1. The processor may for this purpose load program code from a memory and execute this program code.

[0069] In operation 3105, a sequence of images is captured, which may be triggered, for example, by a user command, such as described with respect to operation 3005 of FIG.

[0070] The method of Fig. 4 may for example be triggered by a user command requesting a particular assistance function. For example, the user may request automatic alignment and / or automatic focusing of the field of view of the microscope camera. In general, it is also envisaged that the user command does not specify a particular object on which the assistance function is performed or which defines said assistance function. This therefore means that it is possible to receive a user command requesting an assistance function with respect to an unspecified one of the displayed objects. The user command may therefore contain ambiguities regarding the surgical instrument to be taken into account.

[0071] In process 3105, a camera is activated, for example a microscope camera or an environmental camera. The image shows the surgical scene over a certain period of time. An example of an image 220 is shown in Figure 5. In Figure 5, it can be seen that a total of three surgical instruments 231, 232, 233 are visible.

[0072] Referring again to Figure 4, in optional process 3110, visible surgical instruments are located and optionally their orientation is determined. In this way, positioning information may be determined in process 3110. Such positioning may include locating the surgical instruments in one or more images of process 3105. Such positioning information may include the orientation of the surgical instruments in two or more images from process 3105. Positioning information may be obtained, for example, by point localization or by bounding boxes. Object segmentation of the surgical instruments in the images may also be performed. Instance segmentation may be performed.

[0073] Positioning can be ascertained, for example, based on optical flow, for example using the technique disclosed in U.S. Pat. No. 6,399,363.

[0074] Positioning may also optionally be performed in a reference coordinate system. For example, based on the positioning of the surgical instrument in the images from process 3105, it is possible to estimate the absolute positioning of the surgical instrument in the reference coordinate system, given the pose of the corresponding camera and the imaging characteristics of the camera. Such techniques may be used in particular with respect to images taken by an environmental camera.

[0075] Then, process 3115 includes determining the movement information of the surgical instrument from the image sequence. It is conceivable, but not required, that the movement information is determined based on the positioning information from process 3110. As already explained, process 3110 is optional and positioning information may not be required to determine the movement information. For example, the movement information may also be determined without a previous position determination, for example based on optical flow between two images taken in succession. Details regarding the determination of the movement information are explained later.

[0076] Next, in operation 3120, priority information is determined based on the movement information.

[0077] Then (in operation 3125) an assistance function is performed based on the positioning in operation 3110 and / or the movement information from operation 3115, and based on the priority information from operation 3120. For example, an automatic alignment and in particular an automatic centering of the surgical instrument to the center of activity may be performed. For this purpose, for example, one or more active areas may be identified from the movement information and their geometric centers or geometric centroids may be used as the center of activity. An automatic alignment and in particular an automatic centering of the specific surgical instrument or instruments to the geometric center may be performed. An automatic focusing on the specific surgical instrument, for example on its tip, may be performed.

[0078] The assistance functions here relate to the surgical instruments, for example to positioning information of the surgical instruments or to other geometrical properties (for example distance measurements between the surgical instruments or opening angles of fixation devices, etc.).

[0079] Various examples are based on the observation that in certain variants, it may be beneficial if the assistance function takes into account only a portion of all of the surgical instruments visible in the corresponding image, or is performed based on positioning information from process 3110. This means, for example, that the positioning and / or movement information relating to a particular one of the visible surgical instruments is given more weight than the positioning and / or movement information relating to other of the visible surgical instruments (which positioning and movement information may not be taken into account at all).

[0080] In other words, and generally, a distinction can be made in this manner between more relevant and less relevant surgical instruments, the more relevant surgical instruments then being given more weight in their support functions than the less relevant surgical instruments.

[0081] This will be explained in one embodiment with reference to the example of FIG. 5. The example scene of FIG. 5 includes an aspirator 231 (purpose: aspirating blood), a bipolar coagulation tourniquet 233 (purpose: hemostasis), and a retractor 232 (purpose: retraction of brain tissue). For the surgeon, the relevant instruments are the aspirator and the bipolar coagulation tourniquet, since they perform the main surgical actions (hemostasis, aspiration) in the illustrated image. However, the retractor is irrelevant in this scene, since it passively retracts and spatially fixes the brain tissue, and does not perform any main surgical action. This is just one example of prioritization of the relevant surgical instruments. Alternatively, the assistant surgeon's instruments may also be visible in the image, but are of little relevance to the surgeon. In the example of FIG. 5, the geometric center of gravity of the surgically relevant instruments should now be identified, and then automatic centering to this geometric center of gravity should be performed. The geometric center of gravity 291 of only the surgically relevant instruments (i.e., the retractor 232 is not included) is far from the geometric center of gravity 292 of all the instruments 231, 232, 233. From the user's point of view, if the instruments 231, 233 are not correctly prioritized or prioritized with respect to the surgical instrument 232, a poor auto-centering may occur because they are centered at the center of gravity 292 instead of the center of gravity 291. A corresponding scenario may also be described with respect to auto-focusing. Compared to the above-mentioned variant in which auto-centering is performed at the geometric center of gravity 291, it may be advantageous to perform auto-focusing on the tip or other characteristic position of the highest priority instrument (i.e., for example, the surgical instrument 231). This is based on the observation that a surgeon typically performs surgery mainly with one instrument, and the other instruments are of secondary importance to this main instrument.

[0082] The various examples are based on the observation that it is particularly easily possible to distinguish between relevant and irrelevant surgical instruments on the basis of movement information, in other words it is certainly possible to determine priority information on the basis of movement information.

[0083] Various implementations of the mobility information are possible, some examples are given below, which may also be combined with each other.

[0084] For example, the motion information may include optical flow between successive images of an image. The motion information may indicate one or more active regions. A corresponding technique is disclosed in detail in U.S. Pat. No. 6,399,343, the disclosure of which is incorporated herein by cross-reference.

[0085] For example, it may be envisaged that the movement information indicates a time-averaged movement amount of each individual object among the plurality of objects. Such a movement amount may be determined, for example, by which the plurality of objects are located (see operation 3110), and then the movement of the object is traced / tracked each time within the corresponding area in which the corresponding object is present. In general, the movement information may therefore be determined based on the positioning of the objects.

[0086] For example, the movement information may indicate one or more movement patterns of the object. For example, it may be assumed that a certain type of surgical instrument, e.g., an aspirator, is preferably used in a circular movement, and such a movement pattern (circular movement) may be recognized from which it may be inferred that the aspirator is a secondary instrument with a low priority. On the other hand, other types of surgical instruments, e.g., a scalpel, may move primarily translationally, i.e., back and forth. Such a movement pattern (translation between two end points) may be recognized from which it may be inferred that, e.g., the scalpel is a primary instrument with a high priority. In general, a periodic movement pattern predefined by the surgical use of a particular instrument may be taken into account when identifying the priority of the instrument. Such movements of the instrument are used for the utilization of the instrument in the surgical procedure itself (e.g., in the above-mentioned example of an aspirator for aspirating blood or for the above-mentioned example of a scalpel for cutting tissue), and they are therefore not movements made as part of a particular gesture recognition and have no purpose in themselves other than the execution of the gesture.

[0087] Another example of movement information would be for example information about the directional distribution of the movements of the corresponding object. It would also be conceivable to specify the movement frequency of the object's movements.

[0088] Various exemplary implementations of movement information have been disclosed above. Combinations of such variations of the disclosed movement information may also be used in the various techniques described herein.

[0089] Various ways of implementing the priority information are conceivable. For example, a segmentation map can be output, which classifies, for one of the captured images, the areas of the image in which objects are displayed into different priority classes. It is also conceivable to output labels, which are placed at the object positions. These labels may then indicate the priority information. It would therefore be conceivable that the priority information comprises a localization (for example, a point localization, a bounding box, etc.) and an associated priority label. A corresponding example is shown in FIG. 6 for a point localization. Said figure shows corresponding labels 241, 242, 243 for the surgical instruments 231, 232, 233 at the center positions of the surgical instruments 231, 232, 233, respectively. These labels 241-243 indicate the priority of the surgical instruments, with the labels 241, 243 indicating a high priority and the label 242 indicating a low priority. For example, the priority information may indicate the center of activity of relevant and irrelevant objects. In general, the priority information may therefore be associated with the positioning of various objects.

[0090] The priority information may be either a continuous value (e.g., 1=lowest priority to 10=highest priority, recursive) or a class assignment of objects to a given class, such as a class assignment to a first class and a second class (optionally one or more other classes). The class assignment may be binary. For example, the first class may relate to related objects and the second class may include unrelated objects. Assistance functions may then be performed based solely on the positioning of one or more objects assigned to the first class (i.e., objects assigned to the second class are ignored). For example, with respect to FIG. 6, a scenario was discussed in which surgical instruments 231, 233 are assigned to a first class and surgical instrument 232 is assigned to a second, unrelated class.

[0091] The priority information can be taken into account in various ways with respect to the assistance functions. A few examples are described below.

[0092] The assistance function may for example relate to the measurement of a surgical instrument, so that it is possible for example to measure the surgical instrument with the highest priority.

[0093] The assistance function may also include an automatic configuration of one or more components of the surgical microscopy system. As part of the assistance function, a target configuration may be identified for one or more components of the surgical microscopy system. When identifying the target configuration, the positioning of the two or more objects that are most highly prioritized based on the priority information may be given more importance compared to the positioning of the two or more objects that are less highly prioritized based on the priority information. A sliding transition between more and less importance is conceivable, e.g. a binary transition, i.e. the positioning of the higher priority object is taken into account and the positioning of the lower priority object is not taken into account. Depending on the assistance function, different target configurations are identified. For example, the target configuration of the surgical microscopy system may include aligning the field of view of a microscope (e.g. a microscope camera or an eyepiece) of the surgical microscopy system with respect to at least one of the two or more surgical instruments selected based on the priority information. This may thus, for example, be that the centers of gravity of all higher priority surgical instruments are identified based on the corresponding positions, and then the robotic stand is driven so that the centers of gravity overlap the center of the field of view of the microscope, as already described with respect to FIG. 5. However, this is only an example. It would also be conceivable to select only one surgical instrument from the visible surgical instruments, i.e. the one with the highest priority, and to automatically position the field of view of the microscope on a relevant point of this selected surgical instrument, for example its tip. In another variant, the target configuration of the surgical microscope observation system would include an automatic focusing taking into account the surgical instruments, the prioritization being performed among the surgical instruments. Therefore, in another variant, it would be possible for the target configuration of the surgical microscope observation system to include focusing the microscope on one of two or more predetermined surgical instruments selected on the basis of a priority method. It would therefore be possible to provide an automatic focus on the surgical instrument with a high priority.When focusing on the geometric centroids of two or more surgical instruments (or objects in general), for automatic focusing, in addition to the geometric centroids, the depth centroids can also be taken into account. This therefore means that the depths for all objects considered during the determination of the local centroids are determined, and then the focus is placed on the average value. This should be distinguished from the variant where the focus is placed on the depth at the local centroids (which is also possible in principle).

[0094] Next, details about how the movement information is mapped to the priority information will be disclosed. In other words, the following describes how the priority information can be accurately determined based on the movement information in the process 3120.

[0095] In the first example, the priority information is determined based on the movement information using a predefined criterion. This therefore means, in other words, that the criterion used to determine the priority information based on the movement information does not depend on the movement information itself. For example, a fixed threshold value can be used, which is compared with the value of the movement information. A look-up table can be used, which maps different values ​​of the movement information to different priority information. A fixed predefined function can be used, which converts the movement information into the priority information. As an example, for example, the amount of movement (quantified for example by optical flow) can be taken into account. This amount of movement can be compared with a predefined threshold. If the amount of movement is greater than the predefined threshold, the corresponding surgical instrument is assigned to a first, relevant class, otherwise to a second, irrelevant class. Regarding the example of FIG. 5, in said figure, the amount of movement shows that the aspirator 231 and the bipolar coagulation tourniquet 233 are largely moved (i.e. the amount of movement is greater than the threshold), but the retractor 232 is not. That is, the retractor 232 is fixed to the patient and only moves slightly with the brain tissue. In this case, the surgical instruments 231, 233 are classified into a high priority class and the surgical instrument 232 into a low priority class. In general, fixed movement thresholds may be used, including with respect to the variously defined movement information. Such predefined criteria with respect to the movement information may be set by the user. For example, different users may have different preferences with respect to relevant and irrelevant classification of surgical instruments (i.e. with respect to classification of surgical instruments as associated with high or low priority). However, it may be envisaged that such criteria are fixedly preprogrammed and not changed by the user.

[0096] In a second example (as an alternative or in addition to the first example above), the priority information is determined based on the movement information using a relative criterion. The relative criterion is ascertained based on the movement information. This means, in other words, that for example the relative ratio of the values ​​of the movement information ascertained for different surgical instruments or a threshold adjusted based on the value of the movement information (for example 50% of the maximum value, etc.) is taken into account. For example, the ranking of the values ​​of the movement information may be taken into account. For example, a first class associated with a high priority may contain only one object, such a scenario being particularly beneficial for automatic focusing. The movement of the various surgical instruments relative to each other may be taken into account. It may be taken into account whether the surgical instruments are moving towards each other or away from each other. In one variant, the object moving faster is classified as relevant and all other objects are classified as irrelevant, which means, in other words, that the object moving faster is classified into the first class and all other surgical instruments are grouped into the second class. Optionally, a tolerance range may also be used (for example 5%). The center of gravity is taken for the second fastest moving object if it moves at a similar speed (e.g. 95% of the movement amplitude of the fastest object). However, if one object moves significantly faster than all others, only this one object is classified as high priority and relevant. Comparisons can be made between the movement frequency spectra of various surgical instruments. For example, it can be checked whether all of the specific surgical instruments have the same movement spectrum (this would indicate that these surgical instruments are not guided by the surgeon but are fixed to the patient and move with the patient's movements).

[0097] The following is a practical example: for example, again for the scenario shown in FIG. 5, such a displacement can be determined (for example, based on optical flow). The order of displacement is aspirator 231 - bipolar coagulation tourniquet 233 - retractor 232. The displacement (defined relatively) of the retractor 232 is therefore significantly less than the aspirator 231 and the bipolar coagulation tourniquet 233. It can therefore be envisaged that the aspirator 231 and the bipolar coagulation tourniquet 233 are assigned to a high priority class, and the retractor 232 to a low priority class. In such an example, there is no need to use a fixed threshold value. This can be particularly useful when various different scenes are to be considered and it is not known a priori how the displacement will evolve with respect to the relevance of the objects. In such a case, a relative criterion is used, which can be applied flexibly and adaptively to any scene, as described above.

[0098] In a third example (again, as an alternative or in addition to the examples above), a machine learning model is used to identify the priority information. The machine learning model receives the movement information as input. The machine learning model may be trained, for example, to recognize movement patterns of relevant surgical instruments and distinguish these from movement patterns of less relevant surgical instruments. The machine learning model may then output the priority information along with the corresponding class assignments.

[0099] The machine learning model can derive more complex decision rules than the first and second examples above. For example, one or more of the following factors can be cumulatively considered: distance between moving instruments, type of movement (e.g., fast or slow), direction of movement, angle of incidence of instruments (to distinguish assistant from surgeon), movement pattern, etc. It can learn to take such criteria into account through appropriate training. For this purpose, an expert can manually annotate the corresponding input data to the machine learning model by identifying the corresponding ground truth for priority information.

[0100] In general, the machine learning model may also receive other inputs besides the input of the movement information, such as the positioning of two or more surgical instruments, i.e. corresponding positioning information, such as corresponding bounding boxes or center localizations, or corresponding instance segmentation maps.

[0101] Information regarding the semantic context can also be transmitted, ie for example the phase of the surgery or the type of surgery.

[0102] Above, it has been described how the priority information is determined based on the movement information. It has already been noted in connection with the machine learning model that in addition to the movement information, other data may also be taken into account when determining the priority information (this applies generally to various examples and is not limited to the use of machine learning models).

[0103] For example, priority information may further be determined based on the positioning of two or more surgical instruments, and therefore, it may be assumed that objects located relatively centrally in the field of view will tend to have a higher priority than objects located on the periphery.

[0104] Alternatively or additionally, the priority information could be determined based on semantic context information for a scene associated with two or more surgical instruments. An example of semantic context information is information indicating the type of surgical intervention or the phase of the surgical intervention, e.g. "coagulation hemostasis" or "blood aspiration", etc.). It could also be envisaged to identify the type of surgery, i.e., e.g. "spine, skull, tumor, blood vessel". For example, the corresponding context information could be transmitted as another input to the machine learning model. Depending on the semantic text information, for example, different lookup tables could be used to map the movement information to the priority information. Also, depending on the semantic context information, different thresholds could be used to classify into higher or lower priority classes based on the amount of movement, these are just a few examples.

[0105] As another example, consideration can be given to whether a particular surgical instrument is held in the left or right hand, which can be compared to the corresponding hand preference of the surgeon (i.e., whether the surgeon is left-handed or right-handed).

[0106] In summary, in the examples of Figs. 4, 5 and 6, the priority information is determined based on the movement information. This has the advantage that it allows a particularly suitable and robust prioritization that can handle different types of objects. This is explained in more detail below. For example, in (Patent Document 3), a distinction is made between "auxiliary" and "critical" instruments. This distinction is made based on three classifications: firstly, a classification of the type of instrument, secondly, a classification as to whether the instrument is assistive or non-assistive, and thirdly, a classification as to the hand (right or left) with which the instrument is held. In (Patent Document 3), therefore, a clear classification of the type of instrument (aspirator, retractor, etc.) is used to determine whether an instrument is "critical" or "auxiliary" or "assistive" or "non-assistive". The type of instrument must be classified. However, in neurosurgery, there are more than 100 types of instruments, which are often difficult to distinguish visually. Therefore, a clear classification is technically difficult to achieve. Even if a robust and clear classification of the type of instrument were possible, the solution of (Patent Document 3) has another problem. That is, one and the same instrument may be surgically relevant or irrelevant depending on the situation. For example, an aspirator is relevant if it is currently aspirating blood, whereas it is irrelevant if it is not being used to aspirate blood but merely to hold down brain tissue (as a kind of handheld "dynamic retractor"). This illustrates that the type of instrument does not directly state the surgical relevance of the instrument or its priority in terms of support functions. This problem is solved in the present disclosure by determining priority information in operation 3120 based on movement information from operation 3115.

[0107] The features mentioned above and those described below can be used not only in the corresponding combinations explicitly stated, but also in other combinations or alone, without departing from the scope of protection of the invention.

[0108] For example, although various aspects of the assistance functions have been described above with respect to surgical instruments, it would generally be conceivable to take into account other types of objects, such as characteristic anatomical features of a patient.

[0109] In addition, various aspects related to the robotic stand have been described above. It is not absolutely necessary for the surgical microscopy system to include a robotic stand. The surgical microscopy system may include a partial robotic stand or a manual stand. [Explanation of symbols]

[0110] 60 Data processing device 61 processors 62 Memory 63 User Interface 64 Communication Interface 78 Surgical instruments 79 patients 80 Surgical Microscope Observation System 81 Head part 82 Robot Stand 82a Handle 83 Environmental Camera 84 Microscope 85 Optical Components 86 Microscope Camera 121, 122, 123 field of view 231, 232, 233 surgical instruments

Claims

1. 1. A computer-implemented method for controlling a surgical microscopy system (80) having a stand (82) and a microscope (84) carried by the stand (82), comprising: - driving (3105) the cameras (83, 86) of said surgical microscopy system to obtain a sequence of images; - determining (3115) movement information for each of two or more objects (78, 231, 232, 233) shown in said image sequence based on said image sequence; determining (3120) priority information of said two or more objects (78, 231, 232, 233) based on said movement information; - executing (3125) an assistant function related to said two or more objects (78, 231, 232, 233) based on said priority information; 23. A computer-implemented method comprising:

2. the priority information includes a class assignment of the two or more objects (231, 232, 233) to at least a first class and a second class; said assist function being executed in relation to at least one object (231, 233) assigned to said first class; the assist function is not executed in relation to at least one other object (232) assigned to the second class; 10. The computer-implemented method of claim 1.

3. The priority information is determined based on the movement information using at least one predetermined criterion; the at least one predetermined criterion includes a fixed movement threshold; 3. A computer-implemented method according to claim 1 or 2.

4. said priority information is obtained without any classification of said objects, in particular without any instance segmentation of said objects; A computer implemented method according to any one of claims 1 to 3.

5. the priority information is determined based on the movement information using at least one relative criterion determined based on the movement information; A computer implemented method according to any one of claims 1 to 4.

6. the at least one relative criterion comprises a ranking of the movement information with respect to the two or more objects (231, 232, 233); 6. The computer-implemented method of claim 5.

7. the at least one relative criterion comprises a relative movement of the two or more objects (231, 232, 233) with respect to one another; 7. A computer implemented method according to claim 5 or 6.

8. The priority information is determined by a machine learning model that receives the movement information as input. A computer implemented method according to any one of claims 1 to 7.

9. The priority information is also determined based on a positioning of the two or more objects (231, 232, 233). A computer implemented method according to any preceding claim.

10. The priority information is also determined based on semantic context information regarding a scene associated with the two or more objects. A computer implemented method according to any one of claims 1 to 9.

11. receiving (3005) a user command requesting said assistance function with respect to a non-specific one of said two or more objects (231, 232, 233); The computer-implemented method of any one of claims 1 to 10, further comprising:

12. the motion information comprises optical flow between successive images of the image sequence; A computer implemented method according to any one of claims 1 to 11.

13. the priority information is associated with a position of the two or more objects; A computer implemented method according to any preceding claim.

14. A computer-implemented method according to any preceding claim, wherein the priority information comprises a segmentation map and an associated priority label.

15. the priority information includes a point location specification and an associated priority label; A computer implemented method according to any preceding claim.

16. Executing the support function includes: - determining a target configuration of said surgical microscopy system (80) based on said priority information and on a positioning of said two or more objects (231, 232, 233); - driving at least one component of the surgical microscopy system based on the target configuration; The computer-implemented method of any one of claims 1 to 15, further comprising:

17. the target configuration of the surgical microscopy system includes an alignment of a field of view (122, 123) of the microscope (84) with respect to at least one of the at least two or more objects (231, 232, 233) selected based on the priority information.

20. The computer-implemented method of claim 16.

18. the target configuration of the surgical microscopy system includes automatic focusing of the microscope (84) on one of the two or more objects (231, 232, 233) selected based on the priority information; 18. A computer-implemented method according to claim 16 or 17.

19. A data processing device (60) for controlling a surgical microscope observation system (80), comprising: a processor (61) configured to load program code from a memory (62) and execute same, said execution of said program code causing said processor (61) to perform the following steps: - driving (3105) the cameras (83, 86) of said surgical microscopy system to obtain a sequence of images; - determining (3115) movement information for each of two or more objects (78, 231, 232, 233) shown in said image sequence based on said image sequence; - determining (3120) priority information of said two or more objects (78, 231, 232, 233) based on said movement information; - executing (3125) an assistant function associated with said two or more objects (78, 231, 232, 233) based on said priority information; A data processing device (60) for executing the above.

20. A data processing apparatus (60) according to claim 19, wherein said execution of said program code causes said processor to perform the method according to any one of claims 1 to 18.

21. A surgical microscopy system including the data processing device (60) according to claim 19 or 20.

Images