Computer-implemented method, data processing device, and surgical microscope system

Abstract

The present disclosure provides a computer-implemented method for controlling a surgical microscope system, a data processing device for controlling a surgical microscope system, and a surgical microscope system. Techniques related to performing auxiliary functions of a surgical microscope system are described. Auxiliary functions, such as automatic centering or measurement of a surgical instrument, are performed in conjunction with a plurality of objects (231, 232, 233) depicted in an image (220) captured by the surgical microscope system. Furthermore, prioritization information (241, 242, 243) for these objects (231, 232, 233) is also considered, e.g., in order to classify the objects as relevant objects and unrelevant objects.

Description

Technical Field

[0001] Various examples of the present disclosure relate to techniques for performing auxiliary functions for surgical microscope systems. The various examples relate in particular to prioritization between different objects related to such auxiliary functions. Background Art

[0002] Medical surgical microscope systems (also called robotic visualization systems or surgical visualization systems) known from the prior art have a robotic stand for positioning the microscope; see, for example, DE 10 2022100 626 A1.

[0003] Here, the robot support can be controlled manually. However, there are also known techniques for automatically controlling the robot support, for example, in order to achieve automatic centering and / or automatic focusing on a specific object (e.g., a surgical instrument, also called surgical equipment or surgical tool or operating tool). In this case, a user command triggers a positioning process, during which the robot support and / or the objective optical unit of the microscope are driven. Such a technique is known, for example, from US 10,456,035 B2.

[0004] It has been observed that, particularly in relatively complex surgical scenarios (e.g., involving a large number of surgical instruments and / or surgical instruments of different types), such techniques of the prior art may lead to undesirable results. For example, positioning may sometimes be performed on the wrong surgical instrument that the surgeon does not want to use.

[0005] To compensate for such shortcomings, for example, US 10,769,443 B2 discloses identifying a primary surgical instrument from a set of visible surgical instruments. Such a technique also has certain disadvantages and limitations. For example, it has been found that the technique described in this document does not work well for some types of surgical instruments. This means that, depending on the type of surgical instrument, the results obtained are poor. Due to the large number of surgical instruments available, this weakens the acceptance of the system and may lead to unexpected behavior. Summary of the invention

[0006] Therefore, there is a need for improved techniques for auxiliary functions of surgical microscope systems. In particular, there is a need for techniques that can robustly classify the priority of imaging objects that the auxiliary functions can involve. Objects need to be classified into primary objects and non-primary objects.

[0007] This object is achieved by the features described below. The features described below define the embodiments.

[0008] The present disclosure provides a computer-implemented method for controlling a surgical microscope system having a stand and a microscope carried by the stand,

[0009] Among them, the method includes:

[0010] - driving a camera of the surgical microscope system to obtain a sequence of images,

[0011] - determining, based on the sequence of images, movement information for each of two or more objects depicted in the sequence of images,

[0012] - determining priority ranking information for the two or more objects based on the movement information, and

[0013] - Based on the priority ranking information, performing auxiliary functions related to the two or more objects.

[0014] In one embodiment, the priority sorting information includes a category assignment that divides the two or more objects into at least a first category and a second category, wherein the auxiliary function is performed in conjunction with at least one object assigned to the first category, and wherein the auxiliary function is not performed in conjunction with at least one additional object assigned to the second category.

[0015] In one embodiment, the prioritization information is determined based on the movement information using at least one predefined criterion, wherein the at least one predefined criterion includes a fixed movement threshold.

[0016] In one embodiment, the prioritization information is obtained without performing any classification and in particular without performing any instance segmentation of the objects.

[0017] In one embodiment, the prioritization information is determined based on the movement information using at least one relative criterion, the at least one relative criterion being determined based on the movement information.

[0018] In one embodiment, the at least one relative criterion includes a ranking of movement information associated with the two or more objects.

[0019] In an embodiment, the at least one relative criterion comprises a relative movement of the two or more objects with respect to each other.

[0020] In one embodiment, the prioritization information is determined by a machine learning model that receives the movement information as input.

[0021] In one embodiment, the prioritization information is also determined based on the positioning of the two or more objects.

[0022] In one embodiment, the priority ranking information is also determined based on semantic context information of the scene associated with the two or more objects.

[0023] In one embodiment, the method further comprises: - receiving a user command requesting an auxiliary function related to a non-specific object among the two or more objects.

[0024] In one embodiment, the motion information includes optical flow between consecutive images of the image sequence.

[0025] In one embodiment, the prioritization information is related to the positioning of the two or more objects.

[0026] In one embodiment, the prioritization information includes a segmentation map having a prioritization label.

[0027] In one embodiment, the prioritization information includes point locations with prioritization tags.

[0028] In one embodiment, performing the auxiliary function includes: - determining a target configuration of the surgical microscope system based on the priority ranking information and the positioning of the two or more objects, and - driving at least one component of the surgical microscope system based on the target configuration.

[0029] In one embodiment, the target configuration of the surgical microscope system includes an alignment of a field of view of the microscope relative to at least one of the two or more objects selected based on the prioritization information.

[0030] In one embodiment, the objective configuration of the surgical microscope system includes automatic focusing of the microscope relative to one of the two or more objects selected based on the prioritization information.

[0031] The present disclosure also provides a data processing device for controlling a surgical microscope system, wherein the data processing device includes a processor, and the processor is configured to load a program code from a memory and execute the program code, wherein the execution of the program code causes the processor to perform the following steps:

[0032] - driving a camera of the surgical microscope system to obtain a sequence of images,

[0033] - determining, based on the sequence of images, movement information for each of two or more objects depicted in the sequence of images,

[0034] - determining priority ranking information for the two or more objects based on the movement information, and

[0035] - Based on the priority ranking information, performing auxiliary functions related to the two or more objects.

[0036] In one embodiment, execution of the program code causes the processor to perform the method described above.

[0037] The present disclosure also provides a surgical microscope system, which includes the data processing device described above.

[0038] Various examples are based on the finding that the techniques known in the prior art for identifying primary surgical instruments show low robustness to changes in the appearance of surgical instruments. This is because there are a large number of different types of surgical instruments in different operating environments, such as surgical instruments in the three-digit range. Various examples are based on the finding that within the scope of surgical instrument type classification, it is difficult to use algorithms to robustly distinguish such a large number of possible result categories. For example, when using machine learning classification models, out of distribution situations may often occur, that is, the appearance of a specific type of surgical instrument was not considered when training the machine learning classification model. The model may bring unpredictable results.

[0039] Additionally, various examples are based on the discovery that techniques known in the prior art for identifying primary surgical instruments ignore the fact that a certain type of surgical instrument may be a primary instrument in one context but not in another context. It has been recognized that considering context can aid in prioritization.

[0040] Various aspects related to a surgical microscope system having a robotic support and a microscope carried by the robotic support are described below. In particular, techniques related to auxiliary functions for assisting a surgeon are described. Here, the auxiliary functions are performed relative to at least one object depicted in a corresponding image (e.g., captured by a microscope camera or an environmental camera). Examples of such auxiliary functions are automatic positioning, automatic centering, automatic orientation, automatic zooming, or automatic focusing relative to one or more objects; further examples include, for example, measuring one or more objects.

[0041] Techniques for distinguishing related and unrelated objects from each other are disclosed.Related objects are sometimes also referred to as primary objects.

[0042] More generally, techniques are described regarding how priority information for various objects can be determined. The priority information can then be considered to perform auxiliary functions. For example, an auxiliary function can consider only those objects that are classified as relevant (i.e., high priority) based on the priority information. A step-by-step determination of the corresponding priority information can also be used so that higher priority objects are given more consideration when performing auxiliary functions.

[0043] Generally speaking, the determination of priority ranking information may correspond to a regression task or a classification task.

[0044] According to various disclosed variants, prioritization information is determined based on movement information of various objects. For reference embodiments in which objects must be classified by type (e.g., instance segmentation of instances with associated classifications), the use of movement information has particular advantages. Even for a large number of different a priori unknown scenarios, robust prioritization can be achieved in particular. For example, there is no need to parameterize the classification model used to determine the object type. For example, there is no need to train any corresponding machine learning classification model, which makes it possible to save complex training activities of collecting images of different types of objects. Prioritization information can be obtained without any classification and in particular without any instance segmentation of the objects.

[0045] However, in some variations, it is contemplated that further information may be taken into account when determining the prioritization information, such as semantic contextual information related to the imaging scene, or whether a particular surgical instrument is carried in the left or right hand.

[0046] A computer-implemented method for controlling a surgical microscope system is disclosed. The surgical microscope system includes a stand. The surgical microscope system also includes a microscope. The microscope is carried by the stand.

[0047] For example, the support may be a robotic support or a portion of a robotic support.

[0048] The method includes driving a camera of a surgical microscope system. For example, a microscope camera and an environment camera may be driven. An image sequence is obtained by driving the camera. For example, the image sequence may thus correspond to a temporal sequence of images depicting a scene.

[0049] The method also includes determining movement information. Determining movement information for each of two or more objects depicted in the image sequence. Determining the movement information based on the image sequence.

[0050] For example, a heuristic or machine learning model may be used to determine the movement information. For example, a threshold comparison may be performed between one or more variables indicated by the movement information and one or more predefined thresholds.

[0051] For example, the movement information may specify an optical flow. Alternatively or additionally, the movement information may specify an activity area, that is, an area in the image sequence where the contrast variation between the images of the sequence is relatively large. It is conceivable that the movement information specifies a movement pattern of the object. For example, the movement information may specify a movement amplitude and / or a movement frequency of the object. Combinations of such contents of the movement information as described above are also conceivable.

[0052] It is conceivable (but not necessary) that the movement information is related to the positioning of the object in the image sequence. For example, the objects in the image sequence may be first positioned and then the corresponding movement information of each positioned object may be determined.

[0053] The method further comprises determining prioritization information for the two or more objects; this is performed based on the movement information. For example, the prioritization information specifies which objects are primary or relevant and which objects are not primary or irrelevant. In addition to such binary class assignments for various objects as part of the prioritization information, it is also conceivable to perform a multidimensional class assignment or regression. For example, a prioritization value may be output, for example in a range from 0 (= irrelevant) to 10 (= relevant).

[0054] Based on the priority ranking information, the auxiliary function related to the two or more objects is then performed. This may mean, for example, that as part of the auxiliary function, the object with a higher (lower) priority ranking is considered to a greater (lesser) extent.

[0055] As part of the auxiliary function, for example, a robot support can be driven, for example, to perform automatic positioning relative to a reference point determined based on one or more objects. As part of the auxiliary function, the appearance of one or more objects in one or more images (e.g., microscope images) can be evaluated.

[0056] A data processing device is disclosed. The data processing device is configured to control a surgical microscope system. The data processing device includes a processor. The processor is configured to load a program code from a memory and execute the program code. Executing the program code causes the processor to perform the above method for controlling a surgical microscope system.

[0057] A surgical microscope system comprising the data processing device is also disclosed.

[0058] The features set out above and the features described below can be used not only in the corresponding combination explicitly set out but also in other combinations or alone, without departing from the scope of protection of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 Surgical microscope systems according to various examples are schematically illustrated.

[0060] Figure 2 Schematically illustrates different fields of view associated with the microscope and environmental cameras of an exemplary surgical microscope system.

[0061] Figure 3 is a flow chart of an exemplary method.

[0062] Figure 4 is a flow chart of an exemplary method.

[0063] Figure 5 An image captured by a camera is shown in which multiple surgical instruments can be seen.

[0064] Figure 6 Corresponds to Figure 5 , which also displays the priority ranking information of various surgical instruments. DETAILED DESCRIPTION

[0065] The characteristics, features and advantages of the present invention described above and the manner in which they are achieved will become clearer and more clearly appreciated in conjunction with the following description of exemplary embodiments, which are explained in more detail in conjunction with the accompanying drawings.

[0066] Based on preferred embodiments, the present invention is explained in more detail with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar elements. The accompanying drawings are schematic representations of various embodiments of the present invention. The elements shown in the accompanying drawings are not necessarily shown in true proportion. Alternatively, the various elements shown in the accompanying drawings are presented in a manner that makes their functions and general purposes easy for those skilled in the art to understand. The connection and coupling between the functional units shown in the figures and the elements can also be implemented as indirect connections or couplings. Connections or couplings can be implemented in a wired or wireless manner. The functional units can be implemented as hardware, software, or a combination of hardware and software.

[0067] The following describes techniques related to the operation of a surgical microscope system. The described techniques enable the execution of auxiliary functions, such as automatic configuration of one or more components of the surgical microscope system. Typically, the auxiliary function is implemented with respect to at least one of a number of objects visible in an image captured by the surgical microscope system.

[0068] According to various examples, priority information is determined for an object. Here, the priority information is determined based on movement information of the object.

[0069] Figure 1 Schematically illustrated are aspects of an exemplary surgical microscope system 80. The surgical microscope system 80 is used for microscopic imaging of an examination area during a surgical intervention. To this end, a patient 79 is placed on an operating table 70. A surgical instrument 78 is shown at position 78.

[0070] The surgical microscope system 80 comprises a robot support 82 which carries a positionable head part 81. Depending on the variant, the robot support 82 may have different degrees of freedom. It is known that the robot support 82 has six degrees of freedom for positioning the head part 81, that is, translation along each of the x-axis, y-axis and z-axis and rotation around each of the x-axis, y-axis and z-axis. Figure 1 As shown, the robot support 82 may have a handle 82a.

[0071] Although Figure 1 A robotic support 82 is discussed, but generally only partially robotic or manual support may also be used.

[0072] The head part 81 comprises a microscope 84 with optical components 85 (e.g. illumination optical unit, objective optical unit, zoom optical unit, etc.). In the example shown, the microscope 84 also comprises a microscope camera 86 (here a stereo camera with two channels; however, a single optical unit is also conceivable), with which images of the examination area can be captured and these images can be reproduced, for example, on a screen 69. Therefore, the microscope 84 is also referred to as a digital microscope. The field of view 123 of the microscope camera 86 is also shown.

[0073] exist Figure 1 In the example of , the microscope 84 also includes an eyepiece 87 with an associated field of view 122. Thus, the eyepiece 87 is optional. For example, the detection beam path can be split by a beam splitter so that both the image can be captured by the camera 86 and the observation can be performed through the eyepiece 87. Not all variants require that the microscope 84 has an eyepiece. A purely digital microscope 84 without an eyepiece is also possible.

[0074] exist Figure 1 In the example of FIG. 8 , the head portion 81 of the surgical microscope system 80 carried by the bracket 82 also includes an environmental camera 83. The environmental camera 83 is optional. Although Figure 1 The environment camera 83 in the example of is shown as being integrated into the microscope 84, but it can also be arranged separately from the microscope 84. For example, the environment camera can be a CCD camera. The environment camera can also have a depth resolution. As an alternative or in addition to the environment camera, it is also conceivable that there are other auxiliary sensors, such as distance sensors (e.g., time-of-flight cameras or ultrasonic sensors or sensors with structured lighting). Figure 1 The field of view 121 of the environment camera 83 is shown in FIG.

[0075] Thus, the surgeon has several options for viewing the examination area: using the eyepiece 87, using the microscope image captured by the camera 86, or using the overview image captured by the environmental camera 83. The surgeon can also view the examination area directly (without magnification).

[0076] Next, various aspects related to the fields of view 121 , 122 , and 123 will be discussed.

[0077] Figure 2 Various aspects related to each field of view are shown. Figure 2 The field of view 121 of the environment camera of the surgical microscope system 80 is shown. Therefore, the overview image depicts a relatively large area. In addition, the field of view 122 of the eyepiece and the field of view 123 of the microscope camera 86 are shown by way of example. The fields of view 121, 122, 123 do not necessarily have to be arranged in a mutually centered manner.

[0078] Reference again Figure 1 : Various components of the surgical microscope system 80 (e.g., the robot stand 82, the microscope 84) or one or more additional components (e.g., the environmental camera 83) are controlled by the processor 61 of the data processing device 60.

[0079] The processor 61 may be designed as, for example, a general-purpose central processing unit (CPU) and / or a field programmable logic module (FPGA) and / or an application-specific integrated circuit (ASIC). The processor 61 is capable of loading program codes from the memory 62 and executing the program codes.

[0080] Processor 61 can communicate with various components of surgical microscope system 80 via communication interface 64. For example, processor 61 can drive support 82 to move head portion 81 relative to operating table 70, such as translation and / or rotation. Processor 71 can, for example, drive optical components 85 of microscope 84 to change zoom and / or focus (focal length). Images from environmental cameras (if any) can be read out and evaluated. In general, images can be evaluated and auxiliary functions can be performed based on the evaluation.

[0081] The data processing device 60 also includes a user interface 63. The user interface 63 can be used to receive commands from the surgeon or generally from the user of the surgical microscope system 80. The user interface 63 can have different configurations. For example, the user interface 63 can include one or more of the following components: a handle on the head portion 81; a foot switch; voice input; input via a graphical user interface; etc. The user interface 63 can provide graphical interaction via menus and buttons on the monitor 69.

[0082] The following describes techniques regarding how auxiliary functions may be prepared by surgical microscope system 80. An auxiliary function may be requested, for example, by a user command.

[0083] As a general rule, such user commands can take different forms. For example, the user command can specifically identify a specific object. For example, the user can specify by voice command: "Automatically center surgical instrument 1". In this way, such user command accurately specifies the object relative to which the surgical microscope system 80 is configured (here relative to "surgical instrument 1"). However, in other examples, it is also conceivable that the user command is not directed to a specific object. For example, the user can specify by voice command or by pressing a button: "Automatically center". In this way, there is no specification of the object relative to which automatic centering should be performed (even if multiple candidates for automatic centering can be seen). In this example, the user command is not clear. The user command does not distinguish between multiple visible surgical instruments. In the reference embodiment, the user command may be misunderstood, and, for example, automatic centering may be performed relative to an incorrect surgical instrument. The user's expectation (e.g., automatic centering of "surgical instrument 1") may be inconsistent with the actual system behavior (e.g., automatic centering of the geometric center of all visible surgical instruments).

[0084] Techniques related to assistive functions triggered by user commands are described below, which allow better fulfillment of user expectations compared to reference implementations. Deterministic system behavior is made possible, which provides repeatable and understandable results in a variety of situations and / or when faced with a variety of scenarios. Such techniques are based on the discovery that, especially in high-stress situations under time pressure (such as often occur in surgical environments), the system behavior of an automated control system must fully match user expectations.

[0085] Figure 3 is a flow chart of an exemplary method. Figure 3 The method relates to technology related to the configuration of a surgical microscope system for imaging an object in response to user commands.

[0086] From Figure 3 The method may be executed by a processor of a data processing device, for example by a Figure 1 The program code is executed by the processor 61 of the data processing device 60 of the surgical microscope system 80 of the example. To this end, the processor can load the program code from the memory and then execute the program code.

[0087] A user command is received in block 3005. The user command may request an auxiliary function. For example, the user command requests that a microscope camera be configured to implicitly or explicitly image an object (e.g., a surgical instrument). A user command is received from a user interface. For example, the user command may request auto-alignment or auto-focus. The user command may request measurement of a surgical instrument.

[0088] A user command may not be directed to a specific object. A user command may not specify which of multiple visible objects it is associated with.

[0089] In block 3010 (optional), a microscope image is captured. To this end, the microscope camera of the microscope is driven; see Figure 1 The microscope camera 87 of the microscope 84 in the example. As described above in conjunction with Figure 2 As discussed, microscope cameras typically have a relatively small field of view, ie in particular smaller than the field of view of the eyepiece or the environmental camera (if any).

[0090] In block 3015 (optional), the object (e.g., surgical instrument) identified by the user command is then searched in the microscope image from block 3010. If the object is already in the field of view of the microscope camera, then the object is found in block 3015, that is, the object is visible in the microscope image: block 3020 is then executed. This involves performing auxiliary functions based on the microscope image (e.g., auto-centering or auto-focusing or measuring the object).

[0091] In block 3015, a situation may arise where the object is not found in the microscope image. This means that the object is not located in the central area of ​​the scene imaged by the microscope camera. In this case, in block 3025, the environment camera is driven to capture an overview image. This is done to check whether the object is located in the peripheral area of ​​the scene imaged by the environment camera (not the microscope camera).

[0092] In block 3030, it may then be determined whether the object is visible in the overview image. In block 3030, it may be determined whether the object is located in the peripheral region, that is, in a region of the scene that is covered by the field of view of the environment camera but not by the field of view of the microscope (in Figure 2 , this is the area outside the field of view 123 but within the field of view 121).

[0093] Generally, the object is searched in the overview image. If the object is not found in the overview image, an error is output in block 3035. Otherwise, block 3040 is executed.

[0094] In block 3040, control commands are provided to the robot support to move the microscope so that the object is located in the field of view of the microscope camera, that is, in the central area of ​​the scene. This means that in block 3040, a rough alignment is performed so that the surgical instrument can be seen in the microscope image captured in another iteration 3041 of block 3010.

[0095] In conclusion, therefore, through Figure 3In the method of the invention, an overview image may first be captured in order to locate a surgical instrument for an auxiliary function using an environment camera. In this case, this overview image from the environment camera is first evaluated and it is determined whether it contains a surgical instrument. The robot support is then driven in order to move the identified surgical instrument into the field of view of the microscope camera. The actual auxiliary function may then be performed based on the evaluation of one or more microscope images.

[0096] Sometimes it may be the case that the user command from block 3005 does not accurately specify at least one object related to the performance of the auxiliary function in block 3020. For example, four surgical instruments may be visible, which are all candidates for the auxiliary function (e.g., automatic centering). Therefore, it is not initially clear which subset of the visible surgical instruments must be searched in block 3030 and then located in block 3040. To address such a problem, in particular, the following method may be applied in combination with Figure 4 Describe the technology.

[0097] Figure 4 is a flow chart of an exemplary method. Figure 4 The method relates to a technique for preparing to perform an auxiliary function. The auxiliary function can, for example, use the position and / or orientation (positioning; also called pose or absolute position) and / or other geometrical properties of a surgical instrument, even if a large number of such surgical instruments can be seen in the corresponding image (e.g. an overview image or a microscope image). As an alternative or in addition, it is conceivable that the auxiliary function takes into account movement information determined based on the image sequence.

[0098] Figure 4 The technology involves various aspects for avoiding any ambiguity caused by a large number of corresponding visible surgical instruments.

[0099] Figure 4 Aspects of the Figure 3 As an alternative or in addition, Figure 4 All aspects can be combined Figure 3 The auxiliary function of box 3020 is used. However, Figure 4 It can also be implemented on a stand-alone basis, that is, without Figure 3 is implemented in the context of .

[0100] Figure 4 The various variations in the invention are described in conjunction with an embodiment of an object in the form of a surgical instrument. However, the corresponding techniques may also be performed on other types of objects.

[0101] Examples of surgical instruments generally include: scalpels, forceps, scissors, needle holders, forceps, aspirators, trocars, coagulators, electrocautery, retractors, drills, dilators, osteotomes, suture materials, knotters, periosteal elevators, hemostats, lancets, drains, wire cutters, scrapers, and ultrasonic aspirators.

[0102] Figure 4 The method may be executed by a processor of a data processing device, for example by a Figure 1 The program code is executed by the processor 61 of the data processing device 60 of the surgical microscope system 80 of the example. To this end, the processor can load the program code from the memory and execute the program code.

[0103] In block 3105, a sequence of images is captured. This may be triggered, for example, by a user command, such as in conjunction with Figure 3 As described in box 3005.

[0104] For example, from Figure 4 The method may be triggered by a user command requesting a specific auxiliary function. By way of example, the user may request automatic alignment and / or automatic focusing of the microscope camera field of view. As a general rule, it may be envisaged that the user command does not specify a specific object on which the auxiliary function is based or on which the auxiliary function is defined. This therefore means that a user command requesting an auxiliary function related to a non-specific object among the displayed objects may be received. As a result, the user command may be ambiguous as to the surgical instrument to be considered.

[0105] In block 3105, a camera, such as a microscope camera or an environmental camera, is driven. These images depict the surgical scene during a specific period of time. Figure 5 An example of image 220 is shown in FIG. Figure 5 As can be seen in the figure, a total of three surgical instruments 231, 232, and 233 can be seen.

[0106] Reference again Figure 4 : In optional block 3110, the visible surgical instrument is located and optionally oriented. Thus, positioning information may be determined in block 3110. Such positioning information may include the positioning of the surgical instrument in one or more images from block 3105. Such positioning may include the orientation of the surgical instrument in two or more images from block 3105. The positioning information may be obtained, for example, by point positioning or by a bounding box. The surgical instrument object in the image may also be segmented. Instance segmentation may be performed.

[0107] For example, positioning can be determined based on optical flow. For example, the technology disclosed in WO 2022 / 161930A1 can be used.

[0108] Positioning can also be optionally performed in a reference coordinate system. For example, based on the positioning of the surgical instrument in the image from block 3105, the absolute positioning of the surgical instrument in the reference coordinate system can be inferred when the pose of the corresponding camera and the imaging properties of the camera are known. This technique can be particularly used in conjunction with images captured by an environmental camera.

[0109] Then, block 3115 includes determining movement information of the surgical instrument based on the image sequence. It is conceivable, but not necessary, to determine the movement information based on the positioning information from block 3110. As explained above, block 3110 is optional, and it is possible that the positioning information is not required to determine the movement information. For example, the movement information can also be determined without prior positioning, for example based on the optical flow between two consecutively captured images. Details on the determination of the movement information will be explained later.

[0110] Prioritization information is then determined in box 3120 based on the movement information.

[0111] Based on the positioning from box 3110 and / or the movement information from box 3115, and based on the prioritization information from box 3120, the auxiliary function is then performed (in box 3125). For example, automatic alignment on the center of activity of the surgical instrument, in particular automatic centering, can be performed. To this end, for example, one or more active areas can be determined based on the movement information, and their geometric center or geometric center point can be used as the center of activity. Automatic alignment on the geometric center of a specific surgical instrument or multiple surgical instruments, in particular automatic centering, can be performed. Automatic focusing on a specific surgical instrument (for example, its tip) can be performed.

[0112] The auxiliary functions here are related to the surgical instruments, for example, related to positioning information, or related to other geometric properties of the surgical instruments (for example, distance measurement between surgical instruments or opening angle of suturing instruments, etc.).

[0113] Various examples are based on the finding that in certain variants, it may be helpful if the auxiliary function considers only a subset of all surgical instruments visible in the corresponding image, or more generally, is performed based on the priority ranking information from block 3110. This means, for example, that the positioning information and / or the movement information associated with a particular surgical instrument among the visible surgical instruments is considered more than the positioning information and / or the movement information associated with another surgical instrument among the visible surgical instruments (whose positioning and movement information may also not be considered at all).

[0114] In other words, and more generally, a distinction can thus be made between more relevant and less relevant surgical instruments; more relevant surgical instruments are taken into account more in the auxiliary functions than less relevant surgical instruments.

[0115] This is a specific example of Figure 5 An example to explain: Figure 5 The exemplary scenario contains an aspirator 231 (purpose: to aspirate blood), a bipolar coagulator 233 (purpose: to stop bleeding), and a retractor 232 (purpose: to restrain brain tissue). For the surgeon, the relevant instruments are the aspirator and the bipolar coagulator, as they perform the main surgical actions in the shown image (hemostasis, aspiration). However, the retractor is not relevant in this scenario: the retractor passively restrains the brain tissue, is fixed in space, and does not perform any main surgical actions. This is just an example of prioritizing relevant surgical instruments. As an alternative, the assistant surgeon's instruments can also be seen in the image, but these instruments are of little relevance to the lead surgeon. Figure 5 In the example of , the geometric center point of the surgically relevant instruments should now be determined, and then automatic centering should be performed on this geometric center point. The geometric center point 291 of only the surgically relevant instruments (i.e. without the retractor 232) is at a certain distance from the geometric center point 292 of all instruments 231, 232, 233. From the user's point of view, if the priority sorting of the instruments 231, 233 relative to the surgical instrument 232 is incorrect or there is no priority sorting, a poor automatic centering situation may occur because the center point 292 is centered instead of the center point 291. The corresponding scenario can also be described in conjunction with automatic focusing. Compared with the variant of performing automatic centering on the geometric center point 291 outlined above, it can be advantageous to perform automatic focusing on the tip of the highest priority instrument or another characteristic position (that is, for example on the surgical instrument 231). This is based on the discovery that surgeons usually use a single instrument mainly for surgery, and other instruments have secondary significance after this primary instrument.

[0116] Various examples are based on the discovery that it is particularly easy to distinguish between relevant and less relevant surgical instruments based on movement information. In other words, prioritization information can be reliably determined based on movement information.

[0117] Various implementations of the mobile information can be envisioned, and some examples are discussed below. These examples can also be combined with each other.

[0118] For example, the motion information may include optical flow between consecutive images of the image sequence. The motion information may specify one or more active regions. The corresponding technology is disclosed in detail in WO 2022161930 A1, and its disclosure is incorporated herein by cross-reference.

[0119] For example, it is conceivable that the movement information indicates a time-averaged movement amplitude of the movement of each individual object in the object. For example, such a movement amplitude can be determined so that a plurality of objects are located (see block 3110), and then in each case the movement of the object is tracked / tracked within the corresponding area in which the corresponding object is located. In general, the movement information can thus be determined based on the location of the object.

[0120] For example, the movement information may indicate one or more movement patterns of the object. For example, it can be imagined that a certain type of surgical tool, such as an aspirator, is preferably used in a circular motion; this movement pattern (circular motion) can then be identified, and for example, it can be inferred from it that the aspirator is a low-priority auxiliary tool. On the other hand, another type of surgical tool, such as a scalpel, can move mainly in a translational manner, that is, back and forth. This movement pattern (translational movement between two end points) can then be identified, and it can be inferred that, for example, the scalpel is a high-priority primary tool. In general, when prioritizing tools, repetitive movement patterns predefined by the surgical use of a specific tool can be taken into account. Such movements of the tool are used to apply the tool in the surgical procedure itself (for example, in the above-mentioned example of an aspirator for aspirating blood, or in connection with the above-mentioned example of a scalpel for cutting tissue); therefore, they are not movements performed as part of a specific gesture recognition, and they themselves have no purpose other than performing a gesture.

[0121] A further example of movement information would be, for example, information about the directional distribution of the movement of the corresponding object. It is also conceivable to specify the movement frequency of the object's movement.

[0122] Various exemplary embodiments of mobile information have been disclosed above. In the various techniques described herein, combinations of such disclosed mobile information variants may also be used.

[0123] There are various possibilities for implementing the prioritization information. For example, a segmentation map may be output which, for one of the captured images, classifies the regions in the image showing the object into different priority classes. It is also conceivable to output labels, which are arranged at the object locations. These labels may then show the priority information. Thus, it is conceivable that the prioritization information comprises localizations (e.g. point localizations, bounding boxes, etc.) with priority labels. Figure 6. The figure shows the corresponding corresponding labels 241, 242, 243 of the surgical instruments 231, 232, 233 at the center positions of the surgical instruments 231, 232, 233. These labels 241 to 243 indicate the priorities of the surgical instruments, wherein labels 241, 243 indicate high priority; and label 242 indicates low priority. For example, the prioritization information can indicate the activity centers of related and unrelated objects. In general, the prioritization information can therefore be related to the positioning of various objects.

[0124] Prioritization information can take the form of continuous values ​​(e.g., from 1 = low priority; to 10 = high priority; and back), or can assign object categories to predefined categories; for example, the categories can be assigned to a first category and a second category (and optionally one or more additional categories). The category assignments can be binary. For example, the first category can involve relevant objects and the second category can include irrelevant objects. The auxiliary function can then be performed based only on the positioning of the one or more objects assigned to the first category (that is, ignoring those objects assigned to the second category). For example, in combination with Figure 6 , a scenario has been discussed in which surgical instruments 231 , 233 are assigned to a first category and surgical instrument 232 is assigned to a second, unrelated category.

[0125] Prioritization information can be considered in various ways when using accessibility features. A few examples are explained below.

[0126] For example, an auxiliary function may involve measuring a surgical instrument. Then, for example, the surgical instrument with the highest priority may be measured.

[0127] The auxiliary function may also include automatic configuration of one or more components of the surgical microscope system. As part of the auxiliary function, a target configuration may then be determined for one or more components of the surgical microscope system. When determining the target configuration, the positioning of those of the two or more objects that are ranked higher in priority based on the priority ranking information may be taken into account more than the positioning of those of the two or more objects that are ranked lower in priority based on the priority ranking information. A sliding transition between a high and a low degree of consideration may be envisaged, as well as a binary transition, i.e., the positioning of objects with higher priority is taken into account while the positioning of objects with lower priority is not taken into account. Different target configurations are determined based on the auxiliary function. For example, the target configuration of the surgical microscope system may include the alignment of the field of view of a microscope (e.g., a microscope camera or an eyepiece) of the surgical microscope system relative to at least one surgical instrument selected from two or more surgical instruments based on the priority ranking information. Thus, this means, for example, that the center point of all surgical instruments with higher priority is determined based on the corresponding positions, and then the robot support is driven so that the center point is located in the center of the microscope field of view, as described above in conjunction with Figure 5 As explained. However, this is only an example. It is also conceivable that only a single surgical instrument, i.e. the surgical instrument with the highest priority, is selected from a large number of visible surgical instruments, and the field of view of the microscope is automatically arranged at the relevant point of the selected surgical instrument, such as its top. In a further variant, the target configuration of the surgical microscope system includes automatic focusing taking into account multiple surgical instruments, wherein priority is performed within the surgical instrument. Therefore, in a further variant, the target configuration of the surgical microscope system may include the focusing of the microscope relative to one surgical instrument selected based on the priority information from two or more predefined surgical instruments. Therefore, automatic focusing related to high-priority surgical instruments can be provided. When focusing on the geometric center point of two or more surgical instruments (or general objects), in addition to the geometric center point, the automatic focusing can also take into account the depth center point. Therefore, this means determining the depth for all objects included in the local center point determination; and then focusing on the average value. This should be distinguished from a variant (which is also possible in principle) that focuses on the depth of the local center point.

[0128] Next, details about mapping movement information to priority information are disclosed. In other words, the following describes how priority information can be accurately determined based on movement information in block 3120.

[0129] In a first example, priority information is determined based on movement information using predefined criteria. In other words, this therefore means that the criteria used to determine priority information based on movement information is independent of the movement information itself. For example, a fixed threshold can be used and compared with the value of the movement information. A lookup table can be used to map different values ​​of the movement information to different priority information. A fixed predefined function can be used to convert the movement information into priority information. As an example, for example, the movement amplitude (e.g., quantized by light flow) can be considered. The movement amplitude can be compared with a predefined threshold. If the movement amplitude is greater than the predefined threshold, the corresponding surgical instrument is assigned to the first relevant category; otherwise it is assigned to the second irrelevant category. Reference Figure 5 Example: In the figure, the movement amplitude indicates that the aspirator 231 and the bipolar coagulator 233 move significantly (that is, the movement amplitude is greater than the threshold); but the retractor 232 does not. That is, the retractor 232 is fixed to the patient and moves only slightly with the brain tissue. In this case, the surgical instruments 231, 233 are classified into the high priority category, while the surgical instrument 232 is classified into the low priority category. In general, a fixed movement threshold can be used, including for different defined movement information. Such predefined criteria regarding movement information can be set by the user. For example, different users may have different preferences for relevant and irrelevant surgical instrument classifications (that is, different preferences for surgical instrument classifications associated with high priority or low priority). However, it is also conceivable that such criteria are fixedly preprogrammed and cannot be changed by the user.

[0130] In a second example (as an alternative or supplement to the first example described above), the priority sorting information is determined based on the movement information using a relative criterion. A relative criterion is determined based on the movement information. In other words, this means, for example, to consider the relative ratio of the movement information values ​​determined for different surgical instruments; or a threshold value adjusted based on the value of the movement information (e.g., 50% of the maximum value, etc.). For example, the ranking of the values ​​of the movement information can be considered. For example, only a single object can be included in the first category associated with a high priority; this situation is particularly useful for autofocus. The movement of various surgical instruments relative to each other can be considered. It can be considered whether the surgical instruments move toward each other or away from each other. In one variant, the fastest moving object is classified as a relevant object; all other objects are classified as irrelevant objects; in other words, this means that the fastest moving object is classified as a first category; and all other surgical instruments are grouped in a second category. Optionally, a tolerance range (e.g., 5%) can also be used. If the second fastest moving object moves at a similar speed (e.g., 95% of the fastest object's movement amplitude), the center point is taken. However, if one object moves significantly faster than all other objects, only this one object is classified as a relevant object with a high priority. Comparisons can be made between the spectra of movement frequencies of various surgical instruments. For example, it can be checked whether specific surgical instruments all have the same movement spectrum (which would indicate that these surgical instruments are not guided by the surgeon, but are fixed to the patient and move with the patient's movements).

[0131] Here is a practical example: For example, Figure 5 In the case shown, the movement amplitude can therefore be determined (e.g., based on optical flow). The ranking of the movement amplitude is aspirator 231-bipolar coagulator 233-retractor 232. Therefore, the movement amplitude of the retractor 232 is significantly lower (relatively limited) than the aspirator 231 and the bipolar coagulator 233. Therefore, it is conceivable to assign the aspirator 231 and the bipolar coagulator 233 to a high priority category, but assign the retractor 232 to a low priority category. In this example, there is no need to use a fixed threshold. This can be particularly useful if a variety of different scenarios are to be considered and it is not known a priori how the movement amplitude behaves with respect to the relevance of the object. In this case, it is helpful if a relative standard that can be applied to all scenarios in a flexible and adaptive manner is used as described above.

[0132] In a third example (again as an alternative or in addition to the examples discussed above), a machine learning model is used to determine the prioritization information. The machine learning model receives the movement information as input. The machine learning model can, for example, be trained to recognize movement patterns of relevant surgical instruments and distinguish them from movement patterns of less relevant surgical instruments. The machine learning model can then output the prioritization information with corresponding category assignments.

[0133] The machine learning model allows for more complex decision rules than in the first and second examples described above. For example, one or more of the following factors may be cumulatively considered: the distance between instruments during movement; the type of movement (e.g., slow vs. fast), the direction of movement, the angle of incidence of the instrument (to distinguish between assistant and head surgeon), the mode of movement, etc. Such criteria may be learned to be considered through appropriate training. To this end, an expert may manually annotate the corresponding input data into the machine learning model by determining the corresponding basic facts of the priority sorting information.

[0134] As a general rule, in addition to the input of movement information, the machine learning model may also receive further inputs. Examples may be, for example, the positioning of two or more surgical instruments, that is to say corresponding positioning information. For example, corresponding bounding boxes or center positioning may be transmitted. Corresponding instance segmentation maps may be transmitted.

[0135] Information about the semantic context can also be transmitted, that is to say, for example, the surgical phase or the type of surgery.

[0136] Various aspects of how to determine priority information based on movement information have been described above. It has been pointed out above in conjunction with machine learning models that other data may be considered in addition to movement information when determining priority information (this generally applies to various examples and is not limited to the use of machine learning models).

[0137] For example, the priority ranking information may also be determined based on the positioning of two or more surgical instruments. Thus, it is conceivable that an object located relatively centrally in the field of view may tend to have a higher priority than an object located peripherally.

[0138] As an alternative or in addition, the prioritization information can also be determined based on semantic context information of the scene associated with two or more surgical instruments. Examples of semantic context information are the type of surgical intervention or information indicating a stage of a surgical intervention, for example: "coagulation" or "aspiration of blood" etc. It is also conceivable to identify the type of surgery, that is to say, for example "spine, skull, tumor, blood vessel". For example, the corresponding context information can be transmitted as a further input to the machine learning model. For example, depending on the semantic context information, different lookup tables can be used to map the movement information to the prioritization information. To give just a few examples, depending on the semantic context information, different thresholds can be used to classify into higher priority categories or lower priority categories based on the movement amplitude.

[0139] As another example, it may be considered whether a particular surgical instrument is carried in the left or right hand. This may be compared to the corresponding priority ranking of the surgeon's hand in question (that is, whether the surgeon in question is left-handed or right-handed).

[0140] In short, Figure 4 , Figure 5 and Figure 6 In the example of , the prioritization information is determined based on the movement information. This has the advantage of achieving a particularly relevant and robust prioritization that can handle different types of objects. This will be explained in detail below. For example, in US10,769,443B, a distinction is made between "non-primary" and "primary" instruments. This distinction is based on three classifications, namely, first, the classification of the tool type, second, the classification of whether the tool is an auxiliary tool or a non-auxiliary tool, and third, the classification of the holding hand (right hand / left hand) of the instrument. In US10,769,443B, a clear classification of the instrument type (aspirator, retractor, etc.) is used to decide whether the instrument is a "primary" instrument or a "non-primary" instrument, or an "auxiliary" instrument or a "non-auxiliary" instrument. The type of instrument must be classified. However, in neurosurgery, there are >100 types of instruments, which are often difficult to distinguish visually. Therefore, it is technically difficult to achieve a clear classification. Even with a robust and unambiguous classification of instrument types, the solution in US10,769,443B presents another problem: in some cases, the same instrument may or may not be relevant to the procedure. For example, when the aspirator is currently aspirating blood, it is relevant; on the other hand, when the aspirator is only used to restrain brain tissue (similar to a handheld "dynamic retractor") instead of aspirating blood, it is irrelevant. This shows that the instrument type does not directly indicate the surgical relevance of the instrument or its priority in terms of auxiliary functions. In the present disclosure, this problem is solved by determining the priority sorting information in box 3120 based on the movement information from box 3115.

[0141] The features set out above and the features described below can be used not only in the corresponding combination explicitly set out but also in other combinations or alone, without departing from the scope of protection of the present invention.

[0142] By way of example, various aspects have been described above in relation to auxiliary functions involving surgical instruments. However, as a general rule, it is conceivable to consider other types of objects, such as representative anatomical features of a patient.

[0143] In addition, various aspects related to the robot support have been described above. The surgical microscope system does not absolutely need to include a robotic support. The surgical microscope system may also include a partial robotic support or a manual support.

Claims

1. A computer-implemented method for controlling a surgical microscope system (80), the surgical microscope system having a stand (82) and a microscope (84) carried by the stand (82), in, The method includes: - driving (3105) the camera (83, 86) of the surgical microscope system to obtain a sequence of images, - determining (3115) movement information for each of two or more objects (78, 231, 232, 233) depicted in the sequence of images based on the sequence of images, - determining (3120) priority ranking information for the two or more objects (78, 231, 232, 233) based on the movement information, and -Based on the priority sorting information, perform (3125) auxiliary functions related to the two or more objects (78, 231, 232, 233).

2. The computer-implemented method of claim 1, in, The prioritization information includes a category assignment that classifies the two or more objects (231, 232, 233) into at least a first category and a second category, wherein the auxiliary function is performed in conjunction with at least one object (231, 233) assigned to the first category, The auxiliary function is not performed in conjunction with at least one further object (232) assigned to the second category.

3. The computer-implemented method according to claim 1 or 2, in, The prioritization information is determined based on the movement information using at least one predefined criterion, Wherein, the at least one predefined criterion comprises a fixed movement threshold.

4. The computer-implemented method according to claim 1, in, This prioritization information is obtained without performing any classification and in particular without performing any instance segmentation of these objects.

5. The computer-implemented method according to claim 1, in, The prioritization information is determined based on the movement information using at least one relative criterion, the at least one relative criterion being determined based on the movement information.

6. The computer-implemented method of claim 5, in, The at least one relative criterion includes a ranking of movement information associated with the two or more objects (231, 232, 233).

7. A computer-implemented method according to claim 5 or 6, in, The at least one relative criterion comprises a relative movement of the two or more objects (231, 232, 233) relative to each other.

8. The computer-implemented method according to claim 1, in, The prioritization information is determined by a machine learning model that receives the movement information as input.

9. The computer-implemented method according to claim 1, in, The prioritization information is also determined based on the positioning of the two or more objects (231, 232, 233).

10. The computer-implemented method according to claim 1, in, The prioritization information is also determined based on semantic context information of the scene associated with the two or more objects.

11. The computer-implemented method according to any one of the preceding claims, wherein: The method further includes: - Receiving (3005) a user command requesting an auxiliary function related to a non-specific object among the two or more objects (231, 232, 233).

12. The computer-implemented method according to claim 1, in, The motion information includes optical flow between consecutive images of the image sequence.

13. The computer-implemented method according to claim 1, in, The prioritization information is related to the positioning of the two or more objects.

14. The computer-implemented method according to claim 1, in, The prioritization information includes a segmentation map having a prioritization label.

15. The computer-implemented method according to claim 1, in, The prioritization information includes point locations with prioritization tags.

16. The computer-implemented method according to any one of the preceding claims, wherein: Implementation of this auxiliary function includes: - determining a target configuration of the surgical microscope system (80) based on the prioritization information and the positioning of the two or more objects (231, 232, 233), and - Driving at least one component of the surgical microscope system based on the target configuration.

17. The computer-implemented method of claim 16, in, The target configuration of the surgical microscope system includes an alignment of a field of view (122, 123) of the microscope (84) relative to at least one of the two or more objects (231, 232, 233) selected based on the prioritization information.

18. A computer-implemented method according to claim 16 or 17, in, The target configuration of the surgical microscope system includes automatic focusing of the microscope (84) relative to one of the two or more objects (231, 232, 233) selected based on the prioritization information.

19. A data processing device (60) for controlling a surgical microscope system (80), wherein: The data processing device (60) comprises a processor (61) configured to load a program code from a memory (62) and execute the program code, wherein the execution of the program code causes the processor (61) to perform the following steps: - driving (3105) the camera (83, 86) of the surgical microscope system to obtain a sequence of images, - determining (3115) movement information for each of two or more objects (78, 231, 232, 233) depicted in the sequence of images based on the sequence of images, - determining (3120) priority ranking information for the two or more objects (78, 231, 232, 233) based on the movement information, and -Based on the priority sorting information, perform (3125) auxiliary functions related to the two or more objects (78, 231, 232, 233).

20. The data processing device (60) according to claim 19, wherein: Execution of the program code causes the processor to perform the method according to any one of claims 1 to 18 .

21. A surgical microscope system comprising a data processing device (60) according to claim 19 or 20.

Images