Method for detecting instruments in image data, and device for analysing image data

Abstract

The present invention relates to a method for detecting surgical instruments in medical image data, in particular for analysis and documentation purposes, said method comprising the steps of: - acquiring (101) image data (3a), preferably provided by means of an endoscope, in particular an endoscopic live image (40), - visually detecting (102) a surgical event (32) in the acquired image data, and - visually detecting (103a) at least one instrument (30a, 30b) in the acquired image data, wherein the method comprises detecting and assigning (103b) an instrument classification to the detected instrument (30a, 30b) based on the detection of the surgical event (32) and a determination of the distance between the detected surgical event (32) and an instrument end (31a, 31b) of the at least one instrument (30a, 30b).

Description

[0001] Method for detecting instruments in image data and device for analyzing image data

[0002] The present invention relates to a method for the visual recognition and classification of instruments in medical image data, in particular in an endoscopic live image.

[0003] Optical visualization systems such as endoscopes or microscopes for medical procedures are well known and enable the representation of a scene and / or a work area within a human body, for example, during a minimally invasive procedure. Such visualization systems output captured image or video data on suitable screens and are typically connected to an image recording device or system to record the captured image or video data, as well as any other device- and / or patient-specific data, and / or analyze them for further processing and / or evaluation.

[0004] Furthermore, methods and devices are known by which the image or video data acquired by an endoscope are processed for analysis and / or documentation purposes. For example, a visual recording of the surgical instruments used or surgical events occurring during the procedure can be performed. The resulting data and acquired information can then be used to monitor the course of treatment, to optimize future surgical interventions of a similar nature, or to train AI applications.

[0005] US 2019 / 0125455 A1 discloses a method for adjusting the operation of a surgical instrument using machine learning. The method includes collecting data during surgical procedures with surgical instruments, analyzing the collected data to determine a suitable operating setting of the surgical instrument, and adjusting operating parameters of the surgical instrument to improve the operation of the surgical instrument. US 2018 / 0271603 A1 discloses a control system for surgical tools with an instrument management system, a database, a display means for displaying at least one surgical event, for determining the identity and 3D position of detected objects and instruments, and at least one computing unit for identifying a necessary medical application or action based on a surgical event.

[0006] EP 3 505 113 A1 discloses a surgical system for wireless communication between surgical devices, comprising a first surgical device with a control unit configured to be informed of situational events occurring in the vicinity of the first surgical device according to data received from a database, a patient monitoring device or another surgical device coupled thereto.

[0007] US 2021 / 142487 A1 discloses a system for evaluating surgical scenes based on computer-assisted visual image analysis. The system includes the visual recognition of instruments and events and the determination and tracking of a position in the acquired image.

[0008] US 2019 / 0200985 A1 discloses a surgical instrument with a control circuit for monitoring a distance of the instrument from a tumor tissue. The instrument has an end effector for grasping tissue and a staple cartridge comprising staples insertable into the tissue and a sensor. The surgical instrument includes a control circuit coupled to the sensor, wherein the control circuit is configured to receive the sensor signal and analyze the proximity of the sensor to cancerous tissue based on the sensor signal.

[0009] Based on the known prior art, the object of the present invention is to provide a method and a device for the visual analysis of medical image data, which enables optimized recognition and classification of instruments in the image data. This object is achieved by the recognition method and the analysis device according to the independent claims. The dependent claims describe advantageous developments of the present invention. All combinations of at least two of the features disclosed in the claims, the description, and / or the figures fall within the scope of the invention.

[0010] In a first aspect, the present invention relates to a method for detecting surgical instruments in medical image data, in particular for analysis and documentation purposes, comprising the steps:

[0011] -Capturing image data, preferably provided by an endoscope, in particular an endoscopic live image,

[0012] -visual detection of an operative event in the acquired image data, and -visual detection of at least one instrument in the acquired image data, wherein the method comprises detection and assignment of an instrument classification to the acquired instrument based on the detection of the operative event and a distance determination between the detected operative event and an instrument end of the at least one instrument.

[0013] In this context, an instrument end is understood to mean, in particular, the distal end or a distal end region of an instrument. The instrument end is, in particular, a part of the instrument that can be visually detected by an image capture device that captures the image data. Alternatively or additionally, the instrument end can also be defined as a region of interest (ROI), i.e., a region in the image in which the instrument end is recognized or suspected. The instrument end can be visually detected and identified within the image, in particular by markings on the instrument end or by segmentation.

[0014] The present invention enables the preferably automatic determination and assignment of at least one attribute and in particular an instrument classification of a surgical instrument captured in the image data based on the evaluation of the captured image data. Visual capture and visual recognition are understood here to mean a respective software-based capture or recognition using known software algorithms and / or using trained artificial intelligence (K1), which performs a visual analysis or evaluation of the captured image data. This involves, for example, capturing contours and / or structures of an instrument and / or tissue and / or analyzing further image elements and / or image data, based on which a surgical event or an instrument can be recognized, e.g., by comparing it with previously stored data.Visual detection and recognition differs in particular from sensor- or antenna-based detection or recognition of objects in image data or a video image.

[0015] An operative event is understood to be an event that can be visually detected in image data, in particular a visually recognizable effect of an instrument application.

[0016] Image data in this case refers to image and / or video data, and in particular image data provided by a video endoscope, exoscope or medical microscope.

[0017] In contrast to the prior art, the invention does not involve visual recognition of an attribute of the surgical instrument based on the image data of the captured instrument alone, but rather recognition and assignment of a classification or classification of the surgical instrument based on a distance determination between a respective instrument end and a detected surgical event. This enables an optimized and, in particular, more precise classification of the respective instrument. For example, different surgical instruments that have a similar contour / surface in image data and are therefore difficult or impossible to distinguish when analyzing the image data of the instrument alone can be more precisely identified or classified using an associated surgical event.The method preferably comprises the further step of storing and / or outputting the determined data, in particular the recognized and assigned instrument classification, preferably together with the acquired image data. For this purpose, an output means, such as a display, can advantageously be provided in which, in addition to the acquired image data such as a live endoscopic image, the determined instrument classification and / or the further acquired and / or determined data are presented to a user. Furthermore, the image data can preferably be stored together with the assigned data in a provided internal or external data storage device. The storage device can, for example, be a buffer connected to an image acquisition device, which transmits the data at regular or irregular intervals to a provided database, in particular for documentation and / or evaluation purposes.

[0018] In a preferred embodiment, upon visual detection of two or more instruments or associated instrument ends in the acquired image data, the instrument classification is recognized and assigned to the instrument whose instrument end has the shortest determined distance to the detected operative event. Preferably, after detection and recognition of an operative event and after detection of at least two instrument ends in the image data, the respective distance between the instrument ends and the operative event is first determined. Subsequently, the instrument end that is detected closest to the operative event is assigned as the instrument classification described above.This is also advantageously done when at least two or more surgical events are detected, in which case a distance determination to the surgical instruments visually captured in the image is then performed for each surgical event. This allows for a more precise classification of the instrument assigned to a surgical event when more than two surgical instruments are captured in the image data.

[0019] The distance determination between the operative event and the instrument end of the detected instrument is preferably based on a distance or range measurement that can be directly visually detected from the image data. Advantageously, this can be a known software-based distance measurement between two image points, in particular between the respective centers of the instrument end and the detected event, and / or an image analysis based on an additional depth measurement, which determines the distance between the image capture device and the scene, in particular the operative event, for individual image points. For example, the depth can be determined by a sensor such as a time-of-flight sensor. Alternatively, a depth can be assigned to individual image points of the scene through triangulation, for example using a stereo instrument.A depth map of the scene can be determined, particularly continuously, and this information can be used to determine the spatial distance between real points in the scene, particularly the end of the instrument and the operative event. The distance determination is then performed in three dimensions. This allows for efficient analysis of the image data with sufficient accuracy, without the need for complex additional devices such as sensors and / or antennas on the respective instruments.

[0020] Alternatively or additionally, a region of interest (ROI) can be defined for the detected instrument end and the surgical event in the image. For example, a rectangle or other shape enclosing the instrument end and the ROI can be defined. The distance can then be a distance between specific points of these rectangles or shapes, for example, between the centers.

[0021] The step of determining the distance between the operative event and the end of the recorded instrument preferably occurs immediately after visual detection of the operative event and / or the detection of the instrument. This allows for a more precise association between a particular instrument and the operative event. In particular, this allows for a measurement to be taken as close as possible to the point of origin of the operative event, thus avoiding any expansion or spatial change, or migration of the operative event in the image data, which could negatively impact the measurement.

[0022] Detection of a surgical event preferably comprises event classification from the analyzed image data into one or more of the following event classifications: smoke development, a color or structural tissue change, a developing and / or changing tissue opening, a change in the water content of the detected tissue, blood development in the tissue, suturing in the tissue, introduction of foreign material into the tissue, in particular introduction of metal, for example a clip, and / or spray mist or gas bubbles.

[0023] For the visual detection of a surgical event, the surgical event is preferably visually recorded and detected using computer- or software-assisted image evaluation or analysis, in particular using known image evaluation algorithms. Advantageously, this involves evaluating the RGB image values, evaluating multispectral and / or hyperspectral image parameters, identifying contours / structures of the detected objects in the image, and / or evaluating them using known algorithms for object recognition in image data, in particular comparing them with previously recorded and / or stored image data of known and thus predefined surgical events. For example, water content can be determined by evaluating hyperspectral or multispectral image parameters.Detection can also be performed by or with the support of artificial intelligence, for example, using a neural network or based on machine learning. These are then trained to detect observed operational events.

[0024] In visual detection, an evaluation can also be performed over time or across multiple consecutive frames of a video image or image data. For example, a tissue change can be detected by evaluating the RGB values ​​over time, particularly by detecting color changes, such as a change in tissue color from pink to black to detect tissue damage, or a change in image color information to white or gray to detect smoke.

[0025] The step of detecting a surgical event can further preferably comprise assigning an event intensity and / or another event attribute. This can also be performed using the aforementioned, known computer- or software-assisted image evaluation or analysis. For example, a visual detection and subsequent assignment of a smoke intensity, a bleeding intensity, and / or a tissue opening size can be performed. For this purpose, a respective intensity or size factor can be assigned to the event. For example, depending on the extent or density of smoke development on the tissue that is visually detectable in the image, a lower or higher intensity value can be determined and assigned to the surgical event.

[0026] In a preferred embodiment, the recognition and assignment of an instrument classification comprises a recognition and assignment of an instrument type and / or a further instrument attribute, in particular comprising the classification: HF instrument, non-HF instrument, forceps, scissors, suture instrument, stapler, biopsy instrument, punch, clip setter and / or an ultrasound device.

[0027] According to the invention, the recognition and assignment of an instrument classification of the detected instrument is based on a detected surgical event in addition to the distance determination described above. The detected event comprises information or data on a detected event classification, a determined event intensity, and / or data on another event attribute. For example, by detecting smoke development on the tissue that can be detected in the image data and which was or is assigned to a detected instrument end by the distance determination, the corresponding instrument can be classified as a high-frequency (HF) instrument for the application of high-frequency current, for example for coagulation or cutting. This makes it possible to differentiate between HF instruments such as HF grasping forceps, which visually do not or only barely differ from grasping forceps without HF functionality.

[0028] In this case, the recognition and assignment of an instrument classification can be carried out not only on the acquired visual image data of the instrument but also based on the evaluation of device and / or procedure data stored in a database for predefined instruments and / or instrument data stored for predefined procedures or predefined events. Thus, the recognition and assignment of an instrument classification can include the step of querying data stored and / or retrievable in an internal or external data storage device, which assigns surgical events to a respective instrument or a respective instrument property. For example, the assignment can be stored in a storage device that the surgical event of smoke development in the tissue can be triggered by a surgical instrument with HF functionality, or that a resulting tissue opening or tissue division can be triggered by scissors.

[0029] The method preferably comprises the further step of determining and assigning a type of medical treatment as an attribute to the analyzed image data. Thus, by recognizing the respective surgical event and / or the recognized instrument classification, an existing medical treatment or treatment type can be inferred, which can be assigned as an attribute to the acquired image data.

[0030] Furthermore, the method can comprise the step of reading out and / or evaluating device and / or instrument data which are used for the respective procedure and which are, for example, connected to a device for carrying out the procedure. For example, electronically readable device data can be analyzed and used for the further classification of an instrument, for the recognition of a surgical procedure and / or for the recognition of a type of medical treatment or an indication. If, for example, an HD rhinoscope is recognized as being connected to a corresponding device, it can be concluded that this is a diagnostic procedure because this endoscope does not have a working channel. If, for example, a cystoscope is recognized as being connected to a corresponding device, it can be assumed that this is an endourological procedure.

[0031] Based on this, it is also possible to determine, prior to image analysis, whether a particular classification and / or surgical event is likely or unlikely. For example, if an endourological procedure is performed underwater, the result will be bubbles rather than smoke. Furthermore, there are purely diagnostic procedures, for example, in which no HF instruments are used. This has the advantage that the respective image analysis algorithm can be designed more efficiently, since only a limited number of instruments or instrument classifications are relevant for the specific procedure, rather than all of them.

[0032] Furthermore, the method can include the step of temporally recording each use of a classified surgical instrument and / or a surgical event. This can include, in particular, the respective time and duration of use or the duration of a detected event. The corresponding data can be assigned to the image data and used for further analysis, in particular for classifying the instrument and / or detecting an event.

[0033] The method preferably further comprises calculating and assigning a validation value for the respective recognized surgical instrument, which is determined at least based on a distance value of the respective distance determination between a recognized surgical event and an instrument end. The validation value can serve to provide appropriate plausibility of the recognized instrument classification for the respective surgical instrument. Preferably, a higher validation value is assigned for a relatively short determined distance between the surgical event and the instrument end than for a relatively greater distance. This is based on the fact that with a directly nearby surgical event, the probability that the surgical instrument is responsible for the surgical event is higher than with a relatively greater distance between the instrument and the event.The validation value is preferably assigned to the image data and can be stored together with the other data and / or values ​​in a corresponding memory.

[0034] The method can optionally include manual validation, for example, validation of the recognized instrument classification by means of user interaction, either during or after the procedure. This can advantageously include outputting the recognized instrument classification to a user, such as a treating physician, using provided output means and capturing manual feedback using provided input means. In addition to outputting the recognized instrument classification, the method can include outputting additional associated data, in particular outputting an associated validation value. Further preferably, optional manual validation can be performed only for the classification data or recognized instruments to which a relatively low validation value is assigned.This allows for an optional manual review of less plausible data, which on the one hand improves the overall validity of the collected data and on the other hand only carries out a manual review of the data identified as less valid, thus optimizing the overall efficiency for valid data collection.

[0035] In a further preferred embodiment, the method can also include the step of transmitting a feedback signal to a device connected to the detected instrument(s) based on the data acquired by the image analysis. This allows, in particular, the respective instrument to be controlled based on the data acquired by the image analysis. For example, if excessive tissue damage is detected during an RF current application, a corresponding control can be initiated to reduce the power of the device or generator connected to the RF instrument in order to prevent further tissue damage.

[0036] In a further aspect, the invention relates to a device for acquiring and analyzing medical image data, which is designed to carry out the method as described above, comprising:

[0037] - a connection unit for selectively connecting an image acquisition device for medical treatment, in particular an endoscope,

[0038] - a data acquisition unit for acquiring image data from at least the image acquisition device connected to the connection unit, and

[0039] - a computing unit configured to visually detect and recognize an operative event in the acquired image data, wherein the computing unit is further configured to determine the distance between the detected operative event and an instrument end of at least one instrument visually detected in the image data, as well as to detect and assign an instrument classification to the acquired instrument based thereon. In a preferred embodiment, the device comprises a documentation unit for recording image data provided by the data acquisition unit and the data determined by the computing unit, in particular comprising the determined instrument classification. In this case, the documentation unit can advantageously be further configured to record depending on a determined instrument classification.This allows selective data collection and documentation to be carried out only for data that is of greater interest for later documentation or analysis, thus enabling more efficient use of data and computing capacity.

[0040] The documentation unit is preferably designed to link the respective image data with the data acquired and / or determined by the computing unit and to store them as associated data or as a correspondingly linked data set. The determined and / or assigned data can be stored as metadata of the respective image data. This allows the image data to be clearly assigned to the acquired data at a later point in time and, in particular, to avoid human error during the separate recording of image data known from the prior art, on the one hand, and during the respective examination of additionally acquired data sets, on the other. The documentation unit can further be designed to store the respectively assigned data as metadata of the image data in predefined frames or, more preferably, in each individual frame of the image data.

[0041] Furthermore, the computing unit can be configured to adjust and / or adapt the data acquisition unit, in particular the recording parameters that influence data recording, and to control the data acquisition unit depending on the determined instrument classification and / or the additionally acquired data. For example, a frame rate, an image section, and / or other image parameters of the acquired image signal can be adjusted or adapted depending on the determined and / or assigned data.

[0042] The computing unit advantageously comprises a memory unit configured to store and / or assign attributes such as an instrument classification, an event classification, an event intensity, a type of medical treatment, an assigned validation value for a respectively recorded event and / or a respectively recorded surgical instrument. The computing unit may, in particular, comprise one or more lookup tables, by means of which, for example, predefined and / or modifiable attributes are assigned to a respectively stored surgical event, which can be used, for example, for instrument classification.

[0043] In a preferred embodiment, the data acquisition unit is configured to acquire additional device data and / or parameter data from devices connected to the device either wired or wirelessly. The data acquisition unit is also advantageously configured to acquire provided patient data and / or vital parameters of a patient. The computing device can advantageously be configured to determine attributes of the respective surgical instrument or the respective medical procedure from a combination of the acquired device data and / or parameter data and the acquired image data.

[0044] In a preferred embodiment, the device further comprises a data transmission unit which is designed for preferably bidirectional data exchange with an external network and / or an external server unit. The data transmission unit can have a LAN and / or WLAN interface for connection to an internal or external network, such as a hospital LAN or the Internet. The device can be designed to capture data capture commands from the external server unit and / or for the automated and / or non-automated transmission of recordings from the data capture unit to the external server unit. The transmission to an external network and / or to an external server unit can take place in real time and / or with a time delay to the recording of the documentation unit.Particularly preferably, the acquired data sets can be transmitted at predefined time intervals or at predefined transmission times. The data can be temporarily stored in external or internal storage units. The device described above can be designed as a separate or integral component of an endoscopy device or an endoscopy system.

[0045] To avoid repetition, reference is made here to the above-mentioned statements on the method according to the invention, which are to be considered equally as disclosed and claimable for the device according to the invention, and vice versa.

[0046] Details, advantageous effects and details of the present invention are explained below with reference to the schematic, merely exemplary drawings.

[0047] Showing:

[0048] Fig.1: a schematic representation of a preferred embodiment of the device according to the invention for analyzing medical image data;

[0049] Fig. 2: an exemplary live image of an endoscopic procedure with analysis of the acquired image data; and

[0050] Fig. 3: a preferred embodiment of the method according to the invention as a schematically simplified block diagram.

[0051] Fig. 1 shows a preferred embodiment of the device 10 according to the invention for analyzing acquired medical image data. The device 10 initially comprises a connection unit 1 for selectively connecting an image acquisition device 2 for a medical treatment or a medical procedure. This is, in particular, an endoscope, more preferably a video endoscope. The connection unit can have known plug connections, which serve for the wired connection of the endoscope to the device.

[0052] The connection unit 1 can also be configured to connect additional devices, such as an input unit 7, for example, a keyboard and / or a mouse, and a display unit 8 for outputting a captured video image to a treating physician. For this purpose, the connection unit 1 can have wired or wireless interfaces for the respective connection and / or communication of the devices 7, 8.

[0053] The device further comprises a data acquisition unit 3 for acquiring at least image data 3a provided by the image acquisition device 2. In addition, the data acquisition unit 3 can be configured to acquire device data 3b from at least the image acquisition device 2 connected to the connection unit 1. The device data 3b can include, for example, a device identification number, a device identifier, or the like, based on which an identification and / or recognition of the endoscope 2 connected to the device 10 is possible.

[0054] The data acquisition unit 3 can additionally acquire patient data and / or vital parameters 3c associated with the patient. The patient data and / or vital parameters 3c can either be entered manually using appropriate input devices 7 or transmitted automatically by other devices connected to the device.

[0055] The device further comprises a computing unit 4, which is configured to visually detect and recognize a surgical event in the acquired image data 3a. Furthermore, the computing unit 4 is configured to determine the distance between the detected surgical event 32 and an instrument end 31a, 31b of at least one surgical instrument 30a, b visually detected in the image data 3a (see Fig. 2), as well as to detect and assign an instrument classification to the detected instrument based thereon. The computing unit 4 preferably comprises an internal memory 4a, on which suitable software algorithms are stored and / or retrievable.

[0056] The device 10 preferably further comprises a documentation unit 5 for recording image data 3a, device data 3b, and / or the further patient data and / or vital parameters 3c provided by the data acquisition unit 3. For this purpose, the documentation unit 5 advantageously comprises an internal or external storage unit 5a, in particular a non-volatile memory, on which the supplied data can be stored.

[0057] The computing unit 4 can further advantageously be designed to control the documentation unit 5 for activating and / or deactivating a recording, and / or for setting and / or adapting recording parameters influencing the data recording depending on the determined and / or assigned attributes.

[0058] In contrast to pure data logging of all data during an examination with an endoscope, a recording can, for example, only be activated or deactivated if a predefined instrument classification and / or a predefined attribute of the surgical instrument or a recognized surgical procedure is met or not met. This can be particularly advantageous for data collection for training AI applications, since, for example, only data is recorded that appears relevant for the respective desired application. In addition to or as an alternative to the aforementioned configuration, the computing unit 4 can be configured to enable manual activation of the documentation unit 5, in particular by a treating physician.For example, data recording could be triggered when there is interest in a particular type of treatment, so that the next time a similar procedure is available, the corresponding image data is also saved.

[0059] The device 10 preferably further comprises a data transmission unit 6, which is designed for preferably bidirectional data exchange with an external network and / or an external server unit 20. The data transmission unit 6 can have a LAN and / or WLAN interface for connection to an internal or external network, such as a hospital LAN or the Internet. This allows the device to transmit the recorded data to an external network and / or an external server unit 20. The respective transmission can take place during the actual recording of data and thus during a medical treatment. Alternatively, a respective transmission can also take place at predefined time intervals.The device 10 can further be designed to capture data capture commands by the external server unit 20 and / or to transmit records of the documentation unit 5 to the external server unit 20 in an automated and / or non-automated manner.

[0060] With reference to Figs. 2 and 3, a preferred embodiment of the method according to the invention is described below, which can be carried out with the device 10 described above.

[0061] In a first step 101, the image data provided by the endoscope 2 is acquired. In particular, a live image 40 of the endoscope is acquired, as shown in Fig. 2, and is output, for example, on a display unit 8 of the device 10.

[0062] In a further step 102, an operative event 32 is detected in the acquired image data. This step preferably initially comprises a visual detection 102a, i.e., a spatial localization of an operative event 32 in the image data. This is performed using a corresponding software-based algorithm in the acquired image data and can be displayed or output in the live image 40 as a preferably rectangular, frame-like boundary 32a.

[0063] Simultaneously and / or subsequently, a visual recognition 102b of the surgical event in the image data occurs. A corresponding algorithm is used to preferably compare the acquired image data with stored and / or retrievable image data of predefined surgical events, and to classify the event based thereon, for example, into one or more of the following event classifications: smoke development, a color or structural tissue change, a developing and / or changing tissue opening, a change in tissue water content, blood development, suture placement, introduction of foreign material into the tissue, in particular the introduction of metal, for example, a clip, and / or a spray mist or gas bubbles. The recognized event classification, for example, the presence of smoke, can be output in a text field A assigned to frame 42.Optionally, this step may also include detecting and assigning an event intensity and / or another event attribute in a corresponding step 102c. For example, the intensity of the event, such as smoke intensity, can be classified by appropriate image analysis and output as associated information in a text field B assigned to frame 32a.

[0064] In a further step 103a, the surgical instruments 30a, 30b are visually detected in the image data. This step preferably comprises the detection, i.e., a spatial localization of the surgical instruments 30a, 30b visually recognizable in the image data and their respective associated instrument ends 31a, 31b. This is performed using a corresponding software-based algorithm by analyzing the acquired image data and can be displayed or output in the live image 40 as a preferably rectangular, frame-like boundary 33a, 33b.

[0065] In a further step 103b, the surgical instruments 30a, 30b are visually recognized in the image data. A corresponding algorithm can first be used to compare the acquired image data with stored and / or retrievable image data of predefined surgical instruments and their instrument ends, and then, based on this, to recognize an instrument type. The corresponding recognized instrument type, for example, a hook or a gripper, can then be output in a text field C1, C2 assigned to the frame 33a, 33b.

[0066] Step 103b further comprises recognizing and assigning an instrument classification to the instrument. According to the invention, this is done based on the recognition of the operative event in the preceding step 102 and a distance determination between the recognized operative event and a respective instrument end. First, the respective distance between the instrument ends 31a, 31b and the operative event 32 is determined. The distance determination between the operative event 32 and the respective instrument end 31a, 31b of the detected instrument is preferably performed using a software-based distance measurement between two image points, in particular between the respective centers of the instrument end 31a, 31b and the detected event 32, and / or based on an additional depth measurement, which methods are known per se.The instrument end 31a, 31b, which is recorded closest to the operative event 32, is then assigned as the relevant instrument classification. In this case, the instrument end 31a is closer to the operative event 32 than the instrument end 31b, so the former is assigned to event 32.

[0067] Based on the detected surgical event 32, the instrument 30a most closely associated with the event can now be classified. For example, by detecting the surgical event as smoke development in the tissue, it can be concluded that the corresponding instrument is to be classified as a high-frequency or HF instrument. In particular, HF instruments such as HF grasping forceps, which are visually indistinguishable or only slightly different from grasping forceps without HF functionality, can be differentiated from such instruments. Further examples of instrument classification include: HF instrument, non-HF instrument, forceps, scissors, suture instrument, stapler, biopsy instrument, punch, clip applicator, and / or an ultrasound device.

[0068] The corresponding classification is preferably performed by retrieving and / or evaluating data stored, for example, in look-up tables. The recognized instrument classification is also assigned to the image data, i.e., stored in a corresponding memory for later evaluation and / or documentation purposes. Furthermore, the recognized instrument classification can also be output in the live image, for example, in a text field D assigned to frame 33a. This enables optimized instrument recognition in medical image data such as an endoscopic live image. In particular, an instrument classification can be performed that would not be possible using purely optical or visual image analysis.

[0069] In a further preferred step 104, the recorded and recognized data are saved, in particular for documentation and / or evaluation purposes. The recorded and recognized data, in particular the recognized operational event and / or a recognized intensity of the event, can be used not only for pure analysis and / or documentation purposes but also to control the device 10 and / or a device connected to the device. For example, based on the recognized event intensity, such as the smoke intensity, an RF power of an RF device connected to the device can also be controlled. For example, in the case of a relatively high event intensity, for example very heavy smoke development, the current strength of the device connected to the device can be reduced by appropriate control.

[0070] List of reference symbols

[0071] 1 connection unit

[0072] 2 image capture device

[0073] 3 Data acquisition unit

[0074] 3a Image data

[0075] 3b Device data

[0076] 3c Patient data

[0077] 4 Control unit

[0078] 4a Storage unit

[0079] 5 Documentation unit

[0080] 5a Storage unit

[0081] 6 Data transmission unit

[0082] 7 Input devices

[0083] 8 Display unit

[0084] 10 Device

[0085] 20 server units

[0086] 30a, b surgical instrument

[0087] 31a,b Instrument end

[0088] 32 operational event

[0089] 32a visual boundary event

[0090] 33a, b visual boundary instrument end

[0091] 40 Live image endoscope

[0092] 101 Recording operational events

[0093] 102 Detection of operational event

[0094] 103a Recording Instrument

[0095] 103b Recognition Instrument Classification

[0096] 104 Storage

Claims

Patent claims 1 . A method for detecting surgical instruments in medical image data, in particular for analysis and documentation purposes, comprising the steps: - capturing (101) image data (3a) preferably provided by means of an endoscope, in particular an endoscopic live image (40), - visual detection (102) of an operative event (32) in the acquired image data, and - visual detection (103a) of at least one instrument (30a, 30b) in the detected image data, characterized in that the method comprises a recognition and assignment (103b) of an instrument classification to the detected instrument (30a, 30b) based on the recognition of the operative event (32) and a distance determination between the recognized operative event (32) and an instrument end (31a, 31b) of the at least one instrument (30a, 30b).

2. Method according to one of the preceding claims, characterized in that, when two or more instruments (30a, 30b) are visually detected in the detected image data, the recognition and assignment of the instrument classification is carried out for the instrument whose instrument end (31a, 31b) has the shortest determined distance to the detected operative event (32).

3. Method according to claim 1 or 2, characterized in that the determination of a distance between the operative event (32) and the instrument end (31a, 31b) of the detected instrument (30a, 30b) is carried out based on a distance measurement that can be directly visually detected from the image data, in particular between respective centers of an instrument end and a detected event, and / or based on additional depth measurement data.

4. Method according to one of the preceding claims, characterized in that the determination of a distance between the operative event (32) and the instrument end (31 a, 31 b) of the detected instrument (30a, 30b) takes place immediately after visual detection of the operative event and / or detection of the instrument.

5. Method according to one of the preceding claims, characterized in that the visual detection of the operative event (32) comprises an event classification into one or more of the following event classifications: a smoke development, a color or structural tissue change, a developing and / or changing tissue opening, a change in the water content of the tissue, a blood development, a suture formation, an introduction of foreign material into the tissue, in particular a metal introduction, for example a clip, and / or a spray mist or gas bubbles.

6. Method according to one of the preceding claims, characterized in that the visual recognition of the operative event (32) comprises an assignment of an event intensity and / or a further event attribute, in particular comprising a smoke intensity, a bleeding intensity and / or a tissue opening size.

7. Method according to one of the preceding claims, characterized in that the recognition and assignment of an instrument classification comprises a recognition and assignment of an instrument type and / or a further instrument attribute, in particular comprising HF instrument, non-HF instrument, forceps, scissors, suture instrument, stapler, biopsy instrument, punch, clip setter and / or an ultrasound device.

8. Method according to one of the preceding claims, characterized in that the recognition and assignment of an instrument classification of the detected instrument (30a, 30b) is additionally based on a recognized event classification, an event intensity, and / or another event attribute.

9. Method according to one of the preceding claims, characterized in that the visual recognition of an operative event (32) is carried out by means of computer-assisted image evaluation algorithms, in particular by evaluating the RGB image values ​​over time, by evaluating multispectral and / or hyperspectral image parameters and / or by using algorithms for object recognition in image data.

10. Method according to one of the preceding claims, characterized in that the recognition and assignment of an instrument classification of the detected instrument (30a, 30b) is carried out not only on the basis of the detected visual image data of the instrument but also on the basis of the evaluation of device and / or intervention data stored in a database for predefined instruments.

11. Method according to one of the preceding claims, characterized in that the method comprises the further step of determining a recognition and assignment of a type of medical treatment as an attribute for the analyzed image data.

12. Method according to one of the preceding claims, characterized in that the method comprises calculating and assigning a validation value for the respective instrument (30a, 30b), which is determined at least based on a distance value of the respective distance determination between a detected operative event (32) and an instrument end (31a, 31b).

13. Method according to one of the preceding claims, characterized in that the method comprises a manual validation of the recognized instrument classification by means of user interaction, in particular comprising an output of the recognized instrument classification to a user by means of provided output means (8) and a recording of a manual feedback by means of provided input means (7).

14. Method according to one of the preceding claims, characterized in that the method comprises the further step (104) of storing the recorded and recognized data, for example the recognized event, the instrument classification, the distance or other attributes for documentation and / or evaluation purposes.

15. Device for analyzing medical image data, which is designed to carry out the method according to one of the preceding claims, comprising: - a connection unit (1) for selectively connecting an image capture device (2) for medical treatment, in particular an endoscope, - a data acquisition unit (3) for acquiring image data (3a) from at least the image acquisition device (2) connected to the connection unit (1), and - a computing unit (4) which is designed to carry out a visual detection and recognition of an operative event (32) in the captured image data (3a), characterized in that the computing unit (4) is further designed to carry out a distance determination between the detected operative event (32) and an instrument end (31a, 31b) of at least one instrument (30a, 30b) visually detected in the image data (3a), and a detection and assignment of an instrument classification to the detected instrument based thereon.

16. Device according to claim 15, characterized in that the device comprises a documentation unit (5) for recording image data (3a) provided by the data acquisition unit (3) and the data determined by the computing unit (4), in particular comprising the determined instrument classification, wherein the documentation unit is preferably designed for recording as a function of a determined instrument classification.

Images