Multi-step classification

Abstract

The present invention relates to a computer-implemented multi-step method for detecting and classifying objects in medical image data, comprising: capturing a first digital image of an image field having a first image information density; creating a first version of the first digital image, wherein at least one section has a lower image information density; applying a first object detection model, directed to the detection of at least one object, to the first version of the image; identifying at least one object in the first version of the image and determining a corresponding region of interest (ROI); extracting a first partial image, corresponding to the ROI, from the first image with the first image information density; applying a second object detection model, directed to the detection of at least one object, to the first partial image; and identifying at least one object in the first partial image.

Description

Multistep classification Technical field [OOO1] The present invention relates to methods and devices for detecting and classifying objects in medical images, in particular in endoscopic images or images taken by a room camera in an operating room. background

[0002] When people look at an image, they automatically identify the objects as different instances and assign them individual definitions; they recognize and classify the objects present in the image. Replicating or performing such object recognition and classification in images using machines is extremely complex and requires considerable computing power.

[0003] Machine-based image recognition (MAC) is a key aspect of computer vision and refers to the ability of computers to identify and understand objects, patterns, or features in images or visual data. The integration of artificial intelligence (AI)-based technologies has revolutionized image recognition in recent years, opening up a multitude of applications and possibilities. These include applications in augmented reality, the detection of irregularities in production processes, medical diagnostics, and the personalization of user experiences. With the continuous development of AI technologies, image recognition capabilities are becoming increasingly versatile and precise, thus expanding the range of applications.

[0004] Image recognition is a complex task that typically requires several processes to analyze and understand the content of an image. The basic processes of image recognition include: • Image capture: The first step is to capture the image or record it using a camera or other sensors. This process involves obtaining a digital representation of the visual scene. • Preprocessing: Before analysis can begin, images often undergo preprocessing steps that include enlarging or reducing, normalization, noise reduction, and other enhancements or modifications to improve image quality and / or make the image suitable for further analysis. • Feature extraction: This step involves the identification and Extraction of relevant features from the image, such as edges, textures, colors, shapes, or other distinctive properties. Feature extraction simplifies the image data while preserving important information. • Segmentation: This step involves dividing the image into meaningful regions or objects. This process helps to isolate different elements in the image, which facilitates the recognition of individual objects. • Feature representation: After segmentation, features are represented in a way that is suitable for analysis. This may involve converting them into vectors or other data structures that can be processed by machine learning algorithms. • Feature classification: In this step, machine-based methods are usually used. Learning algorithms or deep learning models are used. The extracted features are used to classify or categorize the image into predefined classes or labels. This is where the actual recognition of objects or patterns in the image takes place. Post-processing: After classification, post-processing can be performed. These techniques can be applied to refine the results. This can include techniques such as filtering, smoothing, or refining boundaries to improve the accuracy of the detection. • Object localization: In some cases, it is important not only to classify objects but also to locate them within the image. Object localization involves identifying the position or area of ​​an object in the image, often using bounding boxes. • Contextual analysis: Contextual analysis considers the relationships and interactions between different objects in the image. It improves recognition accuracy by taking the scene as a whole into account. • Interpretation and decision-making: Based on the results of classification, localization, and contextual analysis, the system makes decisions or interpretations. This can include providing labels, giving recommendations, or triggering actions based on what is recognized in the image. • Feedback and learning: In many cases, image recognition systems are continuously improved through feedback mechanisms. This can include retraining the model with additional data or fine-tuning the algorithms to improve performance.

[0005] The design of the individual processes and their combination can vary depending on the specific application and the complexity of the recognition task. However, image acquisition and preprocessing are generally of fundamental importance. Regarding preprocessing, it should be noted that a fundamental problem of computer-based image recognition is that models or algorithms typically require the data to be analyzed to be in a specific format and therefore may need to be converted into a suitable format during preprocessing before the actual image recognition process. to be converted. For example, the appropriate image size depends on several factors, including the specific recognition task, the architecture of the neural network, and the available computing resources.

[0006] Many image recognition models are designed to work with standardized image sizes, with square images being a common choice due to their simplification of architecture and computations. For example, various Convolutional Neural Networks (CNNs) use image sizes of 224x224 or 299x299 pixels. Larger image sizes generally result in better recognition accuracy because they contain more detail. However, larger images also require more computing resources, which can impact processing speed. Conversely, smaller images can usually be processed faster, so smaller image sizes are often used for real-time or latency-critical applications where processing speed is critical. Nevertheless, reducing the image size leads to a loss of image information and thus regularly results in a decrease in recognition accuracy.

[0007] Images with varying aspect ratios, as produced by most image capture systems, can be converted into a square format. However, these changes lead to distortion and thus impair image information. To maintain the aspect ratio, images are often converted to a square format, preserving the original aspect ratio, and then filled with black or white bars. This adds "empty" information and therefore does not fully utilize the potential information content of the pixels.

[0008] In cases where very high-resolution images are used, tile- or patch-based approaches are sometimes employed, whereby the image is divided into smaller areas and each area is then analyzed individually. This essentially preserves the image information content, but because all areas are analyzed, these image recognition methods are comparatively computationally intensive and therefore time-consuming – even if the The analysis of each individual area proceeds relatively quickly.

[0009] Accordingly, it is an object of the present invention to overcome, at least partially, the limitations or deficiencies of the methods known from the prior art. Summary of the invention

[0010] The object of the present invention is to overcome, at least partially, the disadvantages known in the prior art. This object is achieved by a method according to claim 1, a system according to claim 7, a computer program according to claim 9, and a computer-readable storage medium according to claim 10. Preferred embodiments of the invention are the subject of the corresponding dependent claims.

[0011] Accordingly, the present invention discloses a computer-implemented, multi-stage method for the detection and classification of objects in medical image data, comprising the acquisition by an imaging device, in particular an endoscope system or a room camera in an operating room, of a first digital image of an image field with a first image information density. A first version of the first digital image is then created, wherein at least one section has a second image information density. The second image information density is lower than the first image information density.Subsequently, a first object recognition model aimed at recognizing at least one object is applied to the first version of the image, at least one object is identified in the first version of the image, a "region of interest" (ROI) containing the at least one object is determined, the ROI is transferred to a corresponding area in the first digital image, a first sub-image corresponding to the ROI is extracted from the first image with the first image information density, a second object recognition model aimed at recognizing at least one object, in particular a binary or multi-class classifier, is applied to the first sub-image, and at least one object is identified in the first sub-image.

[0012] Computer-implemented, multi-stage methods for the detection and classification of objects in medical image data, as defined in the present invention, comprise methods in which two independent object identification methods are applied to the image data. The individual object identification methods can be traditional, multi-stage approaches, single-stage approaches, or combinations thereof, as known to those skilled in the art. The first object identification method according to the invention can also be limited to the detection of regions of interest (ROIs) without requiring classification at that stage. In preferred embodiments, both object identification methods also include classification. In principle, the present invention is not limited to specific object identification methods; the key feature is that two object identification methods are applied sequentially to variations of the medical image data.

[0013] An imaging device within the meaning of the present invention can be any technical and / or electronic device suitable for capturing, processing, and / or transmitting an image of a viewing area. Accordingly, imaging devices within the meaning of the present invention can include, among others, all types of endoscopes known to those skilled in the art, in particular dual endoscopes and stereo endoscopes. An endoscope is a generally narrow and elongated imaging device suitable for insertion into a cavity or through a generally small opening and for capturing an image of a viewing area within the cavity and / or the area behind the small opening by means of at least one camera or at least one image sensor.Imaging devices according to the present invention can further comprise all types of cameras known to those skilled in the art that can image material, processes, or parts thereof, particularly medical material. Examples of this include, in particular, room cameras positioned in or near an operating room.

[0014] Medical image data within the meaning of the present invention are any image data that are in a medical context. Accordingly, medical image data within the meaning of the present invention can be representations. The medical image data includes medically relevant objects, such as endoscopic images of organs or other structures, as well as depictions of scenes from a medical application or treatment, such as scenes recorded by a room camera in an operating room. The medical image data comprises at least one object that is appropriately represented or recognizable in the digital image. An object within the meaning of the present invention can be an item, a part of an item, or a structure.

[0015] The acquisition of a digital image can take place directly before the application of the subsequent process steps or separately, so that the remaining process steps can be applied to previously acquired and now stored digital images. The digital images can be of any size and format; an adaptation of the image size and / or format to the requirements of the object recognition model to be used can, optionally and as required, be carried out within the framework of the method according to the invention.

[0016] Image information density, as defined in the present invention, describes the amount of information contained or discernible in a given image. Image information density can be influenced by various factors, including the resolution and / or size of the image, the scaling of the image and / or the individual pixels of the image, the underlying color space, the number of details, the complexity of the scene, and the presence of patterns or textures. A high image information density indicates that the image contains a substantial amount of discernible data or detail, while a low image information density indicates a simpler or less detailed image.Accordingly, in accordance with the present invention, for example during image preprocessing to create a first version of the image, the image information density can be reduced by reducing the image size, in particular by reducing the image resolution. In accordance with the present invention, the preprocessing can be applied to an entire image or to sections of an image. Thus, when creating a first version of an image, the image information density is reduced, resulting in a smaller version of the image. This results in a version that can be processed faster with the same computing resources. In particular, object recognition processes can be performed more quickly on the smaller version of the image, i.e., the version with lower image information density.

[0017] An image recognition model according to the present invention is a specific implementation of an algorithm trained on annotated data to perform a specific function, in this case, the recognition of specific objects. Depending on the training data used, an image recognition model can recognize specific objects (binary) or groups of objects (multiclass). For example, an image recognition model can recognize specific organs, tissue types, devices, types of people, or combinations thereof, such as a patient in a hospital bed. A first object recognition model according to the present invention can, in particular, be trained to recognize and classify objects of interest, such as polyps. Based on this recognition, we obtain coordinates of regions of interest (ROIs) that contain the recognized objects and their class predictions.If the first object recognition model can detect multiple objects, its class prediction can form the basis for selecting a suitable second object recognition model. A second object recognition model, as defined in the present invention, can in particular be a binary or multi-class classifier trained to classify the objects detected by the first object recognition model more accurately by analyzing the sub-images extracted according to the specified regions of interest (ROIs). In particular, a second object recognition model can be trained on images containing only one object. These images can, for example, be extractions from an annotated dataset used to train the first object recognition model.

[0018] In one embodiment of the method according to the invention, the confidence threshold of the first object recognition model can be lowered to increase the recall rate of the object recognition model and thus detect as many objects as possible. Since a reduced confidence threshold can lead to a larger number of false positive detections, These are filtered by the subsequent second object recognition model or classifier. Therefore, to improve classification, the second object recognition model can also be trained to classify the absence of objects.

[0019] A region of interest (ROI) within the meaning of the present invention refers to a specific, well-defined area or part of an image that is selected for closer examination or analysis. In particular, within the meaning of the present invention, an ROI defines a specific area of ​​a scene or image on which a second phase of object recognition focuses or is limited. The ROI is determined, in particular, with respect to its position and size within the image, for example, via corresponding pixel coordinate values, taking the resolution into account. In preferred embodiments of the present invention, the ROI can be dimensioned such that, when a partial image corresponding to the ROI is extracted from the original image, it at least partially fulfills the requirements of an image recognition model with respect to the dimensions and / or format of the image.

[0020] In preferred embodiments of the method according to the invention, the method can be applied in more than two stages. After a first iteration of recognition and classification, the resulting partial images (ROI extraction) can be used to obtain more detailed representations or structural representations on an image for a more accurate classification of the recognized object.

[0021] In embodiments of the method according to the invention, the image information density can depend on resolution, compression rate, and / or color depth.

[0022] In embodiments of the method according to the invention, the identification can include the classification of the object.

[0023] In embodiments of the method according to the invention, the first digital image can be a single image or an image can be a sequence of digital images, in particular a video sequence.

[0024] In embodiments of the method according to the invention, the first digital image can be a digital image acquired by a digital imaging device. In particular, the digital imaging device can be an endoscope camera.

[0025] The present invention further relates to a device for multi-stage detection and classification of objects in medical image data comprising at least one image processing unit, wherein the image processing is configured to perform the methods described above.

[0026] In embodiments of the device according to the invention, it may further comprise a digital imaging device, in particular an endoscope camera.

[0027] The present invention further relates to a computer program, wherein the computer program includes instructions which, when the program is executed by a computer, cause it to perform the methods described above.

[0028] The present invention further relates to a computer-readable storage medium, wherein instructions are stored on the computer-readable storage medium which, when executed by a computer, cause it to execute the methods described above. Brief description of the images

[0029] The accompanying illustrations depict exemplary embodiments and applications of the present invention.

[0030] Figure 1 shows a schematic representation of the process according to the invention.

[0031] Figure 2 shows a schematic representation of the method according to the invention on an endoscopic image.

[0032] Figure 3 shows a schematic representation of the inventive method on an endoscopic image using two endoscopic devices.

[0033] Figure 4 shows an application of the inventive method to an image of an operating room taken with a room camera. Detailed description

[0034] The present invention will now be described in more detail with reference to the accompanying drawings. However, the invention can be embodied in many different forms and should not be interpreted as limited to the embodiments presented here. It should be noted that the illustrations depict general features of the methods used in the respective embodiments. These illustrations may not, however, accurately represent the exact structure or feature of a given embodiment. Furthermore, identical reference numerals in the illustrations denote corresponding parts across the different views or embodiments.

[0035] Figure 1 shows a schematic representation of the method according to the invention. According to the invention, in a first step 2, at least one digital image of an image field or viewing area is acquired using an imaging device, in particular a medical imaging device such as an endoscope system or a room camera arranged in an operating room. The digital image can represent at least one object. Since the digital image can have any format and / or any size, the digital image can be adapted to the requirements of the object recognition model to be used in an optional step 4, if necessary. In preferred embodiments, a reduced version of the digital image is generated, i.e., a version with reduced image information density, in order to accelerate processing by the object recognition model.In particular, the miniaturized version has a smaller size, especially in terms of its resolution, which can significantly accelerate the analysis by an object recognition model.

[0036] In a subsequent step 6, a first object recognition model is applied to the digital image, which may have been modified. If the digital image depicts or contains at least one object, this first object recognition model can detect at least one object. If multiple objects are depicted or contained in the digital image, multiple objects can be detected accordingly. The first object recognition model was trained to detect and, if necessary, classify objects of interest (for example, polyps or a hospital bed).

[0037] Subsequently, in a later step 8 of the method according to the invention, at least one region of interest (ROI) is determined for the at least one detected object. The ROI is determined such that the detected object is enclosed within the ROI or contained within the respective ROI. Optionally, the at least one detected object can also be assigned to a corresponding class, i.e., it can be classified. Accordingly, in this step, coordinates of ROIs containing the detected objects and, if applicable, their class predictions are obtained.

[0038] In a subsequent step 10, an area corresponding to at least one ROI is extracted from the original, unreduced digital image, thus generating at least one partial image of the digital image with the original image information density. This partial image can then be further processed as an independent image, containing the previously recognized object. However, this object is represented or contained at a higher image information density, for example, with regard to resolution.

[0039] If the at least one generated partial image does not meet the requirements of the second object recognition model to be applied with regard to image information density, in particular with regard to image size and, if applicable, image format, the at least one generated partial image can be in in an optional subsequent step 12, adapted to the requirements of the second object recognition model to be applied.

[0040] In a subsequent step 14, the second object recognition model to be applied, in particular a binary or multi-class classifier, is applied to the at least one sub-image. This second object recognition model is trained to classify the detected objects more precisely by analyzing the sub-images corresponding to the ROIs. If the first object recognition model was trained on a plurality of objects, a corresponding, specific object recognition model can be selected based on the first classification prediction. If several ROIs were determined and several sub-images were generated in the previous process steps, the second object recognition model to be applied is applied separately to each of these sub-images. In this process, the at least one object is assigned to a class.

[0041] In a subsequent step 16, a final decision is made regarding the class of the at least one object. If an initial classification was already performed during the application of the first object recognition model in step 8, this final decision regarding the class of the at least one object can be made by summarizing and / or voting on the two classification predictions in steps 8 and 14.

[0042] Figure 2 schematically illustrates the application of the inventive method to an endoscopic image. Figure 2a) shows a digital image of tissue structures captured with an endoscope camera; the image has a comparatively high image information density. In the example shown, it is assumed that the first object recognition model requires square images with a certain low resolution as input material. Accordingly, Figure 2b) shows a modification of the digital image according to Figure 2a), in which the digital image is converted into a square format by adding information-free, black image areas; Figure 2c) shows a further modification of the digital image in which the image size—and thus also the Image information density is reduced. The order of the modifications is irrelevant; what is essential is that the digital image is modified so that it can be accepted by the first object recognition model. If the first object recognition model is flexible with regard to image size and / or image formats, only one of the two modifications can be applied. In particular, the digital image can also be reduced in size to increase the processing speed of the first object recognition model, even if the first object recognition model could also accept or process the image in its original or a different size. Figure 2d) shows the modified image according to Figure 2c) after application of the first object recognition model; accordingly, an object was detected in the image and labeled with a corresponding ROI.Figure 2e) shows the original, unmodified digital image in its original size and image information density, into which the previously determined ROI, based on the modified image, was transferred at the corresponding position and size. Figure 2f) shows a partial image of the original digital image, based on the extraction of the content determined by the ROI. Figure 2g) shows the partial result of applying the second object recognition model to the partial image in which the object is recognized; however, the classification by the second object recognition model is not shown.

[0043] Figure 3 schematically illustrates the application of the inventive method to an endoscopic image. Figure 2a) shows a digital image of tissue structures and two endoscopic tools; this image was captured with an endoscope camera and has a comparatively high image information density. In this example, it is also assumed that the first object recognition model requires square images with a certain low resolution as input material. Accordingly, Figure 3b) shows a modification of the digital image according to Figure 3a), in which the image size—and thus also the image information density—has been reduced by lowering the resolution. Figure 3c) shows a further modification of the digital image in which the digital image is converted into a square format by adding information-free, black image areas. The image was converted. The order of the modifications is irrelevant; what is essential is that the digital image is modified so that it can be processed by the first object recognition model. If the first object recognition model is flexible with regard to image size and / or image formats, only one of the two modifications can be applied. In particular, the digital image can also be reduced in size—independently of the specific requirements of the first object recognition model—to increase the processing speed of the first object recognition model. Figure 2d) shows the modified image according to Figure 2c) after application of the first object recognition model; accordingly, two objects were detected in the image and labeled with corresponding ROIs 31, 32.Figure 2e) shows the original, unmodified digital image in its original size and image information density, into which the previously determined ROIs 31 and 32, based on the modified image, were transferred at the corresponding position and size as ROIs 33 and 34. It should be noted that the extraction can also be performed directly based on the information regarding the size and position of the ROIs, without transferring this information to the image. Figure 2f) shows two partial images of the original digital image, based on the extraction of the content determined by ROIs 31 and 32, and 33 and 34, respectively. It should be noted that the extraction can also be performed directly based on the information regarding the size and position of the ROIs, without transferring this information to the image.Figure 2g shows the partial result of applying the second object recognition model to the two sub-images, in which the objects were detected. The classification by the second object recognition model is not shown; however, due to the higher image information density of the sub-images, a more specific classification is possible. For example, in the first classification step, the first object recognition model can detect tools and thus distinguish them from the surrounding tissue. In the second classification step, the second object recognition model can then perform a more refined classification and also determine the type of tool, in this case, endoscopic forceps.

[0044] Figure 4 shows the application of the inventive method to an image captured by a room camera positioned in an operating room. Figure 4a) shows the initial image of a typical scene in an operating room, depicting various medical devices, an operating light, three hospital staff members, the operating table, and a patient in a hospital bed. In this example, the first object recognition model is trained to detect patients in hospital beds. Figure 4b) shows the partial image generated after applying the first object recognition model based on the region of interest (ROI) determined during the process. Based on this partial image, the second object recognition model also detected a patient in a hospital bed, allowing for a corresponding classification with a high degree of probability.

[0045] The scope of this disclosure includes all changes, substitutions, variations, alterations, and modifications to the embodiments described or illustrated herein that would be obvious to a person skilled in the art. The scope of this disclosure is not limited to the embodiments described or illustrated herein. Although this disclosure comprehensively describes and illustrates certain embodiments herein as specific components, elements, features, functions, operations, or steps, any of these embodiments may also include any combinations or permutations of any components, elements, features, functions, operations, or steps that would be obvious to a person skilled in the art.A reference in the appended claims to a method, device, or component of a device or system being adapted, set up, capable, configured, enabled, operational, or ready for operation to perform a specific function furthermore includes that device, system, or component, regardless of whether it or that specific function is activated, switched on, or enabled, as long as that device, system, or component is adapted, set up, capable, configured, enabled, operational, or ready for operation. Even if this disclosure contains specific... While embodiments may be described or illustrated as advantageous, certain embodiments may provide none, some, or all of these advantages.

Claims

Claims 1. Computer-implemented multi-stage method for the detection and classification of objects in medical image data, including: Capturing a first digital image of an image field with an initial image information density; Creating a first version of the first digital image, wherein at least one section has a second image information density, where the second image information density is lower than the first image information density; Applying a first object recognition model aimed at recognizing at least one object to the first version of the image; Identify at least one object in the first version of the image; Determine a region of interest (ROI) containing at least one object; Transferring the ROI to a corresponding area in the first digital image; Extract a first sub-image corresponding to the ROI from the first image with the first image information density; Applying a second object recognition model, aimed at recognizing at least one object, to the first sub-image; and Identify at least one object in the first partial image.

2. Method according to claim 1, wherein the image information density depends on resolution, compression rate, and / or color depth.

3. Method according to claim 1 or 2, wherein the identification in each case includes the classification of the object.

4. The method of claim 3, wherein the method further comprises making a final decision on the class of the object by summarizing and / or voting on a first result of the Classification by the first object recognition model and a second result of the classification by the second object recognition model are included.

5. Method according to one of the preceding claims, wherein the first digital image is a single image or an image is a sequence of digital images, in particular a video sequence.

6. Method according to any of the preceding claims, wherein the first digital image is a digital image taken by a digital imaging device.

7. The method of claim 5, wherein the digital imaging device is an endoscope camera.

8. The method of claim 5, wherein the digital imaging device is a room camera.

9. System for multi-stage detection and classification of objects in medical image data comprising at least one image processing unit, wherein the image processing is configured to execute the method according to any one of claims 1 to 8.

10. System according to claim 7, further comprising a digital imaging device, in particular an endoscope camera and / or a room camera.

11. Computer program comprising instructions which, when the program is executed by a computer, cause it to execute the methods according to any one of claims 1 to 8.

12. Computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to execute the methods according to any one of claims 1 to 8.

Images