Method and systems for image optimisation for 3D endoscopes

Abstract

The invention relates to a method and to a system for image analysis of image data captured by a 3D endoscope. The method comprises the following steps: capturing first image data and second image data by means of at least one first image sensor or by means of a first and a second image sensor of the 3D endoscope; wherein the captured first and second image data each comprise a two-dimensional image of a working region from different perspectives; receiving the image data by a control unit; wherein the control unit is designed to analyse the first and second image data in order to determine a quality value in each case; operating the 3D endoscope depending on a determined first and second quality value in a first playback mode or a second playback mode; wherein the control unit is designed, in the first playback mode, to output three-dimensional image data relating to the working region on the basis of the combination of the two two-dimensional images; and wherein the control unit is designed, in the second playback mode, to output a two-dimensional image of the working region on the basis of the first image data or the second image data.

Description

[0001] Methods and systems for image optimization for 3D endoscopes

[0002] The invention relates to methods and systems for image analysis and image optimization for a 3D endoscope, preferably comprising a stereoscope with two image sensors. Furthermore, a computer program product and a trained self-learning model are provided to carry out the steps of the method.

[0003] Technological background

[0004] Endoscopes are well-known medical devices for examining cavities in a body or technical cavities. A frequently used type of endoscope has an optical system at the distal end, i.e. the end facing the body, and is designed to capture images and transmit these to the endoscope operator. In this way, endoscopes can be inserted into the cavities to be reached in patients to be examined through natural openings such as the mouth or through an access created via a minimally invasive incision, and an image of at least one section of the cavity can be transmitted to the specialist personnel. Endoscopes can also be used for inspecting and navigating technical cavities in pipelines or shafts.

[0005] To compensate for the lack of depth information when using monocular endoscopes, various approaches are being pursued. The ever-increasing availability of powerful computers enables real-time monitoring of a selected work area within a body cavity for medical applications using three-dimensional image data, which can be provided by 3D endoscopes with, for example, two CCD sensors at the distal end. This enables depth perception of the work areas and allows the operator of the 3D endoscope to better identify structures, enabling more precise navigation, inspection, or surgery.

[0006] Well-known systems for capturing 3D images are so-called 3D endoscopes, such as stereoscopes. Methods for obtaining 3D image data can be carried out with the aid of at least one image sensor or two image sensors installed at the distal end of the endoscope. The 3D effect is usually created in a two-dimensional display device using the principle of stereoscopy. Two image sensors of a 3D endoscope simulate the binocular vision of a person on a display device, provided the geometric dimensions, such as the distance between the image sensors, are known. This allows 3D images to be generated based on two two-dimensional images from different perspectives or from partial images that are horizontally offset from one another, creating a visual impression with depth information similar to natural vision.

[0007] In principle, a distinction is made between the reproduction methods of 3D images with display devices: those that require no aids to perceive spatial depth (display or microdisplay(s) in glasses), and those methods that require aids such as 3D glasses in addition to the display device. There are several types of 3D glasses as aids. 3D glasses with color-coded image separation or polarization processes, for example, are based on passive image separation. It should be noted that general knowledge of stereoscopy and reproduction methods can be found in specialist literature such as "Stereo-3D: Fundamentals, Technology and Image Design" by Holger Tauer, 2010, Berlin Schiele & Schoen ISBN 978-3-7949-0791-5 and will not be explained further here.

[0008] The following potential disadvantages exist when using 3D endoscopes in cavities of human or animal bodies: if the user is working in situ, i.e. in a body cavity, a single image sensor or cover glass of a stereoscope can become contaminated, for example, by particles or at least partially obscured by steam or smoke. Since the field of view of a single image sensor of a 3D endoscope is relatively small, even the smallest deposits of grease, which can be caused by unintentional tissue contact, or a few tissue particles, which can arise, for example, during HF surgery, can severely impair the user's 3D impression and thus their vision. If an entire individual image sensor is obscured, depth perception is no longer possible. With conventional systems, cleaning of the distal end of the endoscope is necessary in this situation.However, this requires regular removal of the endoscope from the body cavity and extracorporeal cleaning. Therefore, reducing working time by avoiding cleaning procedures is a goal.

[0009] Extracorporeal cleaning of the endoscope at its distal end is particularly disadvantageous if complications arise that require the user to interrupt their diagnostic procedure or surgery, for example. Furthermore, extracorporeal cleaning also requires a certain amount of time, which must be taken into account in the overall duration of an operation. If, for example, complications arise from a perforation of an artery, the bleeding must be stopped as quickly as possible and optimal visibility for the endoscope user must be restored quickly. If hemostasis cannot be achieved quickly, other surgical methods that involve open surgery may be necessary. This should be avoided if possible to keep the intervention as minimal as possible for the patient.

[0010] In particular, in binocular 3D endoscopes, in contrast to regular monocular endoscopes, it is possible for image degradation, e.g. due to contamination, interfering objects, smoke, damage, or other disruptive factors, to affect the two image sensors unevenly across the two image sensors. This can mean that only one lens or certain areas of one or both cover glasses are contaminated. Since minor contamination of partial contamination can be tolerated by a user and a 3D display can be maintained without significant impairment, it is a task to obtain information on the degree of contamination in order to avoid unnecessary cleaning, for example. It can be helpful to determine whether an image sensor is completely or only partially contaminated and whether the degree of contamination differs between the respective image sensors.

[0011] When the 3D image is presented to the user using a 3D-capable display device or 3D glasses, asymmetry caused by interference with only one image sensor can lead to discomfort for the viewer. Possible discomforts include faster eye fatigue, headaches, nausea, or dizziness. Even if the user of the 3D endoscope may not immediately notice such discomfort during an inspection or surgery in the case of unilateral sensor deterioration, the surgeon's attention and concentration can suffer. Therefore, an object of the present invention is to provide a method for ensuring optimal viewing conditions and quality during a surgery.

[0012] Description of the invention

[0013] The invention aims to overcome at least one of the aforementioned disadvantages or to solve the aforementioned problems more effectively than with conventional 3D endoscopes. This approach aims to ensure a clear and distinct visualization of the object field to be observed, while the user's workflow is not interrupted or is interrupted only briefly.

[0014] These objects are achieved with a method for image optimization, a control system, a computer program product, and a trained, preferably self-learning model according to the features of the independent and subordinate claims. Preferred embodiments of the invention are defined in the dependent subclaims. The objective of performing an in-situ examination or operation as uninterrupted as possible can be supported by intracorporeal cleaning modules in a preferred embodiment.

[0015] According to a first aspect of the invention, a method for image analysis of image data acquired by means of a 3D endoscope, in particular for medical applications, is provided, which method comprises the following steps:

[0016] Acquiring first image data and second image data by means of at least one first image sensor or by means of a first and second image sensor of the 3D endoscope; wherein the acquired first and second image data each comprise a two-dimensional image of a work area from a different perspective;

[0017] Receiving the image data by a control unit; wherein the control unit is configured to analyze the first and second image data to determine a quality value in each case;

[0018] Operating the 3D endoscope in a first reproduction mode or a second reproduction mode depending on the determined first and second quality values; wherein the control unit is configured to output three-dimensional image data of the work area in the first reproduction mode based on the combination of the two two-dimensional images; and wherein the control unit is configured to output a two-dimensional image of the work area in the second reproduction mode based on the first image data or the second image data.

[0019] The at least one image sensor or two image sensors of the 3D endoscope each form a stereo instrument for capturing a working area of ​​any non-visible cavity from different perspectives. The term at least one image sensor also encompasses the options of two, three, four or more image sensors. In other words, the mentioned examples of one image sensor or two image sensors are not to be understood as limiting. The stereo images generated by the 3D endoscope can be captured by a common image sensor if a right and left image are generated as stereoscopic half-images that are offset by a parallax. With a single image sensor, this means that the first image data (e.g. right image) and the second image data (left image) are recorded from different spatial positions or angles, whereby the image sensor can have two separate light-sensitive surfaces for this purpose.In this context, the term “stereo” refers to the two image channels offset from each other by a distance, also called the stereo base, which are required to create a 3D image.

[0020] Alternatively, a first image sensor can capture first image data, and a second image sensor can capture second image data in real time from different perspectives to also generate a three-dimensional (3D) image. Image sensors typically include CCD (charge-coupled device) sensors with two-dimensional CCD arrays, CMOS sensors, or similar.

[0021] For example, if the user is working with a 3D endoscope with two image sensors, one of the image sensors can be assigned to the right eye (first image channel with first image data) and the other of the two image sensors to the user's left eye (second image channel with second image data). If a cover glass of the two image sensors becomes dirty or otherwise significantly impaired, a control unit uses analysis to determine a quality value that differs from the quality value of an undamaged image sensor. Based on the determined loss of quality, a two-dimensional image based on the image data from the sensor that does not exhibit any change in the quality value can be output manually or automatically in a second playback mode.In other words, the user can manually or automatically switch to the two-dimensional second playback mode based on the image data of the unaffected image sensor, depending on a quality value automatically determined by the control unit.

[0022] If this switching occurs automatically, the image sensor can be dynamically switched to one that is not dirty or otherwise compromised. Once the user is presented with two-dimensional images, no discomfort such as headaches caused by inadequate 3D representations is to be expected. A particular advantage of automatic switching is that the 3D endoscope user's work is not interrupted.

[0023] Alternatively, the control unit can also issue a recommendation based on the determined quality value to switch to the second playback mode, so that the user can manually switch to the second playback mode or to a 2D view. Any necessary extracorporeal cleaning can be postponed if the remaining image sensor is still functional. Various method steps of the method according to the invention, such as the analysis or the generation or calculation of the 3D images, can be computer-implemented and based on suitable image processing algorithms and / or software programs. The execution of the method steps can be carried out at various locations, such as the 3D endoscope itself, i.e., in a computer processor, a cloud, and / or on decentralized servers.The reproduction of three-dimensional image data or two-dimensional image data can be carried out on a suitable reproduction or display device such as a display, monitor or suitable glasses that have a display for each eye.

[0024] When using 3D glasses with a microdisplay per eye, the poorer image data can be switched off when the control unit switches from the first playback mode (outputting 3D images) to the second playback mode (outputting two-dimensional images), so that one eye would be black. According to an alternative embodiment, the image data with the better quality value can be output to both eyes of the 3D glasses when switching to the second playback mode. The latter has the advantage that the perceived image brightness is not impaired.

[0025] According to a preferred embodiment, the analysis of the first and second image data for determining the quality value comprises an image processing algorithm and / or a self-learning module for detecting at least one of the following events: contamination, at least one interfering object; a loss of image quality, condensation, damage, or smoke.

[0026] With the help of computer-implemented algorithms, even events unnoticed by the user, as mentioned above, can be automatically identified, thus preventing an inadvertent continuation in the first playback mode. Thus, as soon as an event disrupting a partial image is detected using the quality value, the image data not affected by the event from one of two image sensors, or in the case of a single image sensor, the image data from a first light-sensitive area of ​​two areas, can be displayed to the user in two dimensions. In this way, the second playback mode can be activated manually or automatically to avoid irritation, pain, or fatigue for the user.

[0027] The analysis of the first and second image data to determine the quality value using an image processing algorithm can be based on a variety of known image processing algorithms. Furthermore, self-learning modules can be used, preferably in conjunction with training data, whereby the self-learning module can also use known image processing algorithms. For example, a self-learning module can use Gaussian methods to determine pollution or other disturbance events. See well-known literature for an example of an event classification algorithm: C.E. Rasmussen, K.I. Williams, "Gaussian Processes for Machine Learning," in MIT Press (2006).

[0028] In order to recognize the event to be detected, the image data sets of the first and second image data can be analyzed, among other things, for deviations from the normal state or from the other partial image. In this context, a wide variety of image processing algorithms can be used for anomaly detection. Classification problems can be addressed, for example, using what is known as nearest neighbor analysis (see, for example, T. Cover and P. Hart, Nearest neighbor pattern Classification, in IEEE Trans. Information Theory (1967; 21-27). This method classifies images according to their similarity to other images. With the help of the self-learning module, which is based on machine learning, data patterns can be recognized without the need for an exact match with patterns stored in a memory unit or images provided as training data.

[0029] According to a preferred embodiment of the method, the control unit determines that if only one of the two quality values ​​is representative of at least one of the events contamination, disturbing object, loss of image quality, condensation, damage or smoke, a two-dimensional image based on the image data which is assignable to the other quality value is automatically or manually output in the second reproduction mode.

[0030] In other words, if two image sensors are provided, the control unit determines that a quality value that is exclusively attributable to one image sensor in the image data is representative of at least one of the events contamination, interference object, image quality loss, condensation, damage or smoke, the 3D endoscope is operated automatically or manually in the second playback mode based on the image data of the other of the two image sensors.

[0031] According to a preferred embodiment of the method, the analysis of the first and second image data comprises determining an average image contrast in order to determine the quality value based on a comparison of the respective average image contrast by means of a visual threshold condition.

[0032] In this way, the respective quality value can be determined using known image processing algorithms. The average image contrast can be determined, for example, by analyzing whether a difference occurs (e.g., using the aforementioned nearest neighbor analysis). If, for example, smoke appears in front of one of two image sensors, the image contrast is lower compared to the normal image from the image sensor without smoke. According to a preferred embodiment of the method, the average image contrast is determined using at least one of the following analysis methods: histogram method, application of a Gaussian filter, pattern recognition, corner detection, edge detection, and object detection.

[0033] According to a preferred embodiment of the method, the first and second image data each comprise a plurality of colors occurring in the workspace, wherein the analysis of the first and second image data each comprises the detection of a mean color tone in order to determine the quality value based on a comparison of the most frequently occurring color from the plurality of colors with the respective mean color tone.

[0034] In this way, an image processing algorithm can determine the similarity between the first image data and the second image data using the color or hue parameter. If a deviation or difference in the mean hue is greater than a predefined threshold, the second rendering mode can be activated for a two-dimensional image with the image data whose hue does not reach the threshold and thus represents a mean hue typical for the working area.

[0035] According to a preferred embodiment of the method, the first and second image data each comprise a plurality of colors occurring in the work area, wherein, if the most frequently occurring color among the plurality of colors is white or gray, smoke is detected; or if the most frequently occurring color among the plurality of colors is red, blood contamination is detected.

[0036] In this way, with the help of color analysis of the image data, an event such as smoke or a blood spatter that is limited to an image sensor or a light-sensitive area of ​​an individual sensor can be determined, and the affected image sensor or sensor area can be switched off as part of the second playback mode. With regard to the colors occurring, it should be noted in principle that the at least one image sensor or the plurality of image sensors for capturing the image data is / are sensitive to light visible to the human eye, according to a common application example. This includes, for example, a wavelength range between 380 nm and 780 nm (nanometers), although this should not be understood as limiting. The color red is known to be in the high wavelength range between 620 nm and 780 nm.

[0037] In particular, the above-mentioned embodiment does not preclude the possibility that other embodiments can also acquire image data for observation in the infrared or near-infrared range, or in the ultraviolet range. This is helpful, for example, when the 3D endoscope is used for so-called photodynamic methods such as PDD (photodynamic diagnostics) or for fluorescence observation.

[0038] According to a preferred embodiment, the method further comprises the step of automatically issuing a warning message when a control instruction for activating another playback mode is automatically issued by the control unit.

[0039] This informs the user that the playback mode has been changed. This allows the user to prepare for the fact that depth information will no longer be available in the second playback mode. On the other hand, a warning message can be displayed to indicate a return to the three-dimensional image once the interference has been resolved. This allows the next steps of the examination to be adapted to the respective playback mode, increasing patient safety.

[0040] According to a preferred embodiment of the method, in a method step of the pre-analysis, the respective image data is subdivided into image sections, wherein the quality value can be determined for each of the image sections, wherein the control unit is designed to display exclusively the image sections with a permissible percentage of contamination or a permissible loss of quality, and / or to display at least the image sections that can be assigned to an image center of the respective image sensor.

[0041] By pre-analyzing at least a subset of the image data, such as an image section, a local loss of quality can be identified with a corresponding quality value. A storage unit can provide data to determine a quality value for a given subset or image section, and thus permissible or impermissible contamination, by comparing historical image data from the database.

[0042] Based on the preliminary analysis of preferred regions or image sections, the display can be targeted to these regions or image sections. Depending on the quality value, the image center can be displayed preferentially either two-dimensionally or three-dimensionally, as this is usually where the object to be examined is located and the user has a particular interest in ensuring optimal image reproduction. If a 3D endoscope with two image sensors is provided, two-dimensional images of the uncontaminated image sensor can be displayed, depending on the contamination of only one image sensor, as determined by the quality value.

[0043] According to a preferred embodiment of the method, on the basis of the determined quality value, control instructions are issued manually or automatically by the control unit for activating a cleaning module by means of fluid and / or gas, wherein the cleaning module is an activatable unit of the 3D endoscope for preferably intracorporeal cleaning or is provided as an extracorporeal unit.

[0044] In this way, depending on the contamination or other event, one or two image sensors or the associated light-sensitive surfaces can be cleaned intracorporeally or extracorporeally, whereby the cleaning process can be initiated manually or controlled automatically. The cleaning module is designed to clean at least one cover glass of an image sensor or parts thereof with at least one fluid. Suitable fluids for intracorporeal cleaning are preferably water or sodium chloride or another physiological liquid. A gaseous fluid is also selected so that it is physiologically harmless and biocompatible and can be carbon dioxide, for example. The cleaning module can be used to guide a defined flushing quantity or volume of gas or liquid at a predeterminable flushing pressure through the shaft close to the respective image sensor surfaces, preferably intracorporeally.

[0045] For example, if critical contamination has been detected based on the quality value, the decision to clean can be made automatically, and the cleaning module can be informed to open a fluid valve for a predetermined volume of gas or liquid (e.g., carbon dioxide or sodium chloride). As long as cleaning is limited to one image sensor or sensor area, the other image sensor or sensor area can continue to capture images for the second playback mode. After activation, the control unit can automatically issue the control command to abort the cleaning process and, if cleaning is successful, switch back to the first playback mode.

[0046] If the cleaning module has at least two fluid channels and associated nozzles, it is suitable for specifically cleaning one of two image sensors or two sensor surfaces with fluid and / or gas. With one fluid channel per image sensor, switching between fluid and gas is necessary, as only one channel is available per image sensor. If both cleaning methods (liquid cleaning and gas drying) are to be controlled quickly one after the other, two fluid channels per image sensor are advantageous. In all configurations, the control unit can simultaneously control cleaning (liquid and / or gas) and a switch to the second playback mode. By switching to the sensor surface not to be cleaned, the operator is prevented from noticing the cleaning as a disruption and can continue working with the two-dimensional image.

[0047] According to a preferred embodiment of the method, the control unit has a cleaning control algorithm that checks the cleaning effectiveness extracorporeally and / or intracorporeally with the activatable unit of the 3D endoscope for one or two image sensors, so that at least one two-dimensional image or one three-dimensional image with sufficiently good image quality can be reproduced or a repetition of the cleaning is activated.

[0048] According to a preferred embodiment, the method comprising a cleaning module is designed to adapt the type of cleaning by changing the cleaning parameters depending on the degree of contamination, wherein one or more cleaning parameters are selected from a group comprising: type of fluid, fluid volume, fluid volumes, fluid velocity, pressure, pulse duration, pulse number, pulse-pause ratio, and / or total cleaning duration. If the cleaning module is intracorporeal, it can be switched back to a first three-dimensional display mode depending on the determined improvement in the degree of contamination of both image sensors or both sensor surfaces of an image sensor.

[0049] According to a preferred embodiment of the method, the self-learning module determines at least one image structure from the image data of the respective image sensor or a sensor partial area of ​​an individual sensor by means of an artificial intelligence-based analysis, preferably using a neural network, in order to classify an event by means of the at least one determined image structure on the basis of a comparison with image structures of a reference database stored in a storage unit in the event of a detected deviation.

[0050] A storage unit, which can be provided locally in the control unit or decentrally, e.g., via servers or networks, has at least one database for this purpose. This database includes at least one reference database with image data comprising image structures that are indicative of an event such as contamination or damage.

[0051] According to a preferred embodiment of the method, the self-learning module comprises a model that is trained with training data of image structures stored in the storage unit, wherein the trained model of the self-learning module is trained to classify the image data into the following event-dependent database classes: non-soiled images, soiled images that can be cleaned intracorporeally; soiled images that can be cleaned by a cleaning module; fogged images that can be dried by a module using gas; smoky images, wherein the smoke can be extracted by an extraction module;

[0052] Images with quality loss that can be optimized by other modules;

[0053] Images that contain one or more obscuring objects; and / or images that show at least one form of damage.

[0054] With the help of machine-trained database classes, a trained model can reliably and reproducibly classify current first or second image data of an examined work area. The trained model can detect when events as mentioned above, such as the appearance of interfering objects, have occurred. Furthermore, the database classes can be used to automatically determine whether a recognized image structure under investigation has changed, for example, due to smoke, condensate, contamination, or other interfering factors. For this purpose, the 3D endoscope advantageously has a storage unit or is communicatively connected, preferably wirelessly, to a storage unit.

[0055] One advantage of a machine-trained model is that it is not dependent on subjective user judgment. Based on the detected change, manual or automatic control instructions regarding the playback mode and / or cleaning can be issued from a control unit to a unit for activating cleaning using a cleaning module. This ensures image optimization of the viewing conditions and visual quality during surgery.

[0056] For the aforementioned methods based on algorithms or machine learning, the control unit comprises computer elements such as one or more microprocessors or graphics processors. A graphics processor (GPU), for example, can display high-quality 2D or 3D images to a user in real time on a suitable display or playback device such as a monitor or via 3D glasses.

[0057] According to a preferred embodiment of the method, a 3D image is displayed in the first playback mode using a display device and / or 3D glasses, and a 2D image is displayed in the second playback mode.

[0058] In addition, combinations of a display device with special 3D glasses are conceivable as an aid. With passive color-coded image separation, a warning message, which is automatically displayed by the control unit when a different playback mode is selected, can inform the user to remove or put on polarized glasses.

[0059] According to a preferred embodiment of the method, the 3D endoscope is selected from the following group: a stereoscope comprising at least a first image sensor, preferably with two separate light-sensitive surfaces or designed as a facet lens; a stereoscope comprising at least a first image sensor and a second image sensor; an imaging scope comprising at least a first image sensor, a second image sensor, and a third image sensor; an imaging scope comprising at least a first image sensor, a second image sensor, a third image sensor, and a fourth image sensor; and a stereoscope designed as a fiber optic endoscope.

[0060] If the individual image sensor is designed as a facet lens, also called a fly's eye, a multitude of microlenses are provided. The large number of microlenses can capture closely spaced individual microimages with their respective image data, with each microimage depicting the overall image from a different perspective. The facet lenses can reproduce three-dimensional images that require no special viewing aids. If a large portion of the microlenses becomes contaminated, a two-dimensional image can be displayed by the remaining uncontaminated microlenses in the second playback mode without any loss of quality.

[0061] In an imaging scope with three or more image sensors, the control unit analyzes the image data sets from preferably three or more image sensors. Alternatively, an imaging scope with at least three image sensors can be designed by default so that two of the plurality of image sensors are activated at a time to generate stereo images like a stereoscope.

[0062] In an exemplary embodiment, which is intended to take the position of the 3D endoscope into account, three image sensors are arranged on the distal end section of the 3D endoscope such that the image sensors are evenly spaced on a circular line. The control unit is preferably designed such that the pair of three image sensors is activated whose connecting line between the image sensor centers is as parallel as possible and optimally parallel to a horizontal plane. The horizontal plane is a plane perpendicular to the gravity vector. In this way, two image sensors can be selected at a time depending on the position of the 3D endoscope so that the generated 3D image does not rotate as much as possible when the endoscope is rotated about its longitudinal axis with respect to a horizontal plane.For each of the two activated image sensors of an image endoscope with three image sensors, the method according to the invention can perform an analysis of the first and second image data by means of the control unit to determine two quality values. If an image sensor becomes dirty, the system switches to the second playback mode based on the image data of the unaffected image sensor. Furthermore, if the quality value of an image sensor is insufficient, the third image sensor can be activated and an analogous analysis of the image data received from the third image sensor can be performed. The first playback mode or the second playback mode is then selected depending on the determined quality values ​​of the third image sensor and the additional image sensor.

[0063] If more than three image sensors are provided, switching between two sensor pairs is possible. Alternatively, all image sensors can be activated simultaneously, similar to the microlenses in a facet lens. Depending on the evaluation of the respective image data sets and the determined quality values, the first playback mode can then be output based on the corresponding image data from at least one unaffected pair of adjacent image sensors.

[0064] The simultaneous use of three, four, or more image sensors has the advantage of largely avoiding potential ambiguities or inaccuracies that can occur when evaluating stereo image datasets from only two image datasets. The accuracy of the 3D image dataset can be increased by increasing the number of image sensors. Furthermore, with an even number of image sensors, a greater depth of field can be achieved by using multiple pairs of image sensors with different lens focal lengths. This can be controlled using appropriate software.

[0065] According to a further aspect of the invention, a control system for image analysis of image data acquired by means of a 3D endoscope, in particular for medical applications for carrying out a method according to one of the preceding claims, is provided, the system comprising: a 3D endoscope with at least one first image sensor, wherein the first image sensor is designed to acquire first image data and second image data; or a 3D endoscope with a first image sensor designed to acquire first image data and a second image sensor designed to acquire second image data; wherein the acquired first and second image data each comprise a two-dimensional image of a work area from different perspectives; a control unit in communication with the at least one image sensor for receiving the first and second image data;wherein the control unit is designed to analyze the first and second image data in order to determine a quality value in each case; wherein the control unit is designed to automatically or manually output control instructions to the 3D endoscope for a first or second reproduction mode based on a specific first and second quality value; wherein the control unit is designed to output three-dimensional image data of the work area based on the combination of the two two-dimensional images in the first reproduction mode; and wherein the control unit is designed to output a two-dimensional image of the work area based on the first image data or the second image data in the second reproduction mode.

[0066] Furthermore, a computer program product is provided which comprises instructions which, when executed by a computer, cause the computer to perform the steps of one of the methods described above.

[0067] Furthermore, a computer-readable medium is provided on which the said computer program is stored.

[0068] Furthermore, a trained model is provided, preferably self-learning and trained with training data comprising preferably stored image structures and / or a collection of features, and wherein the trained model is designed to perform the steps of at least one of the methods described above. Brief description of the figures

[0069] The invention, as well as further advantageous embodiments and developments thereof, are described and explained in more detail below with reference to the examples shown in the drawings. The drawings are for illustrative purposes only and are not to scale. Terms such as right or left or top or bottom are not to be understood as limiting. The features shown in the following description and the drawings can be used individually or in any combination according to the invention.

[0070] Fig. 1a is a schematic perspective view of a distal end of a 3D endoscope with a first and second image sensor for capturing a stereoscopic image pair;

[0071] Fig. 1b is a schematic perspective view of the 3D endoscope shown in Fig. 1a with a schematic representation of a contamination of an image sensor and the image captured with the uncontaminated image sensor;

[0072] Fig. 1c is a schematic perspective view of the 3D endoscope shown in Fig. 1a, with a schematic representation of further contamination of an image sensor and the image generated with the uncontaminated image sensor; Fig. 2a is a schematic perspective view of a distal end of a 3D endoscope with a single image sensor with two separate light-sensitive surfaces and a schematic representation of the two-dimensional image pair generated with this sensor;

[0073] Fig. 2b is a schematic perspective view of the 3D endoscope shown in Fig. 2a with one-sided contamination of the image sensor;

[0074] Fig. 2c is a schematic perspective view of the 3D endoscope shown in Fig. 2a with further contamination of the image sensor;

[0075] Fig. 3 schematically shows an embodiment of a 3D endoscope with two separate image sensors, four illumination devices, and other components such as a control unit and display device; Fig. 4 shows a flowchart of a method comprising image optimization or image adaptation from 3D to 2D or 2D to 3D and querying to active 3D glasses;

[0076] Fig. 5a shows a perspective view of another embodiment of a 3D endoscope with a single image sensor and a cleaning module comprising two fluid channels and a nozzle arrangement;

[0077] Fig. 5b and 5c show an exploded view of the embodiment according to Fig. 5a and a detailed view of the distal end of the two fluid channels;

[0078] Fig. 6 shows schematically an embodiment of a 3D endoscope with a control unit and units communicating therewith for carrying out a cleaning including a storage unit for storing cleaning parameters;

[0079] Fig. 7 shows schematically the components of a 3D endoscope with image acquisition device, control unit with storable routines or methods for image optimization including 3D to 2D adaptation (output in the second playback mode) and a cleaning module;

[0080] Fig. 8 shows a schematic flow diagram of a further embodiment of the method according to the invention for image optimization, comprising in particular a contamination detection for 3D / 2D adaptation and / or cleaning with a liquid fluid and optionally a gaseous fluid.

[0081] Detailed character description

[0082] Fig. 1a schematically shows a distal end of a 3D endoscope 100 with a shaft 150 and two image sensors (first image sensor 111 and second image sensor 112), wherein the recording axes are arranged horizontally shifted. So-called half images (11, 12) or flat images are recorded by each sensor, which form a stereoscopic image pair.

[0083] Both images show the same object (here, a smiley face to illustrate a possible object in the center of the image), but with a location offset laterally by the so-called stereo base (x, 125). The recording axes must be parallel, and the recording conditions or settings of the two image sensors 111, 112 must be identical (same focal length, focus, and filters used). The stereo image sensor system simultaneously records the two partial, half, or flat images with the two image sensors 111, 112, as long as no contamination or other interference significantly affects one of the two image sensors 111, 112.

[0084] Based on the two captured half-images 11, 12, which are offset from each other by the stereo base x (reference symbol 125), a 3D image can be generated using suitable image software. This can then be displayed using a suitable display device. Suitable devices include, for example, a car stereo monitor aligned to an observer, a monitor based on wavelength division multiplexing technology, or an LCD TV monitor with 3D glasses such as shutter glasses as an aid and / or active 3D glasses.

[0085] Fig. 1b shows the stereo system of Fig. 1a with two image sensors (111, 112), wherein the first image sensor 111 is affected by contamination 113. Both partial or half images are analyzed by an image processing algorithm or a self-learning module to determine a first or second quality value. In the example shown, the analysis of the image results in a quality value Q1, which indicates that the cover glass of the first image sensor 111 is approximately 100% covered by dirt 113. The analysis 124 of the captured image 12 of the second image sensor 112, on the other hand, results in a quality value Q2, which indicates that the cover glass of the second image sensor 112 is clean and thus corresponds to the initial values ​​of the image sensor during white balance. In this way, depending on the determined first and second quality values, switching to the clean image sensor 112 can be performed either automatically or manually after a corresponding indication.The two-dimensional field of the unaffected second image sensor 112 can be shown using a display device not shown here.

[0086] Fig. 1c shows the case of contamination 114 on the second image sensor 112. The

[0087] Image analysis 124 produces a quality value Q2, which indicates an event such as 100% contamination or a significant loss of image quality. Threshold values ​​can also be determined here, at which a switch to a 2D mode instead of a 3D mode occurs. If the image sensor 112 is 100% contamination, as shown in Fig. 1b, only image data can be acquired with the first image sensor 111. Based on the image analysis, which is representative of a contamination-free coverslip, a quality value Q1 is obtained. The user can now continue working with the two-dimensional image 11 of the object (smiley) in a second playback mode (2D mode) without any loss of quality.If the first image sensor 11 also becomes significantly contaminated at a later time and the 3D endoscope does not have a distal cleaning module for intracorporeal cleaning, the user must attempt to remedy the situation without a cleaning module or remove the endoscope and clean it extracorporeally. For this purpose, the 3D endoscope can issue a warning message such as "wipe the liver" or "clean ex situ."

[0088] Fig. 2a is a schematic perspective view of a distal end of a 3D endoscope 100 with a single image sensor 110 with two separate light-sensitive surfaces 110R and HOL. The dotted circle schematically shows an enlargement of the image sensor 110. The left image sensor surface HOL and the right image sensor surface 110R view the same conical object, shown two-dimensionally as a triangle.

[0089] In addition to the perspective view of the 3D endoscope, the two-dimensional half-images as seen by the two image sensor surfaces 110R, HOL from two different perspectives are shown schematically. As an example object, the cone shape is shown here, which is captured from the two perspectives of the recording axes offset by the stereo base x (see double arrow above partial surfaces 110R and 110L), which are each located centrally in the sensor partial surfaces 110R, HOL. The right half-image 12R is captured by the sensor surface 110R and the left half-image 11L, offset by the stereo base 110X, is captured by the surface HOL. A playback device (not shown here) is suitable for displaying both two-dimensional 2D images and 3D images based on the half-images 11L and 12R. The playback mode is selected depending on the quality values ​​determined on the basis of the image data.

[0090] Fig. 2b is a schematic perspective view of the 3D endoscope shown in Fig. 2a with one-sided contamination of the image sensor. The contamination 113R covers approximately 100% of the sensor surface 110R. The dashed rectangle in Fig. 2b shows that only the half-image 11L with the object (here, a cone as a triangle) can be captured using the surface HOL. Image analysis 124 using a conventional image processing algorithm or based on AI (artificial intelligence) with, preferably, a self-learning module, yields corresponding quality values ​​QR and QL for the respective sensor surfaces. Depending on the acquired quality values, the system can switch to the clean cover glass and display the two-dimensional image 11L.

[0091] Fig. 2c is a schematic perspective view of the 3D endoscope shown in Fig. 2a with another example of contamination of the image sensor 110. Here, not only is the image sensor surface HOL completely contaminated with dirt 114L, but also a few parts of the other image sensor surface 110R with further dirt 114R and linear contamination 114r. Thus, by means of the image analysis 124, a quality value QL is determined that represents 100% contamination for the surface HOL. For the other surface 110R, the quality value QR is determined based on the image data 12R and the hidden image portions 114R and 114r, which corresponds to partial contamination of less than 30%.

[0092] The determined associated quality values ​​QR and QL indicate that both sensor surfaces are contaminated to varying degrees, and QL still provides the user with a sufficient field of view. Based on the analysis results, the system automatically or manually switches to the cover glass of sensor 110R, which is less contaminated. The 2D playback mode is illustrated in Fig. 2c with the two-dimensional image containing the image data 12R. In principle, a classification into different classes of contamination can be made. In the case shown, a further distinction can be made in that the image center of surface 110R is not affected by the contamination, so that the conical object to be examined in the image center is still clearly visible and the user can continue working on the object.

[0093] Fig. 3 schematically shows another embodiment of a 3D endoscope 100 with two separate image sensors 111, 112 and, on either side thereof, illumination devices 154, 155 and 156, 157. The illumination devices 154, 155, 156, and 157 can be designed as optical fibers that are guided in the shaft 150 from the distal end to the proximal end of the 3D endoscope 100 for connection to a supply unit. They are designed to transmit light for the surgical area or site. The goal of the plurality of illumination devices is to provide the user with the most natural endoscopic image possible with high resolution for their intervention or inspection. The image sensors 111, 112 can have flat glass cover glasses with lenses behind them and are connected to other components such as a control unit and display device.

[0094] The control unit 120 comprises a processor 121 for image analysis 124 and dirt detection 125, and a memory unit 122. Using the aforementioned analysis methods, a quality value Q1 and Q2 can be determined for each image sensor 111, 112. Depending on these quality values, which can generally be described as a function of a quality value with f(Q), i.e., based on a specific first and second quality value (Q1, Q2), control instructions are automatically or manually output to the 3D endoscope 100 for a first or second playback mode (method step 103).

[0095] If the analysis of image data 11 and 12 reveals that both coverslips are contaminated, the degree of contamination can be determined using analysis 124 and the determination of the quality values ​​(Q1, Q2). If the contamination does not exceed a certain threshold or is present only in the peripheral area and not in the image center, the image can be switched to the less or only partially contaminated coverslip.

[0096] Instead of playback or display devices 430, active glasses can also be used, which can project 3D images onto the displays for each eye. These so-called active 3D glasses can then actively switch to a two-dimensional second playback mode or vice versa, depending on the determined quality value. This display option using 3D glasses is shown in the flowchart in Fig. 4.

[0097] Fig. 4 shows a flowchart of a method 400 comprising image optimization or image adaptation from 3D to 2D or 2D to 3D. As soon as, in method step 103, the playback mode is to be changed from the first to the second or vice versa based on the determined quality values, a warning message is issued to the user (see steps 402, 403).

[0098] Either in step 402 or in step 403, the user is warned that image optimization is possible. In step 402, the user is warned that switching from 3D to 2D is recommended if, for example, only one lens is dirty. Alternatively, in step 403, a warning can be issued before switching from 2D to 3D mode if, for example, one of the two dirty lenses has been successfully cleaned and both can be used again. In process steps 401 (2D OK?) or 411 (3D OK?), an optional query can be made as to whether image optimization is actually desired.

[0099] If the second playback mode, i.e., 2D, is desired, the image optimization software is activated automatically in step 412 via step 414 or after a positive query (YES arrow). In step 422, the control unit's output signal is automatically switched to 2D. If, however, 3D is desired (YES arrow after step 411), the corresponding software is activated in step 413. If no warning message or query and no option for manual adjustment is desired, the image optimization software can also be automatically activated directly from process step 103 via the dashed route (see steps 414 and 415).

[0100] The system then asks whether active 3D glasses are in use. If this is the case, the process continues with "YES" to step 432, where the active 3D glasses 431 are automatically adjusted to the 2D image.

[0101] Similarly, after activating the software in step 413 for adaptation to a 3D image, an output signal from the 3D endoscope can be automatically switched to 3D in step 423 (2D ~> 3D), and then an active pair of 3D glasses can be queried. If the query is positive, the 3D glasses can be adjusted in step 433. If no active 3D glasses are detected, the process continues without activating the 3D glasses. If image adjustment to another playback mode is denied at the beginning of the respective queries 401 and 402, no adjustment takes place (NO arrow).

[0102] Fig. 5a shows a perspective view of another embodiment of a 3D endoscope with a single image sensor and a cleaning module comprising a nozzle arrangement 140. The single image sensor 110 comprises two light-sensitive surfaces 110R, 110L. Illumination devices 151 and 152 are arranged laterally.

[0103] Fig. 5b and c show an exploded view of Fig. 5a and a detailed view of the distal end of two fluid channels 131, 132 that open into the nozzle arrangement 140. If liquid is conveyed in each fluid channel, cleaning can be performed specifically for each partial area. Alternatively, gas can also be provided in each channel for drying.

[0104] Fig. 6 shows schematically an embodiment of a 3D endoscope with a

[0105] Image capture device 115, which was shown in Figures 5a-c and schematically shows further components such as a control unit and units in communication therewith for carrying out a cleaning.

[0106] The captured first image data 11R and 12L are transmitted to the control unit 120 for further processing by at least one processor 121. The control unit 120 can be provided locally as a camera control unit 116 (CCU).

[0107] Furthermore, a storage unit 122 may be provided, which in this embodiment is arranged locally. Alternatively, the control unit 120 and the storage unit 122 may not be arranged locally at the distal end of the endoscope, but may be provided externally via a cable or wirelessly. The local or external storage unit 122 may store a plurality of first and second image data acquired by the image sensor 110.

[0108] The processor 121 can comprise one or more microprocessors and / or graphics processors and can be used for optics recognition 123 and for analysis 124 of the first image data 11R and second image data 12L. The optics recognition 123 can be used as an initialization routine to recognize the connected image sensor 110 and, depending on this, inform the user whether a 3D endoscope has been recognized. The system has a cleaning module 130 at the distal end of the 3D endoscope for intracorporeal cleaning and can be transmitted to the cleaning module 130 for cleaning control or activation with the cleaning parameters 128 stored in the storage unit 122. Cleaning parameters 128 such as volume V, pressure p, and time t of the fluid pulses can also be preselected depending on the recognized optics. The cleaning parameters 128 can then be applied specifically for each fluid channel and assignable partial area.A dashed triangle for each sensor surface 110R and HOL indicates a local cleaning option for each sensor surface, which can be achieved using the two fluid channels (see 131, 132) and suitable nozzles of the nozzle arrangement 140. The range of the respective cleaning nozzle can also be extended so that the illumination devices 151, 152 can also be cleaned. Fig. 7 schematically shows the components of a 3D endoscope system 700, which comprises an image capture device 115 and control unit 120 and is suitable for carrying out the inventive method. The first method step 101 comprises the acquisition (101) of first image data (11, 11L) and second image data (12, 12R) by means of at least one image sensor of an image capture device 115 of the 3D endoscope.Further method steps according to the invention are preferably computer-implemented and comprise automatable analysis steps for image optimization comprising a 3D to 2D adaptation and vice versa as well as optionally a cleaning with a cleaning module 130.

[0109] In the first case, the image capture device 115 may be an image sensor 110 having a single image area divided such that first and second image data (1 IL, 12R) can be captured with at least two separate partial areas (HOL, 110R). These image data (1 IL, 12R) are received (102) (120) via a control unit in method step 102.

[0110] In the second case, there may be at least two image sensors with spatially separated image areas, wherein each image sensor 111, 112 has at least one data output, the information of which (first and second image data 11, 12) is sent to the control unit 120 (see arrows to the processor 121 of the control unit 120).

[0111] Alternatively, an image capture device 115 configured as a facet lens is conceivable, with a plurality of microlenses being provided. The large number of microlenses can capture closely spaced individual microimages with respective image data. Furthermore, other 3D endoscope types can be connected to the system, comprising at least three image sensors or four or more image sensors.

[0112] In addition to the processor 121, the storage unit 120 includes a memory unit 122, which can store parameters and routines. The processor 121 can comprise one or more microprocessors or graphics processors and can be used for image analysis 124 and optics recognition 123. The optics recognition 123 can be used as an initialization routine to recognize the connected image capture device 115 and, depending on the recognized image capture device 115, to transmit stored control or image parameters to the playback device 430 for the respective playback mode (3D or 2D).

[0113] In method step 103 of the method according to the invention, the control unit 120 can automatically or manually output control instructions to the 3D endoscope for a first playback mode (3D) or second playback mode (2D) on the basis of a first and second quality value (Ql, Q2 or QL,QR) determined by means of an analysis, ie depending on image sensor-specific quality values ​​- f(Q) (see double arrow in playback device 430).

[0114] The analysis routines may include image analysis 124, contamination detection 125, contamination degree analysis 126, and average color tone detection 321 or average image contrast determination 317, as well as classification 315 or other image analysis methods. A surgical environment 127 may also be recognized by means of suitable comparisons with stored image structures.

[0115] For image optimization, a self-learning module 320 can be used, which is based in particular on artificial intelligence and can carry out the aforementioned methods with the aid of stored data or databases. The self-learning module has a trained model 322 which is trained to use suitable training data 331 (e.g. in the form of labeled image data) to evaluate individual images, image sections or pixels as to whether contamination is present and how severe this contamination is. In order to be able to quantify the degree of contamination, a classification 315 into corresponding categories is carried out. The classification 315 contains at least two classes, such as uncontaminated images and contaminated images, which can be cleaned by a cleaning module. Not only data but also the above-mentioned routines such as image analysis 124 etc. as well as cleaning parameters 128 for the cleaning module 130 can be stored in the memory 122.The cleaning parameters 128 can define parameters selected from the following group for individual fluid channels: fluid volume V, pressure p, pulse or cleaning duration t, number of pulses, total cleaning duration, pulse-pause ratio, type of fluid, fluid volume V, fluid volumes (liquid or gas quantity) and / or fluid velocity.

[0116] Based on the determined quality values ​​(Q1, Q2 or QR, QL), manual or automatic control instructions can be issued by the control unit for activating (method step 104) a cleaning module using fluid and / or gas. Using the cleaning parameters 128, the pressure p and the pulse duration t of a fluid pulse or a plurality of pulses at the distal end of the 3D endoscope for intracorporeal cleaning can preferably be controlled. In particular, very short fluid pulses with a duration in the range of only 200-1000 ms can only be reliably implemented with the aid of automated control. This allows the system to be controlled much more precisely than through manual activation and deactivation by a user.

[0117] An embodiment of a cleaning module 130 can have two separate fluid channels 131, 132, as shown in Figs. 5a and 5b, in order to clean the partial surfaces 110 and 111 of an image sensor or the two image sensors 111, 112 separately. The contaminated image sensor can then be cleaned, while the uncontaminated image sensor can continue to be used for 2D rendering mode. Each fluid channel 131, 132 can convey both liquid and gas in one channel to the nozzle arrangement 140 to also enable drying of an image sensor surface or image sensor.

[0118] Fig. 8 shows a further embodiment of the method according to the invention with a schematic flowchart. The flowchart is divided into five areas, partially indicated by columns: the first area of ​​initialization 300, the second area of ​​analysis 301, the third area of ​​process execution 303 for optimization, preferably by adapting from 3D to 2D and / or by cleaning, and the fourth area 304 of the optimization result with monitoring routine 129 and possible parameter adjustment 138 of the cleaning parameters 128. Finally, the possible process termination 160. Optimal image quality is achieved when the final values ​​after cleaning correspond to the initial values ​​at the beginning with white balance.

[0119] In the initialization phase 300, the method is started in step 401. Here, a video signal from a 3D endoscope can be provided as an input signal. In step 302, a check is made to determine whether a video signal is present. Based on this check, either a positive result (YES) or a negative result (NO) is obtained. If this query 302 results in a NO, the process is aborted in step 403. Since no video signal is present in the case of process abort 162, a check should be made after process abort 162 to determine whether all connections are present and whether a power supply is present. The user can also be notified that no optics were detected.

[0120] If the check or query 302 yields a positive result (YES), the process continues in the analysis area 301. For a cleaning process, cleaning parameters 128 can be preselected according to the connected optics of the 3D endoscope.

[0121] In step 124, the image analysis of the first and second image data of a 3D endoscope takes place. This analysis can be performed by an algorithm and / or by a self-learning module 320 (see Fig. 7). The images can optionally be divided into image sections. In this way, each half-image can be broken down into different sections (e.g., central region of the image) and an automatic detection of contamination in the relevant central field of view can take place. In the next method step 127, an analysis of the images of a 3D endoscope comprising first and second image data takes place to determine whether a surgical environment or intracorporeal structures can be recognized. For this purpose, the images are compared with images from a database that are stored in the memory 122 of the 3D endoscope or externally. Preferably, a comparison of at least one specific image structure with image structures from a reference database takes place.Object detection of typical intracorporeal structures is preferentially performed. If no surgical environment or intracorporeal structure is detected, the process is aborted (arrow NO to 162). Otherwise, the process continues to the dirt detection step 125.

[0122] The step of classification 315 into different classes or categories can optionally take place between the steps of operating room environment detection 127 and dirt detection 125. Here, not only dirt classification (dirty or not dirty) but also the following classifications can be made: smoky images, where the smoke can be extracted by an extraction module; images with quality loss; which can be optimized by modules other than a cleaning module; images that have one or more interfering objects; and / or images that indicate at least one form of damage. Once a specific class has been recognized, it can also be displayed to the user. If, for example, smoke is detected, communication can be established with at least one other operating room unit in order to extract the smoke.

[0123] Method step 125 is used to detect contamination, i.e. whether contamination is present or not. If there is no contamination, the method can continue with method step 124 of image analysis (see “NO” arrow to step 124). If, on the other hand, contamination is present, the method can continue with method step 126. In this step 126, the type and degree of contamination is determined, for example, with the aid of a comparison with data from the reference database using algorithms, either automatically or preferably with a self-learning module 322 of the control unit 120 (see Fig. 4). Contamination detection 125 and / or analysis of the degree of contamination in method step 126 can result in a specific quality value for each image sensor or sensor sub-area. The quality values ​​can be determined, for example, using a training-based model (see reference number 322 in Fig. 7).This model 322 is part of a self-learning module 320 in a control unit 120 of the image optimization system and works with a reference database. An external or local storage unit 122 can store the history of previous image acquisitions. Labeled video images from a 3D endoscope can also be stored in the memory 122 to determine the respective quality value (Q1, Q2). In this way, using stored parameters or characteristics, a captured image can be qualified with regard to image quality, allowing contamination to be detected more quickly, and the percentage of contamination to be subsequently determined.

[0124] The classification history is stored in memory 122 and is continuously supplemented with the currently acquired image data. Not only the acquired images from the previous steps are saved, but also the associated evaluations of the video signals. Storage in memory unit 122 can be local or decentralized. The so-called image history enables ongoing documentation and the model 322 to be trained can be continuously optimized. The larger the retrievable data sets, the more precise the machine learning can be, thus increasing the learning capacity of model 322. With a sufficiently large data set, for example, artificial intelligence with a deep data model can be used.

[0125] In step 126, the degree of contamination can be determined based on an evaluation of the image history stored in memory 122. A percentage loss of image quality can be indicated by the quality values ​​being worse by a factor of x% than the initial values ​​without contamination (see YES arrow x%). In step 126, the image quality is preferably evaluated as a function of permissible percentage "contamination" shares. Image edge regions can be given less weight than the image center. If only one of two image sensors is dirty, or if one of two dirty image sensors is only slightly dirty, e.g. in the edge regions, this is considered partial contamination and the image can be automatically adjusted from 3D to 2D by switching the output signal of the control unit in method step 422. The user can initiate this switching from 3D to 2D orOnly confirm the suggested image optimization manually after receiving a corresponding warning message.

[0126] If it was assessed in step 126 of the contamination level detection that cleaning of at least one image sensor or a partial area of ​​an image sensor can and should be carried out (cleanable with fluid), the method continues (see arrow YES to method step 141). In the area of ​​process implementation 303, for example, a cleaning module 130 can be activated to carry out cleaning with liquid 141 or optionally drying 142 with a gas. These two cleaning types 141 and 142 can also be combined. The process implementation 303 of the cleaning process can be activated automatically by a self-learning module 320 (see Fig. 7) or, alternatively, after the self-learning module 320 has recommended cleaning, it can be activated manually. If only one image sensor or a partial area is contaminated, cleaning can be carried out simultaneously for image optimization 422.

[0127] If the cleaning module has two fluid channels, only the respective contaminated image sensor or sensor surface can be cleaned using a corresponding nozzle, each directed at a specific image sensor. If condensate is detected, cleaning can also be carried out using gas alone, and drying can be initiated in step 142. For this purpose, the previously defined cleaning parameters are used according to the endoscope used. As a rule, liquids are used for cleaning in the case of heavy contamination, and corresponding liquid parameters 128 are set, while gas is sufficient for cleaning in the case of light contamination. The monitoring routine 129 checks the cleaning effectiveness after an activatable cleaning unit of the cleaning module 130 has been activated. In particular, it is monitored whether the field of view and the image quality are once again optimal and thus correspond to the associated quality value.If this is the case, the system returns to analysis routine 124 (see arrow YES). The cleaning parameters 128 can optionally be continuously optimized in process step 138 by a self-learning module based on artificial intelligence.

[0128] A closed control loop is provided for monitoring routine 129, which checks whether or not predefined target values ​​for image quality are achieved by the activated cleaning 141 and / or drying 142. Patient safety can be increased by monitoring routine 129. If the cleaning result is not optimal, a repeat of cleaning 141 and / or drying 142 can be initiated. If monitoring routine 129 determines that no improvement in image quality is achieved by cleaning 126 or 127, the optimization process can be aborted using the cleaning module in area 160 (see step 161: End).

[0129] The invention and an adaptation of the playback mode to 2D or 3D can be applied to a variety of 3D endoscopes established on the market, enabling continuous display of the respective examination object and thus reliable diagnosis and avoiding disturbances at the site. The optimization method and optimization system according to the invention can output control signals, either automatically or after manual confirmation, indicating whether optimization is necessary and whether image optimization can be performed by the controllable devices. For example, automated dirt detection can occur, and an automated image adjustment to the second playback mode (3D 2D) can be performed, with optional cleaning of only the dirty image sensor.

[0130] Especially in medical technology, quality loss in imaging and quick solutions such as image adjustment are very important. Automated processes advantageously avoid prolonged interruptions for the user and eliminate contamination or image distortion as quickly as possible. The continuous provision of training data continuously improves the quality of the algorithms, and continuous parameter adjustments ensure optimal tuning of the process execution.

[0131] Software for adapting 3D images to 2D or vice versa, as well as a computer program product, can be configured to perform the image optimization process. In this way, image analysis algorithms can quickly detect suboptimal images or disturbances such as image contamination that would not be immediately noticeable or even noticeable to a user in 3D image representations. Adapting to 2D can prevent fatigue or pain caused by inadequate 3D images. Using image analysis, a quality value and existing image data (image history) or parameter data can be evaluated and compared with the most recent image data and previous optimization results.If a self-learning module is used, existing data sets can be corrected by experts in an open artificial network in addition to automatic procedures, and the reference database in memory 122 can be expanded with new data sets automatically or manually. Continuous semi-automatic or automatic review of the optimization procedure increases safety for the user and thus for the person being examined using the endoscopic procedure.

[0132] List of reference symbols

[0133] 11 Image data or field captured by the first image sensor

[0134] 11L Data captured from the left part of the image sensor

[0135] 12 data captured by the second image sensor or field

[0136] 12R data captured from the right part of the image sensor 100 3D endoscope

[0137] 101 Acquiring first image data (11) and second image data (12)

[0138] 102 Receiving the image data (11, 12) by a control unit (120)

[0139] 103 Operating the 3D endoscope -f (Q)- in a first or second playback mode;

[0140] 104 Cleaning

[0141] 105 Stereo base

[0142] 110 image sensor

[0143] 110R right light-sensitive surface

[0144] HOL left light-sensitive area

[0145] 111 first image sensor

[0146] 112 second image sensor

[0147] 113 Contamination on first image sensor

[0148] 114 Contamination on second image sensor

[0149] 114R, r Contamination on right sensor surface 110R

[0150] 115 Image capture device

[0151] 116 cc

[0152] 120 control unit

[0153] 121 processor

[0154] 122 memory

[0155] 123 Initialization routine

[0156] 124 Image analysis

[0157] 125 process steps including contamination detection

[0158] 126 Analyze the degree of contamination

[0159] 127 Detection of the operating room environment

[0160] 128 cleaning parameters or selection of cleaning parameters in

[0161] Dependence on the determined endoscope type

[0162] 129 Monitoring routine

[0163] 130 Cleaning module 131 First fluid channel

[0164] 132 second fluid channel

[0165] 138 Parameter adjustment

[0166] 140 nozzle arrangement

[0167] 141 Cleaning with liquid

[0168] 141 Cleaning with gas: drying

[0169] 150 shaft

[0170] 151 first lighting device

[0171] 152 second lighting device

[0172] 154 first lighting device

[0173] 155 second lighting device

[0174] 156 third lighting device

[0175] 157 fourth lighting device

[0176] 160 Process abort

[0177] 161 End

[0178] 162 Process abort

[0179] 300 Initialization phase

[0180] 301 Analysis

[0181] 302 Video signal query

[0182] 304 Result of optimization

[0183] 315 Classification

[0184] 317 determine average image contrast

[0185] 320 Self-learning module

[0186] 321 determine middle color tone

[0187] 322 trained model

[0188] 400 Image optimization flowchart

[0189] 401 Start

[0190] 402 Warning about possible 2D playback mode

[0191] 403 Warning about possible 3D playback mode 411 Decision Yes / No

[0192] 412 Activation of the image optimization software for adaptation to 2D

[0193] 413 Activation of the image optimization software for adaptation to 3D 414 Automated activation of the software for adaptation without query

[0194] 415 automated activation of the software for adaptation to 3D

[0195] 422 Switching the output signal for second playback mode (2D)

[0196] 423 Switching the output signal for the first playback mode (3D)

[0197] 430 playback device 431 active 3D glasses

[0198] 432 Adjusting the 3D glasses to 2D

[0199] 433 Adjusting the 3D glasses to 3D

[0200] 700 3D Endoscope System

[0201] Q, Ql, Q2,QR,QL quality values

[0202] Stereo base

Claims

Patent claims 1. A method for image analysis of image data acquired by means of a 3D endoscope (100), in particular for medical applications, comprising the following steps: Acquiring (101) first image data (11, 11L) and second image data (12, 12R) by means of at least one first image sensor (110) or by means of a first image sensor (111) and a second image sensor (112) of the 3D endoscope; wherein the acquired first and second image data each comprise a two-dimensional image of a work area from different perspectives; Receiving (102) the image data (11, 12) by a control unit (120); wherein the control unit (120) is designed for an analysis (124) of the first and second image data (11R, 11L, 11, 12) in order to determine a quality value (Q1, Q2, QL, QR) in each case; Operating (103) the 3D endoscope (100) in a first reproduction mode or a second reproduction mode depending on the determined first and second quality values; wherein the control unit (100) is configured to output three-dimensional image data of the work area in the first reproduction mode based on the combination of the two two-dimensional images; and wherein the control unit is configured to output a two-dimensional image of the work area in the second reproduction mode based on the first image data or the second image data.

2. Method according to one of the preceding claims, wherein the analysis (124) of the first and second image data (11, 12) for determining the respective quality value comprises an image processing algorithm and / or uses a self-learning module to detect at least one of the following events: contamination, at least one interfering object, a loss of image quality, condensation, damage or smoke. The method according to claim 1 or 2, wherein, if the control unit (120) determines that a quality value is representative of at least one of the events: contamination, disturbing object, image quality loss, condensation, damage, or smoke, a two-dimensional image based on the image data without an event attributable to the other quality value is output automatically by means of the control unit or manually in the second playback mode. The method according to one of the preceding claims, wherein the analysis of the first and second image data comprises determining an average image contrast; in order to determine the quality value based on a comparison of the respective average image contrast using a visual threshold condition. The method according to claim 4, wherein the average image contrast is determined by at least one of the following analysis methods: Histogram method, application of a Gaussian filter, pattern recognition, corner detection, edge detection and recognition of objects. Method according to one of claims 1 to 5, wherein the first and second image data (1 IL, 11R, 11, 12) each comprise a plurality of colors occurring in the work area; and wherein the analysis of the first and second image data each comprises the detection of a mean hue; in order to determine the quality value based on a comparison of the most frequently occurring color from the plurality of colors with the respective mean hue. Method according to claim 6, wherein, if the most frequently occurring color from the plurality of colors is white or gray, smoke is detected; or if the most frequently occurring color from the plurality of colors is red, blood contamination is detected. Method according to one of the preceding claims, further comprising automatically outputting a warning message if a control instruction for activating a different playback mode is automatically output by the control unit. Method according to one of the preceding claims, wherein, in a method step of the pre-analysis, the respective image data is subdivided into image sections, wherein the quality value can be determined for each of the image sections, wherein the control unit is designed to reproduce exclusively the image sections with a permissible percentage of contamination or a permissible loss of quality, and / or to reproduce at least the image sections that can be assigned to an image center of the respective image sensor.Method according to one of the preceding claims, wherein, based on the determined quality value, control instructions are manually or automatically output by the control unit for activating (104) a cleaning module using fluid and / or gas, wherein the cleaning module is an activatable unit of the 3D endoscope for preferably intracorporeal cleaning or is provided as an extracorporeal unit. Method according to claim 10, wherein the control unit has a cleaning control algorithm that checks the cleaning effectiveness extracorporeally and / or intracorporeally with the activatable unit of the 3D endoscope for one or two image sensors (129), so that at least one two-dimensional image or one three-dimensional image with sufficiently good image quality can be reproduced or a repeat cleaning is activated. . Method according to one of claims 2 to 11, wherein the self-learning module determines at least one image structure from the image data of the respective image sensor by means of an artificial intelligence-based analysis, preferably using a neural network, in order to classify an event in the event of a detected deviation by means of the at least one determined image structure based on a comparison with image structures of a reference database stored in a memory unit. .The method of claim 12, wherein the self-learning module comprises a model that is trained with training data of image structures stored in the storage unit, wherein the trained model of the self-learning module is trained to classify the image data into the following event-dependent database classes: non-soiled images, soiled images that can be cleaned intracorporeally; soiled images that can be cleaned by a cleaning module; fogged images that can be dried by a module using gas; smoky images, wherein the smoke can be extracted by an extraction module;. Images with quality loss; which can be optimized by other modules; images which have one or more disturbing objects; and / or images which indicate at least one damage. . Method according to one of the preceding claims, wherein a 3D image is displayed in the first playback mode using a display device and / or 3D glasses and a 2D image is displayed in the second playback mode. . Method according to one of the preceding claims, wherein the 3D endoscope is selected from the following group: a stereoscope comprising at least one first image sensor, preferably with two separate light-sensitive surfaces; a facet lens; a stereoscope comprising at least one first image sensor and a second Image sensor; an imaging scope comprising at least a first image sensor, a second image sensor, and a third image sensor; an imaging scope comprising at least a first image sensor, a second image sensor, a third image sensor, and a fourth image sensor; and a stereoscope, which is designed as a fiber optic endoscope. A control system for image analysis using a 3D endoscope, in particular for medical applications for carrying out a method, according to one of the preceding claims, wherein the system comprises: a 3D endoscope (100) with at least one first image sensor (110), wherein the first image sensor is designed to acquire first image data (110R) and second image data (HOL); or a 3D endoscope with a first image sensor (111) designed to acquire first image data (11) and a second image sensor (112) designed to acquire second image data (12);wherein the acquired first and second image data each comprise a two-dimensional image of a work area from different perspectives; a control unit in communication with the at least one image sensor for receiving the first and second image data; wherein the control unit is configured to analyze the first and second image data in order to determine a quality value in each case; wherein the control unit (120) is configured to automatically or manually output control instructions to the 3D endoscope (100) for a first or second reproduction mode (103) based on a specific first and second quality value (Q1, Q2); wherein the control unit is configured to output three-dimensional image data of the work area in the first reproduction mode based on the combination of the two two-dimensional images;and wherein the control unit is configured to output a two-dimensional image of the work area based on the first image data or the second image data in the second playback mode; . A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method of any one of claims 1 to 15. . A trained model, which is preferably self-learning and trained with stored image structures and / or a collection of features, and wherein the trained model is configured to perform the steps of the method of any one of claims 2 to 15.

Images