Endoscopic medical imaging analysis and operation system and method and computer program

Abstract

The invention relates to an endoscopic medical imaging analysis and operation system and method and a computer program. A method of endoscopic medical imaging analysis and operation includes: capturing an endoscopic image of a throat or pharyngeal tissue structure during examination of the throat using an endoscope inserted through a nose or mouth of a patient; the captured endoscopic images are fed to two instances of artificial intelligence, a first instance of which has been trained to identify suspicious regions of the laryngeal or pharyngeal tissue structure showing signs of changes relative to healthy laryngeal or pharyngeal tissue, and a second instance of which has been trained to identify suspicious regions of the laryngeal or pharyngeal tissue structure showing signs of changes relative to healthy laryngeal or pharyngeal tissue. And wherein a second instance has been trained to detect and classify anomalies in the mucosal layer or tissue of the laryngeal or pharyngeal tissue; overlaying one or more indicia on the captured endoscopic image, the one or more indicia indicating one or more regions indicated as suspicious by the first instance of artificial intelligence; and displaying the overlaid captured endoscopic image on a monitor together with information provided by the second instance of artificial intelligence.

Description

Technical Field

[0001] The present disclosure relates to systems and methods and computer programs for endoscopic medical imaging analysis and operation. Background Art

[0002] A laryngeal biopsy is a routine diagnostic procedure in ENT (ear, nose, and throat) medicine to evaluate and examine the larynx, and the rest of the procedure is performed using a flexible or rigid endoscope inserted through the patient's nose or mouth. This allows visualization of the larynx, pharynx, and surrounding tissues as well as identification of changes in the mucosal layer or any suspicious areas that may indicate inflammation or cancer.

[0003] In a busy clinic, the procedure is often done in an office setting without anesthesia and takes only a few minutes. However, this time constraint and lack of patient cooperation can make it difficult for the physician to fully examine the anatomy and focus on small details that may indicate early disease. Additionally, the anatomy of an elderly patient can be more difficult to examine, making it difficult to explore all mucosa and identify potential signs of disease.

[0004] It is common for certain types of lesions to be overlooked during medical workup, especially when specialists are not readily available. This is often the case in hospitals where there is a lack of such specialists, as specialists are primarily located in larger university centers. In the field of ENT, early detection is particularly important in order to preserve the function of the larynx and improve patient outcomes. However, these early stage lesions may be frequently missed by traditional methods and human observation (often due to lack of experience).

[0005] The main problem in the early diagnosis of cancerous or benign lesions in ENT is to accurately and efficiently identify and detect potential lesions or abnormal tissue in the early stages of the disease. This is particularly challenging because ENT cancers in the early stages may not be visible using traditional methods such as white light endoscopy, and may be missed by human observation alone. The use of narrow band imaging (NBI) can help identify lesions. NBI is a medical imaging technique that uses specific wavelengths of light to enhance the visualization of blood vessels, mucosa, and other structures. The technology is widely used in the ENT field to improve the visualization of blood vessels, and its efficiency is scientifically proven. The use of this imaging enhancement can help early detection of abnormal tissue or lesions that may indicate inflammation or cancer. However, the effectiveness of this technology depends on the skill level of the user interpreting the image.

[0006] In addition, a lack of expertise and experience among medical professionals in identifying early-stage ENT cancers may occur, which may lead to delayed diagnosis or misdiagnosis. In addition, the complexity of the anatomical structure of the ENT region and the high variability of lesions make it difficult to develop standard diagnostic procedures. This may lead to a lack of consistency in diagnostic results between different hospitals and clinics. Summary of the invention

[0007] The invention described herein aims to address these challenges by supporting physicians and improving early detection and diagnosis. It is therefore an object to provide a method and system that helps medical practitioners in the field of ENT to provide better diagnosis and patient outcomes.

[0008] This purpose can be achieved by an endoscopic medical imaging analysis and operation method, the method comprising:

[0009] capturing endoscopic images of structures of the larynx, pharynx, and / or surrounding tissue during examination of the larynx and / or pharynx using an endoscope inserted through the patient's nose or mouth,

[0010] feeding the captured endoscopic images to two instances of artificial intelligence, a first instance of the two instances having been trained to identify suspicious areas of the structure of the larynx, pharynx and / or surrounding tissue that show signs of changes relative to a healthy larynx, pharynx and / or surrounding tissue, and a second instance of the two instances having been trained to detect and classify abnormalities in the mucosal layer or tissue of the larynx,

[0011] overlaying one or more markers indicating one or more regions indicated as suspicious by the first instance of artificial intelligence on the captured endoscopic image, and

[0012] The overlaid captured endoscopic image is displayed on a monitor along with information provided by the second instance of artificial intelligence.

[0013] The double-A auxiliary diagnostic tool addresses the above challenges and improves the accuracy and efficiency of ENT's early diagnosis of abnormalities such as cancer, enabling ENT physicians to better triage and treat during endoscopy of the upper aerodigestive tract, thereby bringing better outcomes to patients.

[0014] The two-stage AI-assisted method enhances the assistance that the system can provide to the medical professional performing the examination by providing the professional performing the examination with a first-stage general overview suitable for finding any suspicious areas and a second-stage detailed analysis of such suspicious areas, so that the professional can switch between the assisted overview and the assisted detailed analysis according to the needs of the examination.

[0015] In an embodiment, the endoscopic image is one of a WLI image, an NBI image, and a mixture of a WLI image and an NBI image. For example, the first stage can be completed with a WLI (white light imaging) image, and the second stage detail analysis can be performed using an NBI (narrow band imaging) image. Narrow band imaging (NBI) is a medical imaging technique that uses light of a specific wavelength to enhance the visualization of blood vessels, mucosa, and other structures. This technology is widely used in the ENT (ear, nose, and throat) field to improve the visualization of blood vessels. NBI images have been shown to be superior to white light imaging (WLI) in highlighting abnormal mucosal or tissue structures in the larynx and surrounding tissues.

[0016] Many clinical studies have shown correlations between different vascular classifications and various disease states. Changes in vascular morphology and density have been shown to reflect the severity of the disease. At this time, four different vascular classifications based on NBI have been widely adopted in the field of otolaryngology. These classifications help visually identify the type of lesions and diseases. The currently disclosed methods and systems are directed to enhancing the diagnostic capabilities of otolaryngologists by combining instances of artificial intelligence (AI) associated with NB and machine learning (ML) techniques to guide targeted biopsy procedures, thereby improving patient outcomes.

[0017] The overlay of the method of the invention to be displayed, for example, within a specific GUI (Graphical User Interface) will identify lesions and other abnormalities earlier with a higher accuracy than a human observer can do, because small blood vessels that may be part of the lesion are usually not visible to the human eye. If missed, these may lead to recurrence. Combined with the fact that NBI images are used, ENT doctors are assisted by the overlay even if they have less experience (e.g. due to lack of cases in less populated areas).

[0018] The system used in the method of the present disclosure has two different modules with two different instances of artificial intelligence, wherein the first module supports the physician in detecting abnormal tissue and the second module detects and characterizes lesions. The second module can be trained to perform the detection of lesions alone, or it can be fed the findings of the first module and proceed with the characterization of lesions detected and differentiated by the first module.

[0019] In an embodiment, the first instance of artificial intelligence can be a convolutional neural network (CNN) with a classifier that has been trained through at least one of supervised learning and unsupervised learning on a large number of endoscopic images of structures of the larynx, pharynx and / or surrounding tissues to classify suspicious areas of the larynx, pharynx and / or surrounding tissues in the captured endoscopic images.

[0020] The term "unsupervised learning" generally describes a process in which only images of healthy laryngeal tissue are used to train the neural network. Any area that does not conform to what the neural network has learned to be healthy will be labeled as suspicious. Supervised training takes the training a step further. Diseased areas are pre-labeled and classified in the training data so that the neural network is trained to not only identify suspicious areas, but also provide an estimate of what kind of lesion is shown in the NBI image.

[0021] In another embodiment, the second instance of artificial intelligence can be a convolutional neural network (CNN) with a classifier, which has been trained by supervised learning of a large number of pre-classified endoscopic images of the structure of the larynx, pharynx and / or surrounding tissues to classify suspicious areas of laryngeal tissue in captured endoscopic images according to the type and / or severity of the abnormality in the suspicious area. In this case, the second instance of artificial intelligence can be trained to identify changes in the structure of the larynx, pharynx and / or surrounding tissue relative to a healthy larynx as a suspicious object derived from one or more of cancerous and benign lesions, vascular morphology, vascular density, vascular typing and mucosal surface structure. Thus, the disclosed algorithm uses image analysis technology to help otolaryngologists identify vascular typing, density, morphology and structures on the mucosal surface. By providing such information, the algorithm supports the doctor's diagnostic ability and helps identify potential pathologies.

[0022] In an embodiment, the overlay item is created as a rectangle or an outline. The overlay item is the result of the analysis performed by the first instance of artificial intelligence, and its visual recognition is used to support the detection of abnormal mucosal layers. If the suspicious area is visualized after identifying the suspicious area by, for example, changes in vascular morphology, the doctor can distinguish the suspicious area more quickly. In the case of multiple suspicious areas, there may be several marks, such as rectangles, to avoid missing sites. Therefore, the mark is used to enhance and support the monitoring of the entire anatomical structure, thereby reducing the risk of missing any potential abnormalities that may cause cancer development. The number of rectangles can vary according to the number of suspicious areas found. The algorithm can add new overlay rectangles according to the mucosal changes found during endoscopy.

[0023] In an embodiment, the rectangle or outline can be adjusted based on the user's movement, thereby providing a more dynamic and personalized experience. Adjustments regarding movement can be achieved, for example, through motion analysis in the image, where a shift or movement in three dimensions is identified from a comparison between subsequent frames of endoscopic imaging, or where identification of such a shift can be assisted by motion sensors within the endoscope. In addition, the user can have the ability to customize the overlay display, such as highlighting the target lesion with a continuous line and highlighting surrounding lesions with dotted lines. This feature helps the physician focus on the target lesion and obtain a closer view to identify more details, thereby reducing the risk of overlooking small lesions that may lead to recurrence.

[0024] A real-time measure of the classification accuracy of the first instance of the artificial intelligence and / or the second instance of the artificial intelligence may be displayed, the measure of accuracy being based in particular on factors such as the distance of the endoscope from the lesion, the angulation of the length, and the clarity of the image. For example, the measure of accuracy may be displayed as a number (such as a percentage) or as a color coding. The artificial intelligence algorithm may be based on various factors, including distance, image quality, light reflection, and available data sets. Such parameters may be used to determine the real-time accuracy of the instance of the artificial intelligence.

[0025] In an embodiment, the first instance of artificial intelligence or the second instance of artificial intelligence can be trained to identify image defects that require adjustment or cleaning of the endoscope lens, and if the image is not clear enough, notify the user of recommendations for adjusting or cleaning the endoscope lens if such an image defect occurs.

[0026] The artificial intelligence functionality can be activated and deactivated as needed during the endoscopic procedure. Either or both instances of artificial intelligence can be designed to continuously learn from new images, which can be classified as they are generated by the physician performing the examination, where the physician confirms, changes, or adds findings regarding suspicious areas or classifications.

[0027] The user may be able to program rectangles or other markers with different colors (e.g., green, yellow, and red) based on the severity of the lesion using the most recent WHO nomenclature. For example, the classification may depend on preset database images and based on identification guidelines that have been used.

[0028] A graphical user interface (GUI) may be used where patient information and other information (such as settings or setting preferences) and lesion specific information may be displayed side by side with the endoscopic image. The GUI may also indicate whether a CAD mode is active, or WLI or NBI conditions. The complete interface may also be recorded.

[0029] In an embodiment, a second instance of artificial intelligence can be trained to utilize the classification of blood vessels in the NBI image in order to obtain more accurate information. The second instance of artificial intelligence can also be trained to estimate the size of the lesion and provide an estimate of its length. The size information can then be reconstructed into a two-dimensional lesion map, which provides a clearer representation of the anatomical region in question. This will help the physician better identify and demarcate suspicious areas.

[0030] Potential objectives of the present disclosure are also achieved through an endoscopic medical imaging analysis and operation system, which includes: a video endoscope suitable for feeding through a patient's mouth or nose for laryngeal and / or pharyngeal examination; an image analyzer connected to the video endoscope for receiving endoscopic images from the endoscope, the image analyzer having a first instance of artificial intelligence and a second instance of artificial intelligence, the first instance of artificial intelligence being trained to identify suspicious areas of the structure of the larynx, pharynx and / or surrounding tissue, the suspicious areas showing signs of changes relative to a healthy larynx, pharynx and / or surrounding tissue, the second instance of artificial intelligence having been trained to detect and classify abnormalities in the mucosal layer or tissue of the larynx, pharynx or surrounding tissue, the image analyzer being further configured to overlay a marker indicating an area indicated as suspicious by the instance of artificial intelligence on the captured endoscopic image; and a monitor connected to the image analyzer for displaying the endoscopic image provided by the image analyzer and information provided by the second instance of artificial intelligence.

[0031] The system embodies the same features and advantages as the above-mentioned endoscopic medical imaging analysis and operation methods of the present disclosure. It can include a light source capable of generating light for WLI as well as for NBI.

[0032] In an embodiment, a first instance of artificial intelligence is a convolutional neural network (CNN) with a classifier, the CNN having been trained by at least one of supervised learning and unsupervised learning of a large number of endoscopic images of structures of the larynx, pharynx and / or surrounding tissues to classify suspicious regions of the larynx, pharynx and / or surrounding tissues in captured endoscopic images, and / or a second instance of artificial intelligence is a convolutional neural network (CNN) with a classifier, the CNN having been trained by supervised learning of a large number of pre-classified endoscopic images of structures of the larynx, pharynx and / or surrounding tissues to classify suspicious regions of the larynx, pharynx and / or surrounding tissues in captured endoscopic images according to the type and / or severity of abnormalities in the suspicious regions.

[0033] In another embodiment, the type of abnormality includes one or more of cancerous and benign lesions, cancerous and benign vascular morphology, cancerous and benign vascular density, cancerous and benign vascular typing, and cancerous and benign structures of mucosal surfaces.

[0034] In embodiments, overlay items may adjust based on the user's movements.

[0035] In further embodiments, real-time measurements of the classification accuracy of the first instance of artificial intelligence and / or the second instance of artificial intelligence are displayed, particularly based on factors such as the distance of the endoscope from the lesion, the angulation of the length, and the clarity of the image.

[0036] The first instance of artificial intelligence or the second instance of artificial intelligence can be trained to identify image defects that require adjustment or cleaning of the endoscope lens or cleaning of the anatomical region, and when such image defects occur, the user can be notified of recommendations for adjusting or cleaning the endoscope lens or cleaning the anatomical region.

[0037] In an embodiment, either or both instances of artificial intelligence are designed to continuously learn from new images. New images can be classified as they are generated by the physician performing the examination by confirming, changing or adding findings regarding suspicious areas or classifications.

[0038] In one embodiment, a second instance of the artificial intelligence is trained to estimate the size of a lesion and provide an estimate of its length, the size information being particularly useful in generating a two-dimensional lesion map of the anatomical region.

[0039] The underlying objects of the present disclosure are also achieved by a computer program stored on a non-volatile medium, which is designed to perform any of the embodiments of the method of the present disclosure when executed on an image processor of the system of the present disclosure.

[0040] Other features will become apparent from the description of the embodiments and from the claims and drawings.The embodiments may implement individual features or a combination of several features. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Without limiting the overall intent of the invention, the embodiments described below are based on exemplary embodiments, wherein explicit reference is made to the drawings with regard to the disclosure of all details that are not explained in greater detail in the text.

[0042] In the attached picture:

[0043] Figure 1 A schematic representation showing the steps of the image analysis and manipulation method is shown,

[0044] Figure 2 A schematic representation of an implementation of an example of artificial intelligence is shown,

[0045] Figure 3 shows an embodiment of a GUI presenting the results of the method,

[0046] Figure 4 Another embodiment of a GUI presenting the results of the method is shown,

[0047] Figure 5 A representation of training data showing an example of artificial intelligence; and

[0048] Figure 6 A schematic representation of a system according to the present disclosure is shown.

[0049] In the drawings, elements of the same or similar type or respectively corresponding parts are provided with the same reference numerals to prevent the need for reintroduction of the items. DETAILED DESCRIPTION

[0050] Figure 1 A schematic representation of the steps of an image analysis and manipulation method is shown, which may be performed using a system, an exemplary embodiment of which is shown in Figure 5 In step S10, images of narrow band imaging (NBI) or white light imaging (WLI) from the patient's larynx are captured using a video endoscope or a separate camera head attached to a purely optical endoscope, the video endoscope may have an image sensor at its distal end or may be a different type having an image sensor at the proximal end. In step S20, these images are sent to an image processor that runs two instances of artificial intelligence that have been trained with WLI and / or NBI images of the larynx to respectively identify suspicious areas in the mucosal surface or tissue of the larynx and classify abnormalities according to type and severity. In steps S30 and S32, a first instance and a second instance of the artificial intelligence respectively process the images and identify suspicious areas based on their respective training. Identification of suspicious areas can be accomplished by classifying such areas as suspicious.

[0051] The first instance of the artificial intelligence provides suspicious regions as output, so that in step S40, the image analyzer can overlay a marker on the image indicating the suspicious regions found by the first instance of the artificial intelligence. The type and severity information provided by the second instance of the artificial intelligence and any additional relevant information are added to the display. Then, in step S50, the overlaid image is displayed to the person performing the inspection or biopsy on a monitor along with any information provided by the second instance of the artificial intelligence.

[0052] Figure 2A schematic representation of an embodiment of an example of artificial intelligence useful for identification of suspicious areas in laryngeal images is shown. This embodiment of the example of artificial intelligence can be applicable to the first example of artificial intelligence as well as the second example, and is configured as a convolutional neural network (CNN) with a classifier on the output side. The image is fed into the CNN on the input side and feature extraction is performed by being processed within the convolutional base ("convolution") and pooling ("merging") of the CNN according to the known working principles of convolutional neural networks. The convolution and pooling stages can be repeated one or more times. As a result of feature extraction, features of the NBI image that can indicate suspicious areas are identified. These are then set into the classification layer of the CNN (which can be a fully connected layer or several layers) and processed towards the output layer, where each node represents the classification for which the CNN has been trained. In the simplest case, there may be only a few nodes in the output layer that represent the presence or absence of suspicious areas, which may be sufficient for the first example. In a more complex embodiment applicable to the second example of artificial intelligence, the output layer may have several nodes representing different diseases, lesion types, severity, etc.

[0053] The explanation of the principles using convolutional neural networks should not be interpreted as limiting, as other examples of artificial intelligence and machine learning can also be trained and used for this purpose, such as for example support vector machines, learning vector quantization or random forest models.

[0054] Figure 3 A first embodiment of a GUI 10 showing a first result of the method is shown. The GUI 10 will be displayed on a monitor to a physician performing an endoscopic examination and possibly a biopsy of a patient's larynx. The main portion of the GUI 10 is used to display an endoscopic image of a portion of the upper aerodigestive tract including the larynx. An insert in the upper left corner can display patient information and the duration of the procedure as well as an indication of the progress of the examination in the form of a progress bar, above which symbols representing anatomical milestones that have been passed during the examination are displayed, such as the mouth (diamond symbol), tonsils (circular symbol), and larynx (shell symbol). These symbols are highlighted because the endoscope tip has passed or reached these milestones. Other milestones further down have not yet been reached by the endoscope tip, so the other milestones further down remain gray.

[0055] An endoscopic image of the upper aerodigestive tract is overlaid with rectangular markers of areas that the first instance of artificial intelligence has identified as suspicious. Each rectangle is accompanied by a percentage number indicating the confidence level of the result. The abnormal 22 finding with the highest confidence level of 92% is highlighted as a primary marker 11 with a solid border line, while other findings with lower confidence levels between 52% and 88% are marked with dashed lines as secondary markers 12. Findings with confidence levels below 50% are not displayed. Depending on the circumstances, a minimum confidence level for display can be selected that is higher or lower than 50%. A lower border value will have the benefit of being able to highlight very mild abnormalities at the expense of increasing the number of false positive markers. A higher border value will reduce the number of false positives at the expense of sensitivity to very mild abnormalities, thereby increasing the risk of missing malignant changes at very early stages.

[0056] Figure 4 1 shows a second embodiment of the GUI 10 or a second exemplary configuration of the GUI 10. In this configuration, results from the second instance of the artificial intelligence are shown overlaid on and next to the endoscopic NBI image of the laryngeal tissue 20. Since the NBI image has been processed by the second instance of the artificial intelligence trained for this purpose, the suspicious tissue structure 13 has been identified and classified and is surrounded by a rectangular frame.

[0057] In the left part of the GUI 10, several inserts are displayed which provide information about the suspicious area and suggestions about the artificial intelligence based instance findings about the suspicious area. Figure 3 , showing general patient and surgical information and a progress bar. A second insert below the first insert shows the outline of the suspicious area with respect to its location inside the larynx in the form of a heat map generated by the first instance of artificial intelligence. As can be seen from the second insert, the suspicious area is located in the upper right part of the larynx. Its relative size can also be derived from this graphical representation.

[0058] The third insert directly below the heat map display contains two pieces of information, namely a size indication 16 ("5 mm") and a classification indication 18 ("SCC" for "squamous cell carcinoma"). These indications are accompanied by a confidence level indication consisting of three bars. For the size indication 16, two of the three bars are highlighted, indicating a medium confidence level for the size assessment result, while three of the three bars of the classification indication are highlighted, indicating a high confidence level for the classification of the abnormality 22.

[0059] After imaging has been switched to white light imaging (WLI), markings of suspicious areas in the NBI image that have been identified by the instance of artificial intelligence can remain in the display with some lateral movement and magnification (due to the shift in the position of the endoscope detected in the image). This is particularly helpful in the actual act of obtaining a biopsy sample from the suspicious area.

[0060] Figure 5 An exemplary representation of training data for two instances of artificial intelligence is shown. The top training data is an unclassified NBI training image containing only healthy tissue. Such training data is required for so-called unsupervised training of artificial intelligence. The training will prompt the instance of the artificial intelligence to accept any laryngeal tissue that conforms to these training data as not suspicious. If there is a deviation from the structure that has been trained as normal, the affected area will automatically appear and be classified as a suspicious area.

[0061] An alternative to so-called supervised learning is based on free classification and on training images, some of which show only healthy tissue, while others of which contain damaged or diseased areas; where such affected areas are simply indicated and classified as suspicious, or with a higher granularity as to their type and severity. Supervised training leads to an optimization such that specific nodes in the output layer of the CNN are trained to indicate the various types and severities, respectively.

[0062] Figure 5 The third type of input image indicated in is a new endoscopic NBI image captured during an inspection and biopsy performed using a pre-trained instance of the artificial intelligence, and is used to continuously update the training of the artificial intelligence instance in both unsupervised and supervised learning. In the case of unsupervised learning, the physician performing the inspection may indicate that the current view of the larynx has no suspicious areas. In this case, the NBI image can be used to further establish the ground truth of the NBI training image for unsupervised learning. Alternatively, in the presence of a suspicious area, the physician can confirm the finding and classification of the suspicious area or change a finding that he has found to be incorrect, so that the image including its correct classification can be used for further supervised training of the instance of the artificial intelligence.

[0063] Figure 6is a schematic diagram of a system according to the present disclosure. The system includes an endoscope 30, which may be a flexible endoscope having optical elements and an image sensor at its distal end, or a rigid endoscope having internal relay optical elements and an image sensor in its handle or in a separate camera head attached to the endoscope. The image sensor is connected to a controller 32 of a system for controlling imaging, lighting and other parameters of the endoscopic examination. The controller 32 may include a light source 36 capable of producing white light for WLI and, alternatively, narrow-band illumination for NBI. The light source may also be implemented in the endoscope 30 and controlled by the controller 32. The controller 32 also includes an image analyzer 34, which runs an instance of artificial intelligence that is trained to identify and classify suspicious areas in laryngeal tissue according to the previous description. The image analyzer 34 is also configured to provide an overlay item to the NBI image that visualizes or highlights suspicious areas detected by the instance of artificial intelligence. For example, with respect to Figure 3 Examples of such overlay items are shown and described.

[0064] Monitor or display 38 is used to display the overlaid image (e.g., as Figure 3 10 shown), may include further information, such as information regarding the size, type and / or severity of the suspicious area, or recommendations regarding the number and / or locations of biopsy samples to be obtained or regarding the type of forceps to be used.

[0065] In addition, the system may include a feedback terminal 40, which is provided for the doctor performing the examination of the larynx to provide feedback on the finding of the instance of the artificial intelligence on the suspicious area, such as confirming or changing the finding. This feedback can be used for further training of the instance of the artificial intelligence, such as about Figure 4 Such a feedback terminal 14 may be a separate device, such as a computer, or may be integrated into a display, as indicated by the dashed line surrounding the display 38 and the feedback terminal 40, in which case the display has a touch screen, and the GUI 10 displays a defined area of ​​the screen for feedback by touching the touch screen. For example, a touch on an insert area of ​​the GUI 10 displaying information about the type or severity of a lesion or other suspicious area may open a dialog for correcting the automatically generated information with the doctor's findings. Instead of input via a touch screen, correction may also be performed via voice recognition or other suitable means.

[0066] All named features, including those features obtained from the drawings alone, and individual features disclosed in combination with other features, are considered to be important to the present invention individually and in combination. Embodiments according to the present invention can be realized by individual features or a combination of several features. Features combined with the words "particularly" or "particularly" will be considered as preferred embodiments.

[0067] Although what is considered to be an embodiment of the present invention has been shown and described, it should of course be understood that various modifications and changes in form or detail may be easily made without departing from the spirit of the present invention. Therefore, the present invention is not limited to the exact form described and shown, but should be interpreted as covering all modifications that may fall within the scope of the appended claims.

[0068] Reference Signs List 10 GUI

[0069] 11 Main Marks

[0070] 12 marks

[0071] 13Suspicious Organizational Structure

[0072] 14 Indication of suspicious areas

[0073] 16 Recommendations on the number of biopsy samples 18 Recommendations on the forceps to be used 20 Laryngeal tissue

[0074] 22 Abnormal

[0075] 30 Endoscope

[0076] 32 Controller 34 Image analyzer with AI instance 36 Light source

[0077] 38 display 40 feedback terminal

Claims

1. An endoscopic medical imaging analysis and operation method, characterized in that: The endoscopic medical imaging analysis and operation method comprises the following steps: capturing endoscopic images of structures of the larynx, pharynx, and / or surrounding tissue during examination of the larynx, pharynx, and / or surrounding tissue using an endoscope inserted through the nose or mouth of a patient, feeding the captured endoscopic images to two instances of artificial intelligence, a first instance of the two instances having been trained to identify suspicious areas of structure of the larynx, pharynx and / or surrounding tissue, the suspicious areas showing signs of changes relative to a healthy larynx, pharynx and / or surrounding tissue, and a second instance of the two instances having been trained to detect and classify abnormalities in the mucosal layer or tissue of the larynx, pharynx and / or surrounding tissue, overlaying one or more markers on the captured endoscopic image, the one or more markers indicating one or more areas indicated as suspicious by the first instance of artificial intelligence, and The overlaid captured endoscopic image and information provided by the second instance of artificial intelligence are displayed on a monitor.

2. The endoscopic medical imaging analysis and operation method according to claim 1, wherein: The endoscopic image is one of a WLI image, an NBI image, and a mixture of the WLI image and the NBI image.

3. The endoscopic medical imaging analysis and operation method according to claim 1, in, The first example of artificial intelligence is a convolutional neural network (CNN) having a classifier, which has been trained by at least one of supervised learning and unsupervised learning on a large number of endoscopic images of structures of the larynx, pharynx and / or surrounding tissues to classify suspicious areas of the larynx, pharynx and / or surrounding tissues in the captured endoscopic images, and / or Therein, the second instance of artificial intelligence is a convolutional neural network (CNN) with a classifier, which has been trained through supervised learning of a large number of pre-classified endoscopic images of structures of the larynx, pharynx and / or surrounding tissues, so as to classify the suspicious areas of the larynx, pharynx and / or surrounding tissues in the captured endoscopic images according to the type and / or severity of the abnormalities in the suspicious areas.

4. The endoscopic medical imaging analysis and operation method according to claim 3, wherein: The types of abnormalities include one or more of cancerous lesions and benign lesions, cancerous blood vessel morphology and benign blood vessel morphology, cancerous blood vessel density and benign blood vessel density, cancerous blood vessel typing and benign blood vessel typing, and cancerous structures and benign structures on the mucosal surface.

5. The endoscopic medical imaging analysis and operation method according to claim 1, wherein: Overlays are created as rectangles or outlines.

6. The endoscopic medical imaging analysis and operation method according to claim 1, wherein: Overlays can adjust based on the user's movement.

7. The endoscopic medical imaging analysis and operation method according to claim 1, wherein: In particular, a real-time measurement of the classification accuracy of the first instance of artificial intelligence and / or the second instance of artificial intelligence is displayed based on factors such as the distance of the endoscope from the lesion, the angulation of the length, and the clarity of the image.

8. The endoscopic medical imaging analysis and operation method according to claim 1, wherein: The first instance of artificial intelligence or the second instance of artificial intelligence is trained to identify image defects that require adjustment or cleaning of the endoscope lens or cleaning of the anatomical region, and in the event of such image defects, notify the user of recommendations for adjusting or cleaning the endoscope lens or cleaning the anatomical region.

9. The endoscopic medical imaging analysis and operation method according to claim 1, wherein: Either or both of the two instances of artificial intelligence are designed to continuously learn from new images, wherein, in particular, the new images are classified by the physician performing the examination as they are generated, by confirming, changing or adding findings for suspicious areas or classifications.

10. The endoscopic medical imaging analysis and operation method according to claim 1, wherein: The second instance of artificial intelligence is trained to estimate the size of a lesion and provide an estimate of the length of the lesion, the size information being particularly useful for generating a two-dimensional lesion map of the anatomical region.

11. An endoscopic medical imaging analysis and operation system, characterized in that: The endoscopic medical imaging analysis and operation system includes: a video endoscope, which is suitable for feeding through the patient's mouth or nose for laryngeal and / or pharyngeal examination; an image analyzer, which is connected to the video endoscope for receiving endoscopic images from the video endoscope, and the image analyzer has a first instance of artificial intelligence and a second instance of artificial intelligence, the first instance being trained to identify suspicious areas of the structure of the larynx, pharynx and / or surrounding tissue, the suspicious areas showing signs of changes relative to a healthy larynx, pharynx and / or surrounding tissue, and the second instance has been trained to detect and classify abnormalities in the mucosal layer or tissue of the larynx, pharynx or surrounding tissue, and the image analyzer is also configured to overlay a mark indicating the area indicated as suspicious by the instance of artificial intelligence on the captured endoscopic image; and a monitor, which is connected to the image analyzer for displaying the endoscopic image provided by the image analyzer and information provided by the second instance of artificial intelligence.

12. The endoscopic medical imaging analysis and operation system according to claim 11, in, The first example of artificial intelligence is a convolutional neural network (CNN) having a classifier, which has been trained by at least one of supervised learning and unsupervised learning of a large number of endoscopic images of structures of the larynx, pharynx and / or surrounding tissues to classify suspicious areas of the larynx, pharynx and / or surrounding tissues in the captured endoscopic images, and / or Therein, the second instance of artificial intelligence is a convolutional neural network (CNN) with a classifier, which has been trained through supervised learning of a large number of pre-classified endoscopic images of structures of the larynx, pharynx and / or surrounding tissues, so as to classify suspicious areas of the larynx, pharynx and / or surrounding tissues in the captured endoscopic images according to the type and / or severity of abnormalities in the suspicious areas.

13. The endoscopic medical imaging analysis and operation system according to claim 11, wherein: The types of abnormalities include one or more of cancerous lesions and benign lesions, cancerous blood vessel morphology and benign blood vessel morphology, cancerous blood vessel density and benign blood vessel density, cancerous blood vessel typing and benign blood vessel typing, and cancerous structures and benign structures on the mucosal surface.

14. The endoscopic medical imaging analysis and operation system according to claim 11, wherein: Overlays can adjust based on the user's movement.

15. The endoscopic medical imaging analysis and operation system according to claim 11, wherein: In particular, a real-time measurement of the classification accuracy of the first instance of artificial intelligence and / or the second instance of artificial intelligence is displayed based on factors such as the distance of the endoscope from the lesion, the angulation of the length, and the clarity of the image.

16. The endoscopic medical imaging analysis and operation system according to claim 11, wherein: The first instance of artificial intelligence or the second instance of artificial intelligence is trained to identify image defects that require adjustment or cleaning of the endoscope lens or cleaning of the anatomical region, and in the event of such image defects, notify the user of recommendations for adjusting or cleaning the endoscope lens or cleaning the anatomical region.

17. The endoscopic medical imaging analysis and operation system according to claim 11, wherein: Either or both of these instances of artificial intelligence are designed to continually learn from new images.

18. The endoscopic medical imaging analysis and operation system according to claim 11, wherein: New images are classified as they are generated by the physician performing the examination by confirming, changing or adding findings for suspicious areas or classifications.

19. The endoscopic medical imaging analysis and operation system according to claim 11, wherein: The second instance of artificial intelligence is trained to estimate the size of a lesion and provide an estimate of the length of the lesion, the size information being particularly useful for generating a two-dimensional lesion map of the anatomical region.

20. A computer program stored on a non-volatile medium, the computer program being designed to perform the steps according to any one of claims 1 to 10, in particular when run on an image analyzer of an endoscopic medical imaging analysis and operation system according to any one of claims 11 to 19.

Images