Image display system, training dataset, image display program, and diagnostic support system

Abstract

This invention provides an image display system, a training dataset, an image display program, and a diagnostic support system that enable accurate and rapid tracheal intubation. [Solution] The captured image display processing unit 221 displays the captured image on the display screen 21. When images of the airway and other anatomical structures are displayed on the display screen 21, the discrimination processing unit 222 uses a trained model generated by machine learning based on training data using multiple captured images, including captured images of the airway and captured images of other anatomical structures, to distinguish between the airway and other anatomical structures. The auxiliary image display processing unit 223 displays auxiliary images on the display screen 21 to distinguish between the airway and other anatomical structures based on the discrimination result by the discrimination processing unit 222.

Description

Technical Field

[0005]

[0001] The present invention relates to an image display system for displaying an image of the larynx in which an airway is formed on a display screen, a learning dataset used when generating a learned model used therefor, an image display program for displaying an image of the larynx on a display screen, and a diagnostic support system for supporting the diagnosis of the larynx.

Background Art

[0002] It has been proposed to provide observation support by taking and displaying an image of a target site during an examination such as an endoscopic examination (see, for example, Patent Document 1 below). When performing tracheal intubation, an image of the larynx is taken using a laryngoscope. At this time, it is common practice to perform tracheal intubation while displaying the taken image of the larynx on a display screen in real time.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When the patient is a child, especially an infant under 1 year old, the time available for tracheal intubation is short, the glottis is underdeveloped, and only the bone structure may be visible, making tracheal intubation particularly difficult. Therefore, when images of the airway and esophagus are displayed on the display screen, there is a risk of accidentally inserting the tube into the esophagus, and an image display system that can perform tracheal intubation accurately and quickly is desired.

[0005] Furthermore, challenges during tracheal intubation are not limited to infants under one year old; they can also arise in children over one year old and in adults. For example, if the esophagus can be identified using the epiglottis as a reference point, the laryngoscope can be guided from the epiglottis to the esophagus, allowing for accurate and rapid tracheal intubation.

[0006] This invention has been made in view of the above circumstances, and aims to provide an image display system, a learning dataset, an image display program, and a diagnostic support system that enable accurate and rapid tracheal intubation. [Means for solving the problem]

[0007] (1) The image display system according to the present invention is an image display system for displaying an image of the larynx in which the airway is formed on a display screen, and comprises a captured image display processing unit, a discrimination processing unit, and an auxiliary image display processing unit. The captured image display processing unit displays the captured image on the display screen. When images of the airway and anatomical structures other than the airway are displayed on the display screen, the discrimination processing unit discriminates between the airway and anatomical structures other than the airway using a trained model generated by machine learning based on training data using multiple captured images, including captured images of the airway and captured images of anatomical structures other than the airway. The auxiliary image display processing unit displays an auxiliary image on the display screen for distinguishing between the airway and anatomical structures other than the airway based on the discrimination result by the discrimination processing unit.

[0008] With this configuration, when performing tracheal intubation, if images of the airway and other anatomical structures are displayed on the screen, a pre-trained model generated by machine learning can be used to accurately distinguish between the airway and other anatomical structures. Based on the discrimination result, auxiliary images to distinguish between the airway and other anatomical structures are displayed on the screen, allowing for accurate and rapid tracheal intubation while referring to the auxiliary images.

[0009] (2) The discrimination processing unit may, when images of the airway and esophagus are displayed on the display screen, use the trained model to distinguish between the airway and the esophagus. In this case, the auxiliary image display processing unit may display an auxiliary image on the display screen to distinguish between the airway and the esophagus based on the discrimination result by the discrimination processing unit.

[0010] With this configuration, the trachea and esophagus can be distinguished and auxiliary images displayed on the screen. Therefore, the risk of accidentally intubating the esophagus is reduced, and tracheal intubation can be performed more accurately and quickly.

[0011] (3) The discrimination processing unit may, when images of the airway and epiglottis are displayed on the display screen, use the trained model to distinguish between the airway and the epiglottis. In this case, the auxiliary image display processing unit may display an auxiliary image on the display screen to distinguish between the airway and the epiglottis based on the discrimination result by the discrimination processing unit.

[0012] With this configuration, the trachea and epiglottis can be identified and auxiliary images can be displayed on the screen. Therefore, the laryngoscope can be guided from the epiglottis to the esophagus, allowing for more accurate and rapid tracheal intubation.

[0013] (4) The auxiliary image may include an airway symbol image that is displayed in correspondence with the airway image displayed on the display screen.

[0014] With this configuration, the airways can be clearly distinguished using symbolic images, allowing for more accurate and rapid tracheal intubation.

[0015] (5) The auxiliary image may include a vocal cord symbol image that is displayed in correspondence with the vocal cord image displayed on the display screen.

[0016] With this configuration, the vocal cords can be clearly distinguished by the symbolic image of the vocal cords, allowing for more accurate and rapid tracheal intubation.

[0017] (6) The training dataset according to the present invention is a training dataset used to generate a trained model used in the image display system, and includes a plurality of images, including images of the airway and images of anatomical structures other than the airway.

[0018] With this configuration, a trained model can be generated using machine learning with a training dataset, and this trained model can be used to accurately distinguish between the airway and other anatomical structures.

[0019] (7) The image display program according to the present invention is an image display program for displaying an image of the larynx that forms the airway on a display screen, and causes a computer to function as an image capture display processing unit, a discrimination processing unit, and an auxiliary image display processing unit. The image capture display processing unit displays the captured image on a display screen. When images of the airway and anatomical structures other than the airway are displayed on the display screen, the discrimination processing unit discriminates between the airway and anatomical structures other than the airway using a trained model generated by machine learning based on training data using multiple captured images, including captured images of the airway and captured images of anatomical structures other than the airway. The auxiliary image display processing unit displays an auxiliary image on the display screen to distinguish between the airway and anatomical structures other than the airway based on the discrimination result by the discrimination processing unit.

[0020] With this configuration, an image display program is used to display auxiliary images on the screen to distinguish between the airway and other anatomical structures, allowing for accurate and rapid tracheal intubation while referring to the auxiliary images.

[0021] (8) The diagnostic support system according to the present invention is a diagnostic support system for supporting the diagnosis of the larynx, comprising an imaging device for imaging the larynx and an image display system for displaying an image of the larynx captured by the imaging device on a display screen.

[0022] According to such a configuration, when diagnosing the larynx, an image of the larynx captured by the imaging device is displayed on the display screen, and by performing tracheal intubation accurately and quickly while referring to the auxiliary image, the diagnosis can be supported.

Effect of the Invention

[0023] According to the present invention, tracheal intubation can be performed accurately and quickly while referring to the auxiliary image.

Brief Description of the Drawings

[0024] [Figure 1] It is a schematic diagram showing the overall configuration of a diagnostic support system according to an embodiment of the present invention. [Figure 2] It is a schematic diagram showing an example of an image of the larynx displayed on the display screen. [Figure 3] It is a block diagram for explaining the specific configuration of the image display system. [Figure 4] It is a flowchart showing an example of the process when analyzing an image captured by a laryngoscope. [Figure 5] It is a diagram showing the calculation results of the ROC curve and AUC. [Figure 6] It is a diagram showing an example of an embodiment when discriminating the airway, vocal cords, and esophagus using a pre-trained model and displaying an auxiliary image on the display screen.

Mode for Carrying Out the Invention

[0025] 1. Overall Configuration of the Diagnostic Support System Figure 1 is a schematic diagram showing the overall configuration of a diagnostic support system according to one embodiment of the present invention. This diagnostic support system is a system for supporting the diagnosis of the larynx, and is particularly suitable for use when performing tracheal intubation.

[0026] This diagnostic support system comprises a laryngoscope 1 and a computer 2 that receives data from the laryngoscope 1. In this example, the laryngoscope 1 is connected to the computer 2 via a cable, but the transmission and reception of data between the laryngoscope 1 and the computer 2 may be wireless, not limited to wired connections.

[0027] The laryngoscope 1 is an example of an imaging device for taking images of the larynx. The laryngoscope 1 comprises a main body 11 and an insertion part 12 that are connected to each other. The operator grasps the main body 11 of the laryngoscope 1 and inserts the insertion part 12 through the mouth of the patient H. This brings the tip of the insertion part 12 of the laryngoscope 1 close to the vicinity of the larynx inside the mouth of the patient H, enabling the laryngoscope 1 to take images of the larynx.

[0028] The main unit 11 is a so-called handle and contains a circuit board and memory (neither of which are shown). Communication between the laryngoscope 1 and the computer 2 is established via a communication interface mounted on the circuit board of the main unit 11. The main unit 11 may also be provided with a display screen for displaying images taken by the laryngoscope 1 in real time.

[0029] The insertion section 12 is a so-called blade, and either a curved shape such as a Macintosh type or a straight shape such as a mirror type is selectively used. A small camera (not shown) for taking images of the larynx is attached to the tip of the insertion section 12.

[0030] Computer 2 constitutes an image display system for displaying images of the larynx captured by the laryngoscope 1 on the display screen 21. Based on the data input from the laryngoscope 1, Computer 2 displays the images captured by the laryngoscope 1 on the display screen 21. The operator can insert tube T into the trachea of ​​patient H while confirming the images of the larynx captured by the laryngoscope 1 on the display screen 21 of Computer 2.

[0031] 2. Image of the larynx Figure 2 is a schematic diagram showing an example of an image of the larynx 210 displayed on the display screen 21. The airway 211 is formed in the larynx 210. The airway 211 is formed by the vocal cords 212, epiglottis 213, and arytenoid cartilage 214, and when breathing, the vocal cords 212 are open as shown in Figure 2.

[0032] Furthermore, the esophagus 215 is formed near the airway 211, adjacent to the arytenoid cartilage 214. Normally, when the esophagus 215 is open, the vocal cords 212 close, preventing food from entering the airway 211. Thus, the airway 211 and the esophagus 215 are in close proximity, and images of both the airway 211 and the esophagus 215 may be displayed simultaneously on the display screen 21.

[0033] 3. Specific Configuration of the Image Display System Figure 3 is a block diagram illustrating the specific configuration of the image display system. In addition to the computer 2 mentioned above, this image display system is equipped with a storage unit 3 and other components.

[0034] Computer 2 is, for example, a personal computer, and can be a desktop or notebook personal computer. However, Computer 2 may also be a mobile device such as a tablet or smartphone. In other words, Computer 2 can be any device equipped with a display screen 21.

[0035] Computer 2 is equipped with a control unit 22 and a display unit 23. The control unit 22 is equipped with a processor such as a CPU (Central Processing Unit), and the operation of computer 2 can be controlled by the processor executing a program. The display unit 23 is composed of, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display, and is equipped with a display screen 21 for displaying images. Note that the display unit 23 is not limited to being equipped with computer 2, but may be provided separately from computer 2.

[0036] The control unit 22 is connected to the display unit 23, as well as the laryngoscope 1 and the memory unit 3. These connections may be wired or wireless. The control unit 22 functions as an image display processing unit 221, a discrimination processing unit 222, and an auxiliary image display processing unit 223, etc., when the processor executes a program (image display program).

[0037] The storage unit 3 may be an external memory connected to the computer 2 by wired or wireless connection, or a cloud server connected to the computer 2 via the internet. Alternatively, the storage unit 3 may consist of an HDD (Hard Disk Drive) or SSD (Solid State Drive), or other memory built into the computer 2.

[0038] The image capture and display processing unit 221 performs processing to display the image captured by the laryngoscope 1 on the display screen 21 of the display unit 23. In other words, the image capture and display processing unit 221 displays the image data input from the laryngoscope 1 on the display screen 21 in real time, enabling the operator to perform the work while checking the image displayed on the display screen 21. The image displayed on the display screen 21 is preferably a video displayed in real time, but still images captured at regular intervals may be switched and displayed as time progresses.

[0039] The discrimination processing unit 222 performs processing to distinguish between the airway 211 and the esophagus 215 when images of the airway 211 and esophagus 215 are displayed on the display screen 21 as shown in Figure 2. At this time, the discrimination processing unit 222 distinguishes between the airway 211 and the esophagus 215 using a trained model 31 generated by machine learning. This trained model 31 is generated by machine learning based on training data using multiple images, including images of the airway 211 and images of the esophagus 215.

[0040] The auxiliary image display processing unit 223 performs processing to display an auxiliary image on the display screen 21 to distinguish between the airway 211 and the esophagus 215, based on the discrimination result from the discrimination processing unit 222. This allows the operator to perform tracheal intubation accurately and quickly while referring to the auxiliary image. In particular, because the trachea 211 and the esophagus 215 can be distinguished and the auxiliary image displayed on the display screen, the risk of mistakenly intubating into the esophagus 215 is reduced, and tracheal intubation can be performed more accurately and quickly.

[0041] The pre-trained model 31 used for discrimination by the discrimination processing unit 222 is stored in the memory unit 3 beforehand. This pre-trained model 31 is generated by a general-purpose learning algorithm 4, such as a learning algorithm for object detection like YOLO. The training dataset 5 used to generate the pre-trained model includes multiple images, including images of the airway 211 (airway image 51) and images of the esophagus 215 (esophageal image 52).

[0042] 4. Processing during image analysis Figure 4 is a flowchart showing an example of the processing involved when analyzing images captured by the laryngoscope 1. When diagnosing the larynx using the laryngoscope 1, the images (videos) captured by the laryngoscope 1 are continuously analyzed (step S101) until the diagnosis is complete (until Yes is indicated in step S108), and the processing in steps S102 to S107 is performed according to the results of that analysis.

[0043] In this example, when images of the airway 211 and esophagus 215 are displayed on the display screen 21, not only the airway 211 and esophagus 215 but also the vocal cords 212 that form the entrance to the airway 211 are identified. That is, the trained model 31 used for identification by the identification processing unit 222 is generated by machine learning based on training data using multiple images, including images of the airway 211, images of the esophagus 215, and images of the vocal cords 212.

[0044] Image analysis using such a trained model 31 allows for the identification of the airway 211, vocal cords 212, and esophagus 215 included in the captured image. For the image of the airway 211 displayed on the display screen 21 (Yes in step S102), an airway symbol image is displayed as a corresponding auxiliary image (step S103). For the image of the vocal cords 212 (Yes in step S104), a vocal cord symbol image is displayed as a corresponding auxiliary image (step S105). For the image of the esophagus 215 (Yes in step S106), an esophageal symbol image is displayed as a corresponding auxiliary image (step S107).

[0045] However, the system may be configured such that the symbol image for the vocal cords is not displayed, and only the symbol images for the airway and esophagus are displayed, or it may be configured such that only the symbol image for the airway is displayed.

[0046] 5. Examples The following describes an example of distinguishing between the airway 211, vocal cords 212, and esophagus 215. To generate a trained model, 1179 still images were extracted from 653 cases of infants under general anesthesia, and the images were prepared in nine variations. Specifically, the training dataset consisted of 653 images of the airway 211 immediately before intubation, 335 images of the arytenoid cartilage 214, 139 images of the esophagus 215, and 52 images of other anatomical structures, totaling 1179 images.

[0047] For learning algorithm 4, YOLOv8 was used, and the training dataset was prepared by splitting the training data into three groups: train data, validation data, and test data, in a ratio of 6.4:1.6:2. The number of divisions of the training dataset is shown in Table 1 below, and the YOLOv8 training conditions are shown in Table 2 below. A1-A5 are images of the airway 211 immediately before intubation (653 images in total), B1-B2 are images of the arytenoid cartilage 214 (335 images in total), C is images of the esophagus 215 (139 images), and D is images of other anatomical structures (52 images).

[0048] [Table 1]

[0049] [Table 2] (For information on "Rand Augmentation," please refer to Proc of CVPR Workshops 2020, pp. 702-703.)

[0050] The evaluation method involved calculating recall (Re), precision (Pr), F-score, mean average precision (mAP), and mAP50 to assess the classification results. Furthermore, accuracy (Ac), sensitivity (Sn), and specificity (Sp) were calculated to evaluate the detection results. A confidence score of 0.5 was used as the baseline, and the TP (True-Positive), FP (False-Positive), and FN (False-Negative) judgments were calculated according to Table 3 below, based on the combination of ground truth, prediction, and IoU (Intersection over Union).

[0051] [Table 3]

[0052] Furthermore, as an evaluation of the detection results, a Receiver Operating Characteristic (ROC) curve was plotted and the Area Under the Curve (AUC) was calculated. Figure 5 shows the ROC curve and the calculated AUC.

[0053] The evaluation of the classification results is shown in Table 4 below. The evaluation of the detection results is shown in Table 5 below. In Tables 4 and 5, vc represents the vocal cords, aw represents the airway, es represents the esophagus, and All represents the overall average value.

[0054] [Table 4]

[0055] [Table 5]

[0056] First, according to the evaluation results in Table 4, the vocal cords (VC) and airway (AW) could be classified with an accuracy of over 90%, and were higher than the esophagus (ES) in all evaluation indices.

[0057] According to the evaluation results for precision (Ac), sensitivity (Sn), and specificity (Sp) in Table 5, the vocal cords (vc) and airway (aw) also showed high rectangular detection accuracy, and their detection accuracy was higher than that of the esophagus (es).

[0058] According to the AUC in Table 5 and the evaluation results in Figure 5, the vocal cords (vc) and airway (aw) had higher AUCs and superior ROC curves than the esophagus (es).

[0059] As shown in the evaluation results above, in this example, the detection accuracy of the vocal cords (vc) and airway (aw) was high, and since it was possible to identify not only the airway (aw) but also the vocal cords (vc), it was found that there is a possibility of creating an AI that can also detect the arytenoid cartilage. Furthermore, in this example, the detection accuracy of the esophagus (es) was lower compared to the vocal cords (vc) and airway (aw), but this is thought to be due to the difference in the number of images included in the training dataset. Although it is possible to distinguish between the vocal cords (vc), airway (aw), and esophagus (es), it was also found that there is a possibility to further improve the detection accuracy of the esophagus (es).

[0060] Figure 6 shows an example in which the airway 211, vocal cords 212, and esophagus 215 are identified using a trained model, and auxiliary images 230 are displayed on the display screen 21. In this example, the airway symbol image 231, the vocal cord symbol image 232, and the esophagus symbol image 235 are displayed on the display screen 21 as auxiliary images 230 through image analysis as illustrated in Figure 4.

[0061] In other words, when images of the airway 211 and esophagus 215 are displayed on the display screen 21 as shown in Figure 6, the airway symbol image 231 is displayed corresponding to the image of the airway 211, the vocal cord symbol image 232 is displayed corresponding to the image of the vocal cords 212, and the esophagus symbol image 235 is displayed corresponding to the image of the esophagus 215.

[0062] In this example, the airway symbol image 231 is displayed to surround an area that includes not only the airway 211 but also the epiglottis 213 and the arytenoid cartilage 214. The vocal cord symbol image 232 is displayed to surround the vocal cords 212 within the area surrounded by the airway symbol image 231. The esophagus symbol image 235 is displayed to surround the esophagus 215. Thus, in this invention, "airway" may include the airway 211 and the epiglottis 213 and arytenoid cartilage 214 that constitute the airway 211, or it may mean the airway 211 itself or the vocal cords 212 that form the entrance to the airway 211.

[0063] However, each auxiliary image 230 is not limited to a symbol image displayed in a manner that surrounds the target anatomical structure. In other words, each auxiliary image 230 can be any symbol image that clearly indicates the location of the target anatomical structure to the operator, and is not limited to a rectangular symbol image; any symbol image such as a circle, ellipse, or cross can be used as an auxiliary image 230.

[0064] 6. Variations In the above embodiment, when images of the airway 211 and esophagus 215 are displayed on the display screen 21, the airway 211 and esophagus 215 are distinguished, and based on the discrimination result, an auxiliary image 230 for distinguishing between the airway 211 and esophagus 215 is displayed on the display screen 21. However, the configuration is not limited to this, and any configuration that distinguishes between the airway 211 and anatomical structures other than the airway 211, and based on the discrimination result, an auxiliary image 230 for distinguishing between the airway 211 and anatomical structures other than the airway 211 is displayed on the display screen 21 is acceptable.

[0065] In this case, the trained model used to distinguish between the airway 211 and other anatomical structures may be generated by machine learning based on training data using multiple images, including images of the airway 211 and images of other anatomical structures. For example, by using a trained model generated by machine learning based on training data using multiple images, including images of the airway 211 and images of the epiglottis 213, it is possible to distinguish between the airway 211 and the epiglottis 213.

[0066] In other words, when images of the airway 211 and epiglottis 213 are displayed on the display screen 21, the discrimination processing unit 222 may use the trained model generated as described above to distinguish between the airway 211 and the epiglottis 213. In this case, the auxiliary image display processing unit 223 may display an auxiliary image 230 on the display screen 21 to distinguish between the airway 211 and the epiglottis 213 based on the discrimination result by the discrimination processing unit 222.

[0067] In this case, the display screen 21 may be configured to show instructions on how to move the laryngoscope 1 from the epiglottis 213 to the airway 211 (or the vocal cords 212 which form the entrance to the airway 211). [Explanation of Symbols]

[0068] 1 Laryngoscope 2 Computers 3 Storage section 4. Learning Algorithms 5. Training dataset 11 Main body 12 Insertion part 21 Display screen 22 Control Unit 23 Display section 51 Airway images 52 Esophageal images 210 Larynx 211 Airway 212 Vocal cords 213 Epiglottis 214 Arytenoid cartilage 215 Esophagus 221 Image display processing unit 222 Discriminant Processing Unit 223 Auxiliary Image Display Processing Unit 230 auxiliary images 231 Airway Symbol Images 232 Vocal Cord Symbol Images 235 Esophageal Symbol Image

Claims

1. An image display system for displaying an image of the larynx in which the airway is formed on a display screen, A captured image display processing unit that displays the captured image on a screen, When images of the airway and other anatomical structures are displayed on the aforementioned display screen, a discrimination processing unit is provided that uses a trained model generated by machine learning based on training data using multiple images, including images of the airway and images of other anatomical structures, to distinguish between the airway and other anatomical structures. An image display system comprising an auxiliary image display processing unit that displays auxiliary images on the display screen for distinguishing between the airway and anatomical structures other than the airway based on the discrimination result of the discrimination processing unit.

2. When images of the airway and esophagus are displayed on the display screen, the discrimination processing unit uses the trained model to distinguish between the airway and esophagus. The image display system according to claim 1, wherein the auxiliary image display processing unit displays an auxiliary image on the display screen for distinguishing between the airway and the esophagus based on the discrimination result by the discrimination processing unit.

3. When images of the airway and epiglottis are displayed on the display screen, the discrimination processing unit uses the trained model to discriminate between the airway and epiglottis. The image display system according to claim 1, wherein the auxiliary image display processing unit displays an auxiliary image for distinguishing the airway and the epiglottis on the display screen based on the discrimination result by the discrimination processing unit.

4. The image display system according to claim 2 or 3, wherein the auxiliary image includes an airway symbol image that is displayed in correspondence with the airway image displayed on the display screen.

5. The image display system according to claim 4, wherein the auxiliary image includes a symbol image for vocal cords that is displayed in correspondence with the image of vocal cords displayed on the display screen.

6. A training dataset used when generating a trained model for use in the image display system described in claim 1, A training dataset containing multiple images, including images of the airway and images of anatomical structures other than the airway.

7. An image display program for displaying an image of the larynx, which forms the airway, on a display screen. A captured image display processing unit that displays the captured image on a screen, When images of the airway and other anatomical structures are displayed on the aforementioned display screen, a discrimination processing unit is provided that uses a trained model generated by machine learning based on training data using multiple images, including images of the airway and images of other anatomical structures, to distinguish between the airway and other anatomical structures. An image display program that causes a computer to function as an auxiliary image display processing unit, which displays auxiliary images on the display screen for distinguishing between the airway and anatomical structures other than the airway, based on the discrimination results of the discrimination processing unit.

8. A diagnostic support system for assisting in the diagnosis of the larynx, A imaging device for taking images of the larynx, A diagnostic support system comprising: an image display system according to claim 1 for displaying an image of the larynx captured by the aforementioned imaging device on a display screen;

Images