Information processing device, information processing method, and computer program

Abstract

To provide an information processing apparatus, an information processing method, a program, and a recording medium that reduces patient burden, suppresses narrowing of a field of view during specular surgery, and detects lung leaks with high accuracy.SOLUTION: An information processing apparatus for detecting lung leaks from an image of a lung includes a target image acquisition unit that acquires a target image which is the image of the lung, a model acquisition unit that acquires a lung leak detection model, and a detection execution unit. The lung leak detection model is generated by a deep learning by using training data in which a plurality of lung images, lung leak information indicating a lung leak site in each of the plurality of images of the lung are associated with each other. The detection execution unit executes lung leak detection that specifies the lung leak site in the target image by using the target image and the lung leak detection model, and outputs a result of the lung leak detection.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology disclosed in this specification relates to an information processing device or the like for detecting pulmonary leakage from an image of the lungs. [Background technology]

[0002] For example, one of the complications of pneumothorax or lung resection is pulmonary leakage (pulmonary fistula). Pneumonitis occurs when the surface of the lung is damaged, causing air to leak from the lung parenchyma into the thoracic cavity. There are various methods for treating pneumonitis, including surgery, but identifying the site of the pneumonitis is the first step in treatment.

[0003] Conventionally, lung leak sites have been identified by leak tests, which involve injecting water (saline) into the thoracic cavity, then pumping air into the lungs to inflate them, and then searching for air bubbles leaking from the lung surface to identify the lung leak site (see, for example, Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Kazuhiro Ueda, "Effectiveness of pulmonary fistula closure using absorbable mesh and fibrin adhesive in thoracoscopically assisted lung cancer surgery," Yamaguchi Medical Journal, 2010, Vol. 59, No. 4, pp. 161-165 Summary of the Invention [Problem to be solved by the invention]

[0005] Leak testing to identify the site of a lung leak must be performed under general anesthesia, which places a significant burden on the patient. In addition, depending on how pressure is applied, lung leaks may not be detected, so there is room for improvement in the accuracy of lung leak detection. Furthermore, because the lungs need to be inflated, it is difficult to ensure a clear field of view during laparoscopic surgery.

[0006] This specification discloses a technique that can solve the above-mentioned problems. [Means for solving the problem]

[0007] The technology disclosed in this specification can be realized, for example, in the following forms.

[0008] (1) The information processing device disclosed in this specification is an information processing device for detecting pulmonary leaks from images of the lungs, and includes a target image acquisition unit, a model acquisition unit, and a detection execution unit. The target image acquisition unit acquires a target image that is an image of the lungs. The model acquisition unit acquires a pulmonary leak detection model. The pulmonary leak detection model is a model generated by deep learning using training data in which multiple lung images are associated with pulmonary leak information indicating the pulmonary leak site in each of the multiple lung images. The detection execution unit uses the target image and the pulmonary leak detection model to perform pulmonary leak detection to identify the pulmonary leak site in the target image, and outputs the results of the pulmonary leak detection.

[0009] This information processing device can perform lung leak detection by identifying the lung leak site in the target image using a target image of the lungs and a lung leak detection model generated by deep learning, and output the lung leak detection result. In other words, this information processing device can achieve lung leak detection by simply acquiring the target image of the lungs, without performing any other procedures on the lungs. Therefore, this information processing device can detect lung leaks using target images generated and acquired by imaging with a thoracoscope under local anesthesia, without requiring general anesthesia as in conventional lung leak detection using leak tests, thereby reducing the burden on the patient. Furthermore, this information processing device does not require the application of pressure to the lungs as in conventional lung leak detection using leak tests, thereby preventing missed lung leak detections due to the way pressure is applied and improving lung leak detection accuracy. Furthermore, this information processing device does not require the lungs to be inflated as in conventional lung leak detection using leak tests, and can detect lung leaks using target images generated and acquired by imaging the lungs in a collapsed state, thereby preventing the field of view from becoming narrower during endoscopic surgery.

[0010] Machine learning is sometimes used for diagnosis in pathological screening using X-ray images and for detecting lesions in endoscopic camera images of the digestive tract. However, such machine learning-based diagnosis automates diagnosis that was previously performed by experts visually inspecting images. Meanwhile, as described above, lung leak detection has consistently been performed through leak testing, and the technical concept of detecting lung leaks from lung images is unknown. Through extensive research, the present inventors focused on the technical concept of detecting lung leaks from lung images and further developed specific means for detecting lung leaks from lung images, thereby completing the present invention. Thus, the present invention utilizes the technical concept of detecting lung leaks from lung images, which was not anticipated in the prior art and could not have been easily arrived at from the prior art.

[0011] (2) The information processing device may further include a training data acquisition unit that acquires the training data, and the model acquisition unit may acquire the pulmonary leak detection model by creating the pulmonary leak detection model through deep learning using the training data. This information processing device can acquire the pulmonary leak detection model without using another device, and can use the model to detect pulmonary leaks from target images.

[0012] (3) In the information processing device, the pulmonary leak information may include information indicating the degree of pulmonary leak, and the pulmonary leak detection performed by the detection execution unit may include processing to identify the degree of pulmonary leak at each pulmonary leak site in the target image. With this information processing device, the degree of pulmonary leak at the pulmonary leak site can be determined simply by acquiring the target image, and more useful information for treating pulmonary leak can be obtained.

[0013] (4) In the information processing device, the target image acquisition unit may be configured to acquire, in real time, a moving image of the lungs captured by an imaging device as the target image. This information processing device can perform pulmonary leak detection in real time while capturing a moving image of the patient's lungs, thereby enabling efficient diagnosis and treatment of pulmonary leak.

[0014] The technology disclosed in this specification can be realized in various forms, such as an information processing device, an information processing method, a computer program that realizes those methods, a non-transitory recording medium on which that computer program is recorded, etc. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is an explanatory diagram showing a schematic configuration of an information processing device 100 according to an embodiment of the present invention. [Figure 2] 1 is a flowchart showing a pulmonary leak detection model acquisition process according to an embodiment of the present invention; [Figure 3] An explanatory diagram schematically illustrating an example of training data TD. [Figure 4] A conceptual diagram of an example of a lung leak detection model MO. [Figure 5] 1 is a flowchart showing a pulmonary leak detection process according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0016] A. Implementation: A-1. Configuration of information processing device 100: 1 is an explanatory diagram showing a schematic configuration of an information processing device 100 in this embodiment. The information processing device 100 is a computer (PC, server, etc.) for detecting pulmonary leakage from a lung image. In this specification, detecting pulmonary leakage from a lung image means identifying the site of pulmonary leakage in the lung represented by the image. In this embodiment, detecting pulmonary leakage from a lung image also includes identifying the degree of pulmonary leakage.

[0017] The information processing device 100 includes a control unit 110, a storage unit 120, a display unit 130, an operation input unit 140, and an interface unit 150. These units are connected to each other via a bus 190 so as to be able to communicate with each other.

[0018] The display unit 130 of the information processing device 100 is configured, for example, by a liquid crystal display or the like, and displays various images and information. The operation input unit 140 is configured, for example, by a keyboard, mouse, buttons, microphone, etc., and accepts operations and instructions from an administrator. The display unit 130 may also function as the operation input unit 140 by being equipped with a touch panel. The interface unit 150 is configured, for example, by a LAN interface, a USB interface, etc., and communicates with other devices via wired or wireless connections.

[0019] The storage unit 120 of the information processing device 100 is configured with, for example, a ROM, RAM, a hard disk drive (HDD), etc., and is used to store various programs and data, and as a work area when various programs are executed, and as a temporary storage area for data. For example, the storage unit 120 stores a pulmonary leak detection program CP, which is a computer program for executing the pulmonary leak detection model acquisition process and the pulmonary leak detection process described below. The pulmonary leak detection program CP is provided in a state stored on a computer-readable recording medium (not shown), such as a CD-ROM, DVD-ROM, or USB memory, and is stored in the storage unit 120 by installing it in the information processing device 100.

[0020] Furthermore, during the execution of the pulmonary leak detection model acquisition process and pulmonary leak detection process described below, training data TD, pulmonary leak detection model MO, target image Io, and pulmonary leak detection result data RD are stored in storage unit 120 of information processing device 100. The contents of this information will be described in conjunction with the explanation of the pulmonary leak detection model acquisition process and pulmonary leak detection process described below.

[0021] Control unit 110 of information processing device 100 is configured with, for example, a CPU, and controls the operation of information processing device 100 by executing a computer program read from storage unit 120. For example, control unit 110 reads and executes pulmonary leak detection program CP from storage unit 120, thereby functioning as training data acquisition unit 112, model acquisition unit 114, target image acquisition unit 115, and detection execution unit 116, all of which are used to execute the pulmonary leak detection model acquisition process and pulmonary leak detection process described below. The functions of each of these units will be described in conjunction with the explanation of the pulmonary leak detection model acquisition process and pulmonary leak detection process described below.

[0022] A-2. Lung leak detection model acquisition process: Next, a pulmonary leak detection model acquisition process executed by information processing device 100 of this embodiment will be described. FIG. 2 is a flowchart showing the pulmonary leak detection model acquisition process of this embodiment. The pulmonary leak detection model acquisition process is a process for acquiring a pulmonary leak detection model MO, which is a learning model used to detect pulmonary leaks from lung images. In this embodiment, information processing device 100 acquires the pulmonary leak detection model MO by creating the pulmonary leak detection model MO through predetermined deep learning. The pulmonary leak detection model acquisition process is started in response to a start command being input by the user operating operation input unit 140 of information processing device 100.

[0023] In the pulmonary leak detection model acquisition process, first, training data acquisition unit 112 (FIG. 1) of information processing device 100 acquires training data TD (S110). Training data TD is acquired via interface unit 150 and stored in storage unit 120.

[0024] 3 is an explanatory diagram schematically illustrating an example of training data TD. The training data TD is data in which training image data It(N), which are multiple images of the lungs generated by imaging the lungs, are associated with pulmonary leakage information indicating the pulmonary leakage site and the pulmonary leakage level in the lungs represented by each training image data It(N). That is, as shown in FIG. 3, the training data TD includes, for each training image data It(N), information specifying a rectangular region surrounding the pulmonary leakage site (hereinafter referred to as a "pulmonary leakage region Rx") (e.g., information specifying the coordinates of the upper left and lower right points of the pulmonary leakage region Rx) and information specifying one of multiple preset levels of pulmonary leakage level (e.g., three levels: the most severe "heavy," the intermediate "medium," and the mildest "mild"). In the example of Figure 3, among the multiple training image data It(N) that make up the training data TD, the first training image data It(1) is an image of lungs with pulmonary leakage, and a pulmonary leakage region Rx is identified on the training image data It(1), and the degree of pulmonary leakage is identified as "severe." Note that for an image of lungs without pulmonary leakage, such as the fourth training image data It(4), the pulmonary leakage location and the degree of pulmonary leakage are identified as "none" in the pulmonary leakage information.

[0025] Among the training data TD, images of lungs with lung leakage can be obtained, for example, by photographing lungs confirmed to have lung leakage by a conventional leak test. Furthermore, lung leakage information (information indicating the location and extent of lung leakage) for images of lungs with lung leakage can be identified based on the results of the leak test. Furthermore, images of lungs without lung leakage can be obtained, for example, by photographing lungs confirmed to have no lung leakage by a conventional leak test. The lung images (training image data It(N)) constituting the training data TD include multiple images generated by photographing the lungs of multiple people. Furthermore, the lung images constituting the training data TD may include multiple images generated by photographing the lungs of the same person multiple times.

[0026] Next, the model acquisition unit 114 (FIG. 1) of the information processing device 100 creates a pulmonary leak detection model MO through predetermined deep learning using the training data TD (S120). FIG. 4 is an explanatory diagram conceptually showing an example of the pulmonary leak detection model MO. As shown in FIG. 4, the pulmonary leak detection model MO is a model for detecting pulmonary leaks from a target image Io generated by photographing the lungs. In other words, the pulmonary leak detection model MO is a model for identifying a pulmonary leak region Rx, which is the site of pulmonary leak, from the target image Io, and for identifying the level of pulmonary leak.

[0027] The model acquisition unit 114 uses the training image data It(N) included in the training data TD as input data, and uses the lung leakage information (information indicating the lung leakage site and the degree of lung leakage) associated with each training image data It(N) as a response variable, and calculates a predetermined evaluation index (for example, root mean square error (RMSE), mean absolute error (MAE), coefficient of determination (R 2 )) to create a lung leak detection model MO. Various known deep learning algorithms can be used to create the lung leak detection model MO, but for example, YOLO, a real-time object detection algorithm, can be used. The created lung leak detection model MO is stored in the memory unit 120 of the information processing device 100. The above steps complete the lung leak detection model acquisition process for acquiring the lung leak detection model MO.

[0028] A-3. Lung leak detection process: Next, the pulmonary leak detection process executed by the information processing device 100 of this embodiment will be described. Fig. 5 is a flowchart showing the pulmonary leak detection process in this embodiment. The pulmonary leak detection process is a process for detecting pulmonary leaks from an image of the lungs. More specifically, the pulmonary leak detection process is a process for detecting pulmonary leaks from the target image Io using the target image Io generated by photographing the lungs and the above-mentioned pulmonary leak detection model MO, that is, a process for identifying the pulmonary leak region Rx, which is the pulmonary leak site, and identifying the level of pulmonary leak, and outputting the detection results. The pulmonary leak detection process is started in response to a start command being input by the user operating the operation input unit 140 of the information processing device 100.

[0029] In the pulmonary leak detection process, first, the target image acquisition unit 115 (FIG. 1) of the information processing device 100 acquires a target image Io, which is an image generated by imaging the lungs (S210). The target image Io is, for example, an image generated by imaging using a thoracoscope under local anesthesia, acquired via the interface unit 150, and stored in the storage unit 120. The target image Io may be one or more still images, or may be a moving image composed of multiple frames. The target image acquisition unit 115 may perform predetermined processing (e.g., trimming) on ​​the acquired target image Io so that it is in a format suitable for input to the pulmonary leak detection model MO.

[0030] Next, the detection execution unit 116 (FIG. 1) of the information processing device 100 performs pulmonary leak detection from the target image Io using the target image Io acquired in S210 and the pulmonary leak detection model MO described above (S220). As shown in FIG. 4, the detection execution unit 116 inputs data of the target image Io into the pulmonary leak detection model MO and acquires pulmonary leak information (information specifying a pulmonary leak region Rx, which is the pulmonary leak site, and information specifying the level of pulmonary leak) output from the pulmonary leak detection model MO, thereby achieving pulmonary leak detection. For example, in the example shown in FIG. 4, the pulmonary leak region Rx is specified in the target image Io, and the level of pulmonary leak is specified as "medium." The detection execution unit 116 generates pulmonary leak detection result data RD, which is information indicating the pulmonary leak detection result, and stores it in the storage unit 120 of the information processing device 100.

[0031] Next, the detection execution unit 116 of the information processing device 100 outputs the pulmonary leak detection result based on the pulmonary leak detection result data RD (S230). For example, the detection execution unit 116 causes the display unit 130 to display the pulmonary leak detection result. At this time, for example, the target image Io is displayed on the display unit 130, and a rectangular line indicating the pulmonary leak region Rx is displayed on the target image Io. The level of pulmonary leak at each pulmonary leak site is also displayed using text. Note that the method of displaying the level of pulmonary leak is not limited to displaying using text, and other display methods such as displaying using colors can also be used. Furthermore, if multiple pulmonary leak regions Rx are detected, the level of pulmonary leak at each pulmonary leak site is displayed. This display allows the user of the information processing device 100 to understand the pulmonary leak detection result in the target image Io.

[0032] In the pulmonary leak detection process, the target image acquisition unit 115 may acquire a moving image of the lungs captured by an imaging device as the target image Io in real time. The detection execution unit 116 may perform pulmonary leak detection in real time using the target image Io and output the pulmonary leak detection result.

[0033] A-4. Advantages of this embodiment: As described above, the information processing device 100 of this embodiment includes a target image acquisition unit 115, a model acquisition unit 114, and a detection execution unit 116. The target image acquisition unit 115 acquires a target image Io, which is an image of the lungs. The model acquisition unit 114 acquires a pulmonary leak detection model MO. The pulmonary leak detection model MO is a model generated by deep learning using training data TD in which training image data It(N), which are multiple images of the lungs, are associated with pulmonary leak information indicating the pulmonary leak site in each training image data It(N). The detection execution unit 116 uses the target image Io and the pulmonary leak detection model MO to perform pulmonary leak detection to identify the pulmonary leak site in the target image Io, and outputs the pulmonary leak detection results.

[0034] As described above, the information processing device 100 of this embodiment can perform lung leak detection to identify the lung leak site in the target image Io using the target image Io, which is an image of the lungs, and the lung leak detection model MO generated by deep learning, and output the lung leak detection results. In other words, the information processing device 100 of this embodiment can achieve lung leak detection by simply acquiring the target image Io, which is an image of the lungs, without performing any other procedures on the lungs. Therefore, the information processing device 100 of this embodiment can detect lung leaks using the target image Io generated and acquired by imaging using a thoracoscope under local anesthesia, without requiring general anesthesia as in conventional lung leak detection using leak tests, thereby reducing the burden on the patient. Furthermore, the information processing device 100 of this embodiment does not require applying pressure to the lungs as in conventional lung leak detection using leak tests, thereby preventing missed lung leak detections due to the way pressure is applied, and improving lung leak detection accuracy. Furthermore, according to the information processing device 100 of this embodiment, there is no need to inflate the lungs as in conventional leak testing for lung leak detection, and lung leaks can be detected using the target image Io generated and acquired by photographing the lungs in a collapsed state, thereby preventing the field of view from becoming narrower during endoscopic surgery.

[0035] Furthermore, the information processing device 100 of this embodiment further includes a training data acquisition unit 112 that acquires training data TD. The model acquisition unit 114 acquires the pulmonary leak detection model MO by creating it through deep learning using the training data TD. Therefore, the information processing device 100 of this embodiment can acquire the pulmonary leak detection model MO without using any other device, and can use this model to perform pulmonary leak detection from the target image Io.

[0036] Furthermore, in the information processing device 100 of this embodiment, the pulmonary leak information includes information indicating the degree of pulmonary leak, and the pulmonary leak detection performed by the detection execution unit 116 includes processing to identify the degree of pulmonary leak at each pulmonary leak site in the target image Io. Therefore, according to the information processing device 100 of this embodiment, it is possible to grasp the degree of pulmonary leak at the pulmonary leak site simply by acquiring the target image Io, and to obtain more useful information for treating pulmonary leak.

[0037] Furthermore, in the information processing device 100 of this embodiment, the target image acquisition unit 115 may be configured to acquire a moving image of the lungs captured by an imaging device as the target image Io in real time. With this configuration, pulmonary leakage detection can be performed in real time while capturing a moving image of the patient's lungs, and the diagnosis and treatment of pulmonary leakage can be performed efficiently.

[0038] A-5. Working Example: An example conducted to verify the accuracy of lung leak detection by the information processing device 100 of the above-described embodiment will be described below. First, 110 images of lungs in which the presence of lung leaks was confirmed by a conventional leak test were prepared as training data TD, and used as training image data It. Each image was acquired by extracting a 416 pixel x 416 pixel still image from a moving image generated by imaging using a thoracoscope. Furthermore, for each image in which lung leaks were present, the coordinates of a rectangular lung leak region Rx indicating the lung leak site were identified based on the results of the leak test.

[0039] In addition, in this example, the lung leak detection model MO was created using the above-mentioned training data TD and YOLO, which is a real-time object detection algorithm.

[0040] Next, 17 images of lungs with lung leakage and 10 images of lungs without lung leakage were prepared as test data. Lung leakage detection was performed on each image constituting the test data using the lung leakage detection model MO. As a result, lung leakage sites were detected in the correct positions in 15 of the 17 images of lungs with lung leakage. That is, the detection rate was 88.2% (=15 / 17), confirming that there were very few missed detections. Furthermore, lung leakage sites were not detected in any of the 10 images of lungs without lung leakage. That is, the false detection rate was 0% (=0 / 10), confirming that there were very few false detections. Thus, according to this example, it was confirmed that lung leakage detection can be achieved with very high accuracy by performing lung leakage detection from lung images using the information processing device 100 of this embodiment.

[0041] B. Variations: The technology disclosed in this specification is not limited to the above-described embodiments, and can be modified in various forms without departing from the spirit thereof, for example, the following modifications are also possible.

[0042] The configuration of information processing device 100 in the above embodiment is merely an example and can be modified in various ways. Furthermore, the details of the pulmonary leak detection model acquisition process and pulmonary leak detection process in the above embodiment are merely an example and can be modified in various ways. For example, in the above embodiment, information processing device 100 acquires a pulmonary leak detection model MO by creating the pulmonary leak detection model MO, but information processing device 100 may also acquire a pulmonary leak detection model MO generated by another device. In this case, information processing device 100 does not need to have a training data acquisition unit 112.

[0043] In the above embodiment, the pulmonary leakage information includes information indicating the degree of pulmonary leakage, but the pulmonary leakage information does not necessarily have to include information indicating the degree of pulmonary leakage.

[0044] In the above-described embodiment, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software may be replaced by hardware. [Explanation of symbols]

[0045] 100: Information processing device 110: Control unit 112: Training data acquisition unit 114: Model acquisition unit 115: Target image acquisition unit 116: Detection execution unit 120: Memory unit 130: Display unit 140: Operation input unit 150: Interface unit 190: Bus CP: Lung leak detection program Io: Target image It: Training image data MO: Lung leak detection model RD: Lung leak detection result data Rx: Lung leak area TD: Training data

Claims

1. An information processing device for detecting pulmonary leakage from an image of the lungs, a target image acquisition unit that acquires a target image, which is an image of the lungs; a model acquisition unit that acquires a pulmonary leak detection model, the pulmonary leak detection model being a model generated by deep learning using training data in which a plurality of lung images are associated with pulmonary leak information indicating a pulmonary leak site in each of the plurality of lung images; a detection execution unit that executes pulmonary leak detection to identify a pulmonary leak site in the target image using the target image and the pulmonary leak detection model, and outputs a result of the pulmonary leak detection; An information processing device comprising:

2. The information processing device according to claim 1, further comprising: a training data acquisition unit that acquires the training data, The information processing device wherein the model acquisition unit acquires the pulmonary leak detection model by creating the pulmonary leak detection model through the deep learning using the training data.

3. 3. The information processing device according to claim 1, the pulmonary leakage information includes information indicating the degree of pulmonary leakage; An information processing device, wherein the pulmonary leakage detection performed by the detection execution unit includes a process of identifying the degree of pulmonary leakage at each pulmonary leakage site in the target image.

4. 4. The information processing device according to claim 1, The target image acquisition unit acquires, in real time, a moving image of the lungs captured by an imaging device as the target image.

5. An information processing method for detecting pulmonary leakage from an image of the lungs by a computer, comprising: acquiring, by the computer, a target image, the target image being an image of the lung; a step of acquiring a pulmonary leak detection model by the computer, the pulmonary leak detection model being a model generated by deep learning using training data in which a plurality of lung images are associated with pulmonary leak information indicating a pulmonary leak site in each of the plurality of lung images; a step in which the computer performs pulmonary leak detection to identify a pulmonary leak site in the target image using the target image and the pulmonary leak detection model, and outputs a result of the pulmonary leak detection; An information processing method comprising:

6. 1. A computer program for detecting pulmonary leaks from lung images, comprising: On the computer, obtaining a target image, the target image being an image of the lungs; A process for acquiring a pulmonary leak detection model, the pulmonary leak detection model being a model generated by deep learning using training data in which a plurality of lung images are associated with pulmonary leak information indicating a pulmonary leak site in each of the plurality of lung images; A process of performing pulmonary leak detection to identify a pulmonary leak site in the target image using the target image and the pulmonary leak detection model, and outputting the result of the pulmonary leak detection; A computer program that executes

Images