Information processing device, information processing method, and program

Abstract

In this information processing device according to the present invention, an image acquiring means acquires an endoscope image captured by an endoscope. A lesion diagnostic means diagnoses a lesion from the endoscope image. A surgical method inference means infers a recommended surgical method from the endoscope image and diagnostic information about the lesion. A prognostic inference means infers a state of prognosis from the endoscope image and the information about the surgical method. An output means outputs the diagnostic information, the information about the surgical method, and the state of prognosis.

Description

Information processing device, information processing method, and recording medium

[0001] The present disclosure relates to inferential processing of lesions in endoscopy.

[0002] When a lesion is discovered during an endoscopic examination, a surgical procedure needs to be planned, but because the progress (prognosis) after the procedure is unknown, it has been difficult to plan the procedure taking the prognosis into consideration.Patent Document 1 describes a method for proposing a surgical procedure for joint surgery using a trained model that has learned the relationship between the surgical procedure performed on the joint and the state of the joint after surgery.

[0003] Japanese Patent Application Laid-Open No. 2021-115188

[0004] However, even with Patent Document 1, it is not always possible to appropriately suggest treatment details or prognosis for lesions discovered by endoscopic examination.

[0005] One object of the present disclosure is to provide an information processing device capable of estimating a recommended surgical procedure and prognosis for a lesion discovered during an endoscopic examination.

[0006] In one aspect of the present disclosure, an information processing device includes: an image acquisition means for acquiring an endoscopic image captured by an endoscope; a lesion diagnosis means for diagnosing a lesion from the endoscopic image; a surgical procedure inference means for inferring a recommended surgical procedure from the endoscopic image and diagnostic information on the lesion; a prognosis inference means for inferring a prognostic state from the endoscopic image and information on the surgical procedure; and an output means for outputting the diagnostic information, information on the surgical procedure, and the prognosis.

[0007] In another aspect of the present disclosure, an information processing method includes acquiring an endoscopic image captured by an endoscope, diagnosing a lesion from the endoscopic image, inferring a recommended surgical procedure from the endoscopic image and diagnostic information on the lesion, inferring a prognosis from the endoscopic image and information on the surgical procedure, and outputting the diagnostic information, information on the surgical procedure, and the prognosis.

[0008] In yet another aspect of the present disclosure, a recording medium records a program that causes a computer to execute the following processes: acquire an endoscopic image captured by an endoscope; diagnose a lesion from the endoscopic image; infer a recommended surgical procedure from the endoscopic image and diagnostic information on the lesion; infer a prognostic state from the endoscopic image and information on the surgical procedure; and output the diagnostic information, information on the surgical procedure, and the prognostic state.

[0009] According to the present disclosure, it is possible to estimate the recommended surgical procedure and prognosis for a lesion discovered during an endoscopic examination.

[0010] 1 is a block diagram showing a schematic configuration of an endoscopic examination system. FIG. 2 is a block diagram showing the hardware configuration of an information processing device. FIG. 3 is a block diagram showing the functional configuration of an information processing device. FIG. 4 shows a learning method for a surgical procedure inference model and input / output data of the surgical procedure inference model. FIG. 5 shows a learning method for a prognosis inference model and input / output data of the prognosis inference model. FIG. 6 shows a learning method for a surgical procedure / prognosis inference model and input / output data of the surgical procedure / prognosis inference model. FIG. 7 shows an example of display by a display device. FIG. 8 shows another example of display by a display device. FIG. 9 shows another example of display by a display device. FIG. 10 is a flowchart of data output processing by an information processing device. FIG. 11 is a block diagram showing the functional configuration of a second modified example of the first embodiment. FIG. 12 shows an example of the data structure of patient information. FIG. 13 is a block diagram showing the functional configuration of an information processing device of a second embodiment. FIG. 14 is a flowchart of processing by an information processing device of a second embodiment.

[0011] Preferred embodiments of the present disclosure will be described below with reference to the drawings. <First Embodiment> [System Configuration] Fig. 1 shows a schematic configuration of an endoscopic examination system 100. When a lesion is detected during an examination (including treatment) using an endoscope, the endoscopic examination system 100 proposes a surgical procedure for the lesion and predicts the prognosis if the surgical procedure is adopted. This allows a doctor to plan a procedure taking the prognosis into consideration.

[0012] As shown in FIG. 1 , the endoscopic examination system 100 mainly includes an information processing device 1 , a display device 2 , and an endoscope 3 connected to the information processing device 1 .

[0013] The information processing device 1 acquires from the endoscope 3 an image (i.e., a moving image; hereinafter, also referred to as "endoscopic image Ic") captured by the endoscope scope 3 during an endoscopic examination, and displays on the display device 2 display data for the examiner (doctor) performing the endoscopic examination to check. Specifically, the information processing device 1 acquires a moving image of the inside of an organ captured by the endoscope scope 3 during the endoscopic examination as the endoscopic image Ic. Furthermore, when the doctor finds a lesion during the endoscopic examination, he or she operates the endoscope scope 3 to input an instruction to capture the lesion position. The information processing device 1 generates a lesion image that captures the lesion position based on the doctor's imaging instruction. Specifically, the information processing device 1 generates a lesion image, which is a still image, from the endoscopic image Ic, which is a moving image, based on the doctor's imaging instruction.

[0014] The display device 2 is a display or the like that displays a predetermined image based on a display signal supplied from the information processing device 1 .

[0015] The endoscope 3 mainly comprises an operating unit 36 ​​that allows the doctor to input instructions such as air supply, water supply, angle adjustment, and photography instructions, a flexible shaft 37 that is inserted into the subject's organ to be examined, a tip 38 that incorporates a photography unit such as a micro-imaging element, and a connection unit 39 for connecting to the information processing device 1.

[0016] The following explanation will be mainly based on the processing in an endoscopic examination of the large intestine, but the subject of examination is not limited to the large intestine, and may be the digestive tract (digestive organs) such as the stomach, esophagus, small intestine, and duodenum.

[0017] 2 shows the hardware configuration of the information processing device 1. The information processing device 1 mainly includes a processor 11, a memory 12, an interface 13, an input unit 14, a light source unit 15, a sound output unit 16, and a database (hereinafter referred to as "DB") 17. These elements are connected via a data bus 19.

[0018] The processor 11 performs predetermined processing by executing programs stored in the memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0019] The memory 12 is composed of various volatile memories used as working memories, such as RAM (Random Access Memory) and ROM (Read Only Memory), and non-volatile memories that store information necessary for processing by the information processing device 1. The memory 12 may include an external storage device such as a hard disk connected to or built into the information processing device 1, or may include a storage medium such as a removable flash memory or disk medium. The memory 12 stores programs that allow the information processing device 1 to execute each process in this embodiment.

[0020] The memory 12 also temporarily stores a series of endoscopic images Ic captured by the endoscope 3 during an endoscopic examination under the control of the processor 11. The memory 12 also temporarily stores lesion images captured during an endoscopic examination based on imaging instructions from a doctor. These images are stored in the memory 12 in association with, for example, the subject's identification information (e.g., patient ID), timestamp information, and the like.

[0021] The interface 13 performs interface operations between the information processing device 1 and an external device. For example, the interface 13 supplies the display data Id generated by the processor 11 to the display device 2. The interface 13 also supplies illumination light generated by the light source unit 15 to the endoscope 3. The interface 13 also supplies an electrical signal indicating the endoscopic video Ic supplied from the endoscope 3 to the processor 11. The interface 13 may be a communication interface such as a network adapter for wired or wireless communication with an external device, or may be a hardware interface compliant with USB (Universal Serial Bus), SATA (Serial AT Attachment), or the like.

[0022] The input unit 14 generates an input signal based on an operation by a doctor. The input unit 14 is, for example, a button, a touch panel, a remote controller, or a voice input device. The light source unit 15 generates light to be supplied to the tip 38 of the endoscope 3. The light source unit 15 may also incorporate a pump or the like for sending water or air to be supplied to the endoscope 3. The sound output unit 16 outputs sound based on the control of the processor 11.

[0023] The DB 17 stores the subject's medical record information (hereinafter also referred to as "patient information"). The DB 17 also stores endoscopic images acquired in past endoscopic examinations of the subject and lesion information. The lesion information includes lesion images and information related to the lesion (hereinafter referred to as "related information"). The DB 17 may include an external storage device such as a hard disk connected to or built into the information processing device 1, or may include a storage medium such as a removable flash memory. Instead of providing the DB 17 within the endoscopic examination system 100, the DB 17 may be provided on an external server or the like, and related information may be obtained from the server via communication.

[0024] 3 is a block diagram showing the functional configuration of the information processing device 1. Functionally, the information processing device 1 includes, in addition to the interface 13 described above, a lesion diagnosis unit 21, a surgical procedure inference unit 22, a prognosis inference unit 23, and an output unit 24.

[0025] An endoscopic video Ic is input to the information processing device 1 from the endoscope 3. The endoscopic video Ic is input to the interface 13. The interface 13 extracts frame images (hereinafter also referred to as "endoscopic images") from the input endoscopic video Ic and outputs them to the lesion diagnosis unit 21, the surgical procedure inference unit 22, and the prognosis inference unit 23. The interface 13 also outputs the input endoscopic video Ic to the output unit 24.

[0026] The lesion diagnosis unit 21 detects and diagnoses lesions based on endoscopic images input from the interface 13. Specifically, the lesion diagnosis unit 21 detects lesions from endoscopic images and diagnoses the lesions using a pre-prepared image recognition model or the like. This image recognition model is a machine learning model that has been trained in advance to detect lesions contained in endoscopic images and diagnose the lesions, and is hereinafter also referred to as a "lesion diagnosis model." Note that "diagnosing a lesion" refers to estimating the location of the lesion, the progression of the lesion, the degree of infiltration of the lesion, etc. When the lesion diagnosis unit 21 detects a lesion, it outputs information such as a timestamp and diagnostic information to the surgical procedure inference unit 22 and the output unit 24.

[0027] The surgical procedure inference unit 22 infers a recommended surgical procedure (hereinafter also referred to as a "recommended surgical procedure") based on the endoscopic image input from the interface 13 and the diagnostic information input from the lesion diagnosis unit 21. Specifically, the surgical procedure inference unit 22 infers a recommended surgical procedure from the endoscopic image and the diagnostic information using a surgical procedure inference model described below. The surgical procedure inference unit 22 outputs information such as a timestamp and the recommended surgical procedure to the prognosis inference unit 23 and the output unit 24.

[0028] The prognosis inference unit 23 estimates a prognosis based on the endoscopic image input from the interface 13 and the recommended surgical procedure input from the surgical procedure inference unit 22. Specifically, the prognosis inference unit 23 estimates a prognosis from the endoscopic image and the recommended surgical procedure using a prognosis inference model described below. The prognosis includes information such as the 5-year survival rate, the length of hospital visits, dietary restrictions, and the use of an artificial anus. The prognosis inference unit 23 outputs information such as a timestamp and the prognosis to the output unit 24.

[0029] The output unit 24 generates display data based on the endoscopic image Ic input from the interface 13, the diagnostic information input from the lesion diagnosis unit 21, the recommended surgical procedure input from the surgical procedure inference unit 22, and the prognosis input from the prognosis inference unit 23, and outputs the data to the display device 2.

[0030] In the above configuration, the interface 13 is an example of an image acquisition means, the lesion diagnosis unit 21 is an example of a lesion diagnosis means, the surgical procedure inference unit 22 is an example of a surgical procedure inference means, the prognosis inference unit 23 is an example of a prognosis inference means, and the output unit 24 is an example of an output means.

[0031] The information processing device 1 may be configured by combining the lesion diagnosis unit 21, the surgical procedure inference unit 22, and the prognosis inference unit 23 into an integrated unit. For example, the information processing device 1 may be configured by combining the surgical procedure inference unit 22 and the prognosis inference unit 23. The combination of the surgical procedure inference unit 22 and the prognosis inference unit 23 will hereinafter be referred to as the "surgical procedure / prognosis inference unit." The surgical procedure / prognosis inference unit estimates a recommended surgical procedure and a prognosis based on the endoscopic image input from the interface 13 and the diagnostic information input from the lesion diagnosis unit 21. Specifically, the surgical procedure / prognosis inference unit can estimate a recommended surgical procedure and its prognosis from the endoscopic image and the diagnostic information using a surgical procedure / prognosis inference model described below.

[0032] [Inference Model] (Surgical Procedure Inference Model) Next, the surgical procedure inference model used by the surgical procedure inference unit 22 will be described. FIG. 4(A) is a block diagram showing a learning method for the surgical procedure inference model. The surgical procedure inference model is generated by so-called supervised learning. FIG. 4(A) includes training data 410 and a training device 411. The training data 410 is data showing the relationship between a lesion image, diagnostic information about the lesion (location, progression, and degree of infiltration) and a recommended surgical procedure for that lesion. The training device 411 generates a surgical procedure inference model that has learned the relationship between the lesion image, diagnostic information about the lesion, and a recommended surgical procedure for that lesion based on the training data 410. FIG. 4(B) is a block diagram showing the relationship between input data and output data of the surgical procedure inference model. The surgical procedure inference model 412 receives a lesion image and diagnostic information about the lesion as input and outputs a recommended surgical procedure.

[0033] In the above, the lesion image and diagnostic information of the lesion are used as input, but it is also possible to generate a surgical procedure inference model that can estimate a recommended surgical procedure even if some of the input information is missing.

[0034] For example, the learning device 411 can generate a surgical procedure inference model that has learned the relationship between a lesion image and part of the diagnostic information of the lesion (the progression and infiltration level of the lesion) and a recommended surgical procedure for the lesion. In this case, the surgical procedure inference model can input the lesion image and part of the diagnostic information of the lesion (the progression and infiltration level of the lesion) and output a recommended surgical procedure.

[0035] The learning device 411 can also generate a surgical procedure inference model that has learned the relationship between part of the lesion's diagnostic information (the lesion's progression and degree of infiltration) and the recommended surgical procedure for that lesion. In this case, the surgical procedure inference model can output a recommended surgical procedure using only part of the lesion's diagnostic information (the lesion's progression and degree of infiltration). The learning device 411 can also generate a surgical procedure inference model that has learned the relationship between a lesion image and the recommended surgical procedure for that lesion. In this case, the surgical procedure inference model can output a recommended surgical procedure using only the lesion image as input.

[0036] (Prognosis inference model) Next, the prognosis inference model used by the prognosis inference unit 23 will be described. FIG. 5(A) is a block diagram showing a learning method for the prognosis inference model. The prognosis inference model is generated by so-called supervised learning. FIG. 5(A) includes learning data 420 and a learning device 421. The learning data 420 is data showing the relationship between a lesion image, a recommended surgical procedure for that lesion, and a prognosis. The learning device 421 generates a prognosis inference model that has learned the relationship between a lesion image, a recommended surgical procedure for that lesion, and a prognosis based on the learning data 420. FIG. 5(B) is a block diagram showing the relationship between input data and output data of the prognosis inference model. The prognosis inference model 422 receives a lesion image and a recommended surgical procedure for that lesion as input, and outputs a prognosis.

[0037] In the above example, the lesion image and the recommended surgical procedure for that lesion are used as input, but it is also possible to generate a prognosis inference model that can estimate a prognosis even if some input information is missing. For example, the learning device 421 can generate a prognosis inference model that has learned the relationship between the recommended surgical procedure for a lesion and the prognosis. In this case, the prognosis inference model can output a prognosis using only the recommended surgical procedure for the lesion as input.

[0038] (Surgical Procedure / Prognosis Inference Model) Next, the surgical procedure / prognosis inference model used by the surgical procedure / prognosis inference unit will be described. FIG. 6(A) is a block diagram showing a learning method for the surgical procedure / prognosis inference model. The surgical procedure / prognosis inference model is generated by so-called supervised learning. FIG. 6(A) includes learning data 430 and a learning device 431. The learning data 430 is data showing the relationship between a lesion image and diagnostic information about the lesion (location, progression, and degree of infiltration of the lesion), and the recommended surgical procedure and prognosis for the lesion. The learning device 431 generates a surgical procedure / prognosis inference model that has learned the relationship between the lesion image and diagnostic information about the lesion, and the recommended surgical procedure and prognosis based on the learning data 430. FIG. 6(B) is a block diagram showing the relationship between input data and output data of the surgical procedure / prognosis inference model. The surgical procedure / prognosis inference model 432 receives a lesion image and diagnostic information about the lesion as input, and outputs a recommended surgical procedure and prognosis.

[0039] In the above example, lesion images and diagnostic information for the lesion are used as input, but it is also possible to generate a surgical procedure / prognosis inference model that can estimate the recommended surgical procedure and prognosis even if some of the input information is missing.

[0040] For example, the learning device 431 can generate a surgical procedure / prognosis inference model that has learned the relationship between a lesion image and part of the diagnostic information of the lesion (the progression and infiltration level of the lesion) and a recommended surgical procedure and prognosis for the lesion. In this case, the surgical procedure / prognosis inference model can input the lesion image and part of the diagnostic information of the lesion (the progression and infiltration level of the lesion) and output a recommended surgical procedure and prognosis.

[0041] The learning device 431 can also generate a surgical procedure / prognosis inference model that has learned the relationship between a portion of the lesion's diagnostic information (the lesion's progression and degree of infiltration) and the recommended surgical procedure and prognosis for that lesion. In this case, the surgical procedure / prognosis inference model can estimate a recommended surgical procedure and prognosis using only a portion of the lesion's diagnostic information (the lesion's progression and degree of infiltration). The learning device 431 can also generate a surgical procedure / prognosis inference model that has learned the relationship between a lesion image and the recommended surgical procedure and prognosis for that lesion. In this case, the surgical procedure / prognosis inference model can output a recommended surgical procedure and prognosis using only the lesion image as input.

[0042] [Display Example] Next, a display example on the display device 2 will be described.

[0043] FIG. 7 is an example of a display by the display device 2. In the display example of FIG. 7, an endoscopic image 51, a lesion area 52, and related information 53 are displayed within a display area 50. The endoscopic image 51 is an endoscopic image Ic during examination. The lesion area 52 is a rectangular area surrounding the detected lesion. The related information 53 is information about the detected lesion, and includes diagnostic information, a recommended surgical procedure, and information regarding prognosis. By viewing the related information 53, a doctor can understand the recommended surgical procedure and prognosis for the detected lesion.

[0044] Fig. 8 shows another example of display by the display device 2. In the example of Fig. 8, related information 53a is superimposed on the endoscopic image 51. The diagnostic information, recommended surgical procedure, and prognosis included in the related information 53a may be displayed together or may be displayed one by one in order. For example, when the doctor presses a button on the input unit 14 once, the diagnostic information may be displayed; when the doctor presses the button on the input unit 14 twice, the recommended surgical procedure may be displayed; and when the doctor presses the button on the input unit 14 three times, the prognosis may be displayed.

[0045] FIG. 9 shows another example of display by the display device 2. This example is a display example in which multiple recommended surgical procedures are output for one lesion. In the example of FIG. 9, the related information 53b includes two types of recommended surgical procedures, and the information processing device 1 sorts and displays them in order of 5-year survival rate. The sorting rule is not limited to 5-year survival rate order, and can be set by the doctor.

[0046] 9 shows two recommended surgical procedures, but if three or more recommended surgical procedures are output for one lesion, three or more recommended surgical procedures may be displayed depending on the display space. Also, if multiple recommended surgical procedures are output for one lesion and it is not possible to display all of the recommended surgical procedures, the recommended surgical procedure with the best prognosis may be displayed preferentially, or the recommended surgical procedure that appears higher in the sorting may be displayed preferentially.

[0047] FIG. 10 shows another example of display by the display device 2. This example shows a case where the basis for outputting a recommended surgical procedure is displayed. In the example of FIG. 10, the information on which the basis for the recommended surgical procedure is based is highlighted in bold and underlined. Specifically, in the example of FIG. 10, the recommended surgical procedure is ESD, and "adenocarcinoma" included in the diagnostic information is highlighted in bold and underlined. This indicates that the basis for selecting ESD as the recommended surgical procedure is "adenocarcinoma." Note that while FIG. 10 shows the basis for the recommended surgical procedure, the basis for the diagnostic information and the basis for the prognosis may also be displayed. In this case, the basis for the diagnostic information, the basis for the recommended surgical procedure, and the basis for the prognosis may be displayed simultaneously in different display modes, or only one of the bases may be displayed based on the doctor's instructions.

[0048] [Image Display Processing] Next, the image display processing for performing the above-described display will be described. Fig. 11 is a flowchart of the image display processing by the information processing device 1. This processing is realized by the processor 11 shown in Fig. 2 executing a program prepared in advance and operating as each element shown in Fig. 3.

[0049] First, an endoscopic video Ic is input to the information processing device 1 from the endoscope 3. The endoscopic video Ic is input to the interface 13. The interface 13 extracts an endoscopic image from the input endoscopic video Ic and outputs it to the lesion diagnosis unit 21, the surgical procedure inference unit 22, and the prognosis inference unit 23. The interface 13 also outputs the input endoscopic video Ic to the output unit 24 (step S11).

[0050] Next, the lesion diagnosis unit 21 detects and diagnoses a lesion based on the endoscopic image input from the interface 13. When the lesion diagnosis unit 21 detects a lesion, it outputs information such as a timestamp and diagnostic information to the operative procedure inference unit 22 and the output unit 24 (step S12).

[0051] Next, the surgical procedure inference unit 22 infers a recommended surgical procedure based on the endoscopic image input from the interface 13 and the diagnostic information input from the lesion diagnosis unit 21. The surgical procedure inference unit 22 outputs information such as a timestamp and the recommended surgical procedure to the prognosis inference unit 23 and the output unit 24 (step S13).

[0052] Next, the prognosis inference unit 23 estimates a prognosis based on the endoscopic image input from the interface 13 and the recommended surgical procedure input from the surgical procedure inference unit 22. The prognosis inference unit 23 outputs information such as a timestamp and the prognosis to the output unit 24 (step S14).

[0053] Next, the output unit 24 generates display data based on the endoscopic image Ic input from the interface 13, the diagnostic information input from the lesion diagnosis unit 21, the recommended surgical procedure input from the surgical procedure inference unit 22, and the prognosis input from the prognosis inference unit 23, and outputs it to the display device 2 (step S15).

[0054] Next, the information processing device 1 determines whether the examination has ended (step S16). The information processing device 1 determines that the examination has ended, for example, when a doctor performs an operation to end the examination on the information processing device 1 or the endoscope 3. Alternatively, the information processing device 1 may automatically determine that the examination has ended when, through image analysis of the image captured by the endoscope 3, the captured image becomes an image outside of the organ. If it is determined that the examination has not ended (step S16: No), the processing returns to step S11. On the other hand, if it is determined that the examination has ended (step S16: Yes), the image display processing ends.

[0055] [Modifications] Next, a description will be given of modifications of the first embodiment. The following modifications can be applied to the first embodiment in appropriate combinations.

[0056] (Variation 1) In the first embodiment, the output unit 24 generates display data including diagnostic information, a recommended surgical procedure, and a prognosis, and outputs the display data to the display device 2. Instead, the output unit 24 may generate audio data including diagnostic information, a recommended surgical procedure, and a prognosis, and output the audio data to the audio output unit 16. This allows the doctor to understand the diagnostic information, the recommended surgical procedure, and the prognosis without the endoscopic image being obstructed by other displays.

[0057] (Variation 2) The lesion diagnosis unit 21, surgical procedure inference unit 22, and prognosis inference unit 23 of the first embodiment can improve the accuracy of the estimation process by taking patient information into consideration. Fig. 12 shows the functional configuration of an information processing device 1a of Variation 2. As shown in the figure, the information processing device 1a is provided with a patient information acquisition unit 25. The patient information acquisition unit 25 acquires patient information from the DB 17. Fig. 13 shows an example of the data structure of the patient information. The patient information includes information such as a patient ID, patient name, gender, age, medical history, and medication history.

[0058] Returning to FIG. 12 , the patient information acquisition unit 25 outputs the acquired patient information to the lesion diagnosis unit 21 , the surgical procedure inference unit 22 , and the prognosis inference unit 23 .

[0059] The lesion diagnosis unit 21 detects and diagnoses a lesion based on the endoscopic image input from the interface 13 and the patient information input from the patient information acquisition unit 25. For example, when the lesion diagnosis unit 21 detects a protrusion from an endoscopic image, it can estimate the possibility that the protrusion is a lesion based on whether the patient has a history of colitis or the like. The lesion diagnosis model used by the lesion diagnosis unit 21 is a trained model that has been trained in advance to detect and diagnose a lesion based on the endoscopic image and the patient information.

[0060] The surgical procedure inference unit 22 estimates a recommended surgical procedure based on the endoscopic image input from the interface 13, the diagnostic information input from the lesion diagnosis unit 21, and the patient information input from the patient information acquisition unit 25. For example, the surgical procedure inference unit 22 can estimate a recommended surgical procedure suitable for an individual patient based on the patient's age, medication history, etc. The surgical procedure inference model used by the surgical procedure inference unit 22 is a trained model that has been trained in advance to estimate a recommended surgical procedure based on the lesion image, the diagnostic information for that lesion, and the patient information.

[0061] The prognosis inference unit 23 estimates a prognosis based on the endoscopic image input from the interface 13, the recommended surgical procedure input from the surgical procedure inference unit 22, and the patient information input from the patient information acquisition unit 25. The prognosis inference model used by the prognosis inference unit 23 is a trained model that has been trained in advance to estimate a prognosis based on a lesion image, a recommended surgical procedure for that lesion, and patient information.

[0062] (Variation 3) The lesion diagnosis unit 21 diagnoses a lesion based on an endoscopic image input from the interface 13. Alternatively, the lesion diagnosis unit 21 may diagnose a lesion based on an image obtained by cropping the lesion area.

[0063] For example, when the lesion diagnosis unit 21 detects a lesion in an endoscopic image, it crops a predetermined region (lesion region) containing the lesion from the endoscopic image. Cropping refers to cutting out a portion of an image. The lesion diagnosis unit 21 then resizes the image with the cropped lesion region to a size that allows image analysis using a lesion diagnosis model. The lesion diagnosis unit 21 diagnoses the lesion based on the resized image (hereinafter also referred to as a "lesion region image"). Note that the surgical procedure inference unit 22 and the prognosis inference unit 23 may also infer a recommended surgical procedure and prognosis based on the lesion region image instead of the endoscopic image. In this way, by preprocessing the endoscopic image, the accuracy of the inference process can be improved.

[0064] (Variation 4) When there are multiple endoscopic images relating to the same lesion, the lesion diagnosis unit 21 may select the endoscopic image that best represents the lesion (hereinafter also referred to as the "champion image") and diagnose the lesion based on the champion image.

[0065] For example, when the lesion diagnosis unit 21 detects a lesion from an endoscopic image input from the interface 13, it groups multiple endoscopic images captured before and after the lesion. The lesion diagnosis unit 21 then selects a champion image from the group of grouped images. The champion image may be, for example, an image in which the lesion is largest, an image in which the lesion is most central, or an image with the best focus among the group of grouped images. The lesion diagnosis unit 21 diagnoses the lesion based on the champion image. The surgical procedure inference unit 22 and the prognosis inference unit 23 may also infer a recommended surgical procedure and prognosis based on the champion image. In this way, the use of the champion image can improve the accuracy of the inference process.

[0066] (Variation 5) The surgical procedure inference unit 22 infers a recommended surgical procedure based on diagnostic information input from the lesion diagnosis unit 21. Alternatively, the surgical procedure inference unit 22 may infer a recommended surgical procedure based on diagnostic information input by a physician. Specifically, if the diagnostic information output by the lesion diagnosis unit 21 differs from the physician's findings, the physician inputs diagnostic information based on his or her own findings to the information processing device 1 via the input unit 14. The surgical procedure inference unit 22 then infers a recommended surgical procedure based on the diagnostic information input by the physician. Note that the prognosis inference unit 23 may also infer a prognosis based on a recommended surgical procedure input by a physician, instead of a recommended surgical procedure input from the surgical procedure inference unit 22. In this way, the information processing device 1 can infer a recommended surgical procedure and a prognosis based on the physician's findings.

[0067] (Variation 6) In addition to the display device 2 of the first embodiment, a display device for a patient (hereinafter also referred to as a "patient monitor") may be provided. The patient monitor is used by the patient to view the progress of the endoscopic examination. The patient monitor basically displays only the endoscopic image, but it can also display information specified by the doctor. For example, if a lesion is detected during the endoscopic examination, the doctor can display diagnostic information, etc. on the patient monitor by performing a predetermined operation. Furthermore, the information output to the patient monitor can be controlled using predetermined conditions or a machine learning model. This allows the doctor to provide an easy-to-understand explanation to the patient.

[0068] 14 is a block diagram showing the functional configuration of an information processing device according to Embodiment 2. The information processing device 70 includes an image acquisition unit 71, a lesion diagnosis unit 72, a surgical procedure inference unit 73, a prognosis inference unit 74, and an output unit 75.

[0069] 15 is a flowchart of processing by the information processing device of the second embodiment. The image acquisition means 71 acquires an endoscopic image captured by an endoscope (step S71). The lesion diagnosis means 72 diagnoses a lesion from the endoscopic image (step S72). The surgical procedure inference means 73 infers a recommended surgical procedure from the endoscopic image and diagnostic information on the lesion (step S73). The prognosis inference means 74 infers a prognosis from the endoscopic image and information on the surgical procedure (step S74). The output means 75 outputs the diagnostic information, information on the surgical procedure, and the prognosis (step S75).

[0070] According to the information processing device 70 of the second embodiment, it is possible to estimate the recommended surgical procedure and prognosis for a lesion discovered during an endoscopic examination.

[0071] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0072] (Supplementary Note 1) An information processing device comprising: an image acquisition means for acquiring an endoscopic image captured by an endoscope; a lesion diagnosis means for diagnosing a lesion from the endoscopic image; a surgical procedure inference means for inferring a recommended surgical procedure from the endoscopic image and diagnostic information on the lesion; a prognosis inference means for inferring a prognostic state from the endoscopic image and information on the surgical procedure; and an output means for outputting the diagnostic information, information on the surgical procedure, and the prognosis.

[0073] (Supplementary Note 2) The information processing device according to Supplementary Note 1, wherein the output means generates image data based on the diagnostic information of the lesion, the information on the surgical procedure, and the prognosis, and outputs the image data to a display device.

[0074] (Appendix 3) The information processing device described in Appendix 2, wherein the output means generates display data including the basis for the diagnostic information of the lesion, the basis for inferring the surgical procedure, and the basis for inferring the prognosis state, and outputs the display data to the display device.

[0075] (Supplementary Note 4) The information processing device according to Supplementary Note 1, wherein the output means, when there are multiple recommended surgical procedures, rearranges the surgical procedures based on the prognosis.

[0076] (Appendix 5) An information processing device as described in Appendix 1, comprising a selection means for selecting a champion image that best represents the lesion from a plurality of endoscopic images corresponding to the same lesion, wherein the lesion diagnosis means diagnoses the lesion from the champion image, the surgical procedure inference means infers a recommended surgical procedure from the champion image and diagnostic information of the lesion, and the prognosis inference means infers a prognosis from the champion image and information on the surgical procedure.

[0077] (Appendix 6) An information processing device according to Appendix 1, comprising a patient information acquisition means for acquiring patient information, wherein the lesion diagnosis means diagnoses a lesion from the endoscopic image and the patient information, the surgical procedure inference means infers a recommended surgical procedure from the endoscopic image, the lesion diagnosis information, and the patient information, and the prognosis inference means infers a prognosis from the endoscopic image, the surgical procedure information, and the patient information.

[0078] (Appendix 7) An information processing method that acquires an endoscopic image taken by an endoscope, diagnoses a lesion from the endoscopic image, infers a recommended surgical procedure from the endoscopic image and diagnostic information on the lesion, infers a prognostic state from the endoscopic image and information on the surgical procedure, and outputs the diagnostic information, information on the surgical procedure, and the prognostic state.

[0079] (Appendix 8) A recording medium having recorded thereon a program that causes a computer to execute the following processes: acquire an endoscopic image taken by an endoscope; diagnose a lesion from the endoscopic image; infer a recommended surgical procedure from the endoscopic image and diagnostic information on the lesion; infer a prognostic state from the endoscopic image and information on the surgical procedure; and output the diagnostic information, information on the surgical procedure, and the prognostic state.

[0080] Although the present disclosure has been described above with reference to the embodiments and examples, the present disclosure is not limited to the above-described embodiments and examples. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure.

[0081] REFERENCE SIGNS LIST 1 Information processing device 2 Display device 3 Endoscope 11 Processor 12 Memory 13 Interface 17 Database (DB) 21 Lesion diagnosis unit 22 Surgical procedure inference unit 23 Prognosis inference unit 24 Output unit 100 Endoscopy system

Claims

1. an image acquisition means for acquiring an endoscopic image captured by an endoscope; a lesion diagnosis means for diagnosing a lesion from the endoscopic image; a surgical procedure inference means for inferring a recommended surgical procedure from the endoscopic image and the diagnostic information of the lesion; a prognosis inference means for inferring a prognosis from the endoscopic image and information on the surgical procedure; an output means for outputting the diagnostic information, the surgical procedure information, and the prognosis; An information processing device comprising:

2. The information processing apparatus according to claim 1 , wherein the output means generates image data based on the diagnostic information of the lesion, the information on the surgical procedure, and the prognosis, and outputs the image data to a display device.

3. The information processing device according to claim 2, wherein the output means generates display data including the basis for the diagnostic information of the lesion, the basis for inferring the surgical procedure, and the basis for inferring the prognosis, and outputs the display data to the display device.

4. The information processing device according to claim 1 , wherein, when there are a plurality of recommended surgical procedures, the output means rearranges the surgical procedures based on the prognosis.

5. a selection means for selecting a champion image that best represents the same lesion from a plurality of endoscopic images corresponding to the same lesion; the lesion diagnosis means diagnoses a lesion from the champion image; the surgical procedure inference means infers a recommended surgical procedure from the champion image and the diagnostic information of the lesion; The information processing device according to claim 1 , wherein the prognosis inferring means infers a prognosis from the champion image and information on the surgical procedure.

6. patient information acquisition means for acquiring patient information; the lesion diagnosis means diagnoses a lesion based on the endoscopic image and the patient information, the surgical procedure inference means infers a recommended surgical procedure from the endoscopic image, the diagnostic information of the lesion, and the patient information; The information processing device according to claim 1 , wherein the prognosis inferring means infers a prognosis from the endoscopic image, the surgical procedure information, and the patient information.

7. A computer comprising: Acquire an endoscopic image taken by an endoscope, Diagnosing a lesion from the endoscopic image; Inferring a recommended surgical procedure from the endoscopic image and the diagnostic information of the lesion; Inferring a prognosis from the endoscopic image and the surgical procedure information; An information processing method that outputs the diagnostic information, the surgical procedure information, and the prognosis state.

8. Acquire an endoscopic image taken by an endoscope, Diagnosing a lesion from the endoscopic image; Inferring a recommended surgical procedure from the endoscopic image and the diagnostic information of the lesion; Inferring a prognosis from the endoscopic image and the surgical procedure information; A program that causes a computer to execute a process of outputting the diagnostic information, the surgical procedure information, and the prognosis state.