Medical support system, report creation support method, and information processing apparatus

Abstract

An image acquisition unit 86 acquires a plurality of endoscopic images captured during an endoscopy. An additional information acquisition unit 88 acquires additional information for each of the plurality of endoscopic images. If an image analysis device does not detect any endoscopic image including a lesion to be attached to a report during the endoscopy, an image selection unit 102 selects a predetermined quantity of images to be attached to a report from the plurality of endoscopic images, on the basis of a priority order set with respect to the additional information. A report generation unit 104 generates a report including information indicating the absence of lesions, and the images to be attached to the report.

Description

Technical Field

[0001] The present disclosure relates to a medical support system and a medical support method for assisting in the creation of a report.

Background Art

[0002] In an endoscopic examination, a doctor observes an endoscopic image displayed on a display device, and when finding a lesion, operates the release switch of the endoscope to capture (save) the endoscopic image of the lesion. After the examination, the doctor inputs the examination results into a report input screen, and selects an endoscopic image to be attached to the report from among the plurality of captured endoscopic images to create a report. Patent Document 1 discloses a report input screen that displays a list of a plurality of captured endoscopic images as candidate attached images. Patent Document 2 discloses a system that recognizes a treatment or a lesion for a subject from an endoscopic image and selects an endoscopic image to be used in a report.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, research and development of computer-aided diagnosis (CAD) systems have been underway. When creating an examination report, it is expected that by automatically selecting, as attached images to the report, the images in which the CAD system has detected lesions, the burden of the doctor's report creation work can be reduced. In an examination where the CAD system does not detect a lesion, there is no image in which the CAD system has detected a lesion, but even in such a case, it is preferable that the report creation work can be efficiently supported.

[0005] The present disclosure has been made in view of such circumstances, and its object is to provide a technique for supporting the report creation work by a computer such as a CAD system when an endoscopic image including a lesion to be attached to the report is not detected.

Means for Solving the Problems

[0006] In order to solve the above problems, a medical support system according to an aspect of the present invention includes one or more processors having hardware, and the one or more processors acquire a plurality of endoscopic images taken in an endoscopic examination, acquire additional information for each of the plurality of endoscopic images, and when the computer does not detect an endoscopic image including a lesion to be attached to the report in the endoscopic examination, based on the priority order set for the additional information, select a predetermined amount of report-attached images from among the plurality of endoscopic images, and generate a report including information indicating the absence of a lesion and the report-attached images.

[0007] Another aspect of the present invention is a medical support method, which includes acquiring a plurality of endoscopic images taken in an endoscopic examination, acquiring additional information for each of the plurality of endoscopic images, and when the computer does not detect an endoscopic image including a lesion to be attached to the report in the endoscopic examination, based on the priority order set for the additional information, select a predetermined amount of report-attached images from among the plurality of endoscopic images, and generate a report including information indicating the absence of a lesion and the report-attached images.

[0008] In addition, any combination of the above components, as well as those obtained by converting the expressions of the present disclosure among methods, apparatuses, systems, recording media, computer programs, etc., are also effective as aspects of the present disclosure.

Brief Description of Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Modes for Carrying Out the Invention

[0010] FIG. 1 shows the configuration of the medical support system 1 according to the embodiment. The medical support system 1 is provided in a medical facility such as a hospital that performs endoscopic examinations. In the medical support system 1, the server device 2, the image analysis device 3, the image storage device 8, the endoscope system 9, and the terminal device 10b are communicably connected via a network 4 such as a LAN (Local Area Network). The endoscope system 9 is provided in the examination room and includes an endoscope observation device 5 and a terminal device 10a. In the medical support system 1, the server device 2, the image analysis device 3, and the image storage device 8 may be provided outside the medical facility, for example, as a cloud server.

[0011] The endoscope observation device 5 is connected to an endoscope 7 that is inserted into a patient's digestive tract. The endoscope 7 has a light guide for transmitting the illumination light supplied from the endoscope observation device 5 to illuminate the inside of the digestive tract. At the tip, there are provided an illumination window for emitting the illumination light transmitted by the light guide to the biological tissue, and an imaging unit for imaging the biological tissue at a predetermined cycle and outputting an imaging signal to the endoscope observation device 5. The imaging unit includes a solid-state imaging device (for example, a CCD image sensor or a CMOS image sensor) that converts incident light into an electrical signal.

[0012] The endoscope observation device 5 performs image processing on the imaging signal photoelectrically converted by the solid-state imaging device of the endoscope 7 to generate an endoscope image and displays it on the display device 6 in real time. The endoscope observation device 5 may be provided with a function of performing special image processing for the purpose of highlighting display, etc., in addition to normal image processing such as A / D conversion and noise removal. The endoscope observation device 5 generates an endoscope image at a predetermined cycle (for example, 1 / 60 second). The endoscope observation device 5 may be configured by one or more processors having dedicated hardware, or may be configured by one or more processors having general-purpose hardware.

[0013] The doctor observes the endoscopic image displayed on the display device 6 in accordance with the inspection procedure. While moving the endoscope 7, the doctor observes the endoscopic image. When the biological tissue to be captured appears on the display device 6, the doctor operates the release switch of the endoscope 7. When the release switch is operated, the endoscopic observation device 5 captures (saves) the endoscopic image at the timing when the release switch is operated, and transmits the captured endoscopic image to the image storage device 8 together with information (image ID) for identifying the endoscopic image. Note that the endoscopic observation device 5 may transmit a plurality of captured endoscopic images to the image storage device 8 collectively after the inspection is completed. The image storage device 8 records the endoscopic images transmitted from the endoscopic observation device 5 in association with an inspection ID for identifying the endoscopic examination.

[0014] In the embodiment, "imaging" means the operation in which the solid-state imaging device of the endoscope 7 converts incident light into an electrical signal, and "capturing" means the operation of saving (recording) the endoscopic image generated by the endoscopic observation device 5. Note that "imaging" may include the operation from the converted electrical signal until the endoscopic observation device 5 generates an endoscopic image.

[0015] The terminal device 10a includes an information processing device 11a and a display device 12a, and is provided in the examination room. The terminal device 10a may be used for a doctor, a nurse, etc. to confirm information regarding the biological tissue being imaged in real time during an endoscopic examination.

[0016] The terminal device 10b includes an information processing device 11b and a display device 12b, and is provided in a room other than the examination room. The terminal device 10b is used when a doctor creates a report of an endoscopic examination. The terminal devices 10a and 10b are configured by one or more processors having general-purpose hardware.

[0017] In the medical support system 1 according to the embodiment, the endoscope observation device 5 causes the display device 6 to display the endoscope image in real time, and supplies the endoscope image to the image analysis device 3 in real time together with the meta information of the image. Here, the meta information includes at least the frame number of the image and the shooting time information, and the frame number may be information indicating which frame it is since the endoscope 7 started shooting.

[0018] The image analysis device 3 is an electronic computer (computer) that analyzes the endoscope image, detects a lesion included in the endoscope image, and performs a qualitative diagnosis on the detected lesion. The image analysis device 3 may be a CAD (computer-aided diagnosis) system having an AI (artificial intelligence) diagnosis function. The image analysis device 3 may be configured by one or more processors having dedicated hardware, or may be configured by one or more processors having general-purpose hardware.

[0019] The image analysis device 3 uses a learned model generated by machine learning using, as teacher data, endoscope images for learning and information on lesion regions included in the endoscope images. The annotation work of the endoscope image is performed by an annotator having specialized knowledge such as a doctor, and CNN, RNN, LSTM, etc., which are a kind of deep learning, may be used for machine learning. When this learned model receives an endoscope image as input, it outputs information indicating the imaged organ, information indicating the imaged site, and information regarding the imaged lesion (lesion information). The lesion information output by the image analysis device 3 includes at least lesion presence / absence information indicating whether or not a lesion is included in the endoscope image. When a lesion is included, the lesion information may include information indicating the size of the lesion, information indicating the position of the contour of the lesion, information indicating the shape of the lesion, information indicating the depth of the lesion, and the qualitative diagnosis result of the lesion. The qualitative diagnosis result of the lesion includes the type of the lesion. During the endoscope examination, the image analysis device 3 is provided with the endoscope image from the endoscope observation device 5 in real time, and outputs information indicating the organ, information indicating the site, and lesion information for each endoscope image.

[0020] The image analysis device 3 has a function of recognizing the image quality of the endoscopic image. The image analysis device 3 checks whether blurring, shaking, fogginess, or the like has occurred in the image, or whether residues are included, by image analysis, and determines whether the image has been appropriately captured. The image analysis device 3 outputs image quality pass / fail information indicating whether the image has been appropriately captured for each endoscopic image. Hereinafter, the information indicating the organ, the information indicating the site, the lesion information, and the image quality pass / fail information output for each endoscopic image are collectively referred to as "image analysis information".

[0021] When the user operates the release switch (capture operation), the endoscopic observation device 5 provides the image analysis device 3 with information indicating that the capture operation has been performed (capture operation information), together with the frame number, shooting time, and image ID of the captured endoscopic image. When the image analysis device 3 acquires the capture operation information, it provides the server device 2 with the image ID, frame number, shooting time information, and image analysis information of the frame number provided, together with the inspection ID. Here, the image ID, frame number, shooting time information, and image analysis information constitute "additional information" that expresses the characteristics and properties of the endoscopic image. When the image analysis device 3 acquires the capture operation information, it transmits the additional information together with the inspection ID to the server device 2, and the server device 2 records the additional information in association with the inspection ID.

[0022] When the user finishes the endoscopic examination, the user operates the examination end button of the endoscopic observation apparatus 5. The operation information of the examination end button is supplied to the server apparatus 2 and the image analysis apparatus 3, and the server apparatus 2 and the image analysis apparatus 3 recognize the end of the endoscopic examination. When receiving the examination end information, the image analysis apparatus 3 provides the server apparatus 2 with information indicating whether or not an endoscopic image including a lesion to be attached to the report has been detected in the endoscopic examination, together with the examination ID. Hereinafter, the information indicating whether or not an endoscopic image including a lesion to be attached to the report has been detected is referred to as "lesion image presence / absence information". For example, if the image analysis apparatus 3 has detected even one endoscopic image including a lesion during the examination, the image analysis apparatus 3 may provide the server apparatus 2 with the lesion image presence / absence information indicating that an endoscopic image to be attached to the report has been detected. On the other hand, if the image analysis apparatus 3 has not detected any endoscopic image including a lesion during the examination, the image analysis apparatus 3 may provide the server apparatus 2 with the lesion image presence / absence information indicating that no endoscopic image to be attached to the report has been detected.

[0023] FIG. 2 shows a functional block of the server apparatus 2. The server apparatus 2 includes a communication unit 20, a processing unit 30, and a storage device 60. The communication unit 20 transmits and receives information such as data and instructions to and from the image analysis apparatus 3, the endoscopic observation apparatus 5, the image storage apparatus 8, the terminal apparatus 10a, and the terminal apparatus 10b via the network 4. The processing unit 30 includes an order information acquisition unit 40, an additional information acquisition unit 42, and a lesion image presence / absence information acquisition unit 44. The storage device 60 has an order information storage unit 62, an additional information storage unit 64, and a lesion image presence / absence information storage unit 66.

[0024] The server device 2 includes a computer, and by executing a program on the computer, various functions shown in FIG. 2 are realized. The computer includes, as hardware, a memory for loading a program, one or more processors for executing the loaded program, an auxiliary storage device, and other LSIs. The processor is composed of a plurality of electronic circuits including a semiconductor integrated circuit and an LSI, and the plurality of electronic circuits may be mounted on one chip or may be mounted on a plurality of chips. The functional blocks shown in FIG. 2 are realized by the cooperation of hardware and software. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by only hardware, only software, or a combination thereof.

[0025] The order information acquisition unit 40 acquires order information for an endoscopy examination from the hospital information system. For example, the order information acquisition unit 40 acquires the order information for the day from the hospital information system before the start of the examination operations for one day in the medical facility and stores it in the order information storage unit 62. Before the start of the examination, the endoscope observation device 5 or the information processing device 11a may read the order information for the examination to be performed from the order information storage unit 62 and display it on the display device.

[0026] The additional information acquisition unit 42 acquires the examination ID and the additional information of the endoscope image from the image analysis device 3, and stores the additional information in the additional information storage unit 64 in association with the examination ID. The additional information of the endoscope image includes an image ID, a frame number, shooting time information, and image analysis information. For example, when the user performs the capture operation 17 times during the endoscopy examination, the additional information acquisition unit 42 acquires the additional information of 17 endoscope images from the image analysis device 3 and stores it in the additional information storage unit 64. The image ID is assigned by the endoscope observation device 5, and the endoscope observation device 5 assigns the image ID in order from 1 in the order of shooting time. Therefore, in this case, image IDs 1 to 17 are assigned to the 17 endoscope images respectively.

[0027] The lesion image presence / absence information acquisition unit 44 acquires the examination ID and the lesion image presence / absence information from the image analysis device 3, and stores the lesion image presence / absence information in association with the examination ID in the lesion image presence / absence information storage unit 66. After the examination is completed, the lesion image presence / absence information acquisition unit 44 may acquire the lesion image presence / absence information from the image analysis device 3.

[0028] In the embodiment, when the image analysis device 3 does not detect a lesion in the endoscopic examination, the information processing device 11b proposes a technique for automatically selecting an endoscopic image to be attached to the report of the examination. Therefore, in the embodiment, the lesion image presence / absence information acquisition unit 44 acquires, after the examination is completed, the lesion image presence / absence information indicating that no endoscopic image including a lesion to be attached to the report has been detected from the image analysis device 3.

[0029] FIG. 3 shows an example of additional information associated with the captured endoscopic image. In the item of "organ", information indicating the organ included in the image, that is, information indicating the organ that has been photographed, is stored, and in the item of "site", information indicating the site of the organ included in the image, that is, information indicating the site of the organ that has been photographed, is stored.

[0030] Among the lesion information, in the item of "presence / absence", information indicating whether or not a lesion has been detected by the image analysis device 3 is stored. As described above, in the embodiment, since the image analysis device 3 has not detected a lesion in the endoscopic examination, "none" is stored in the item of "presence / absence" for all the endoscopic images of image IDs 1 to 17.

[0031] In the item of "size", information indicating the longest diameter of the bottom surface of the lesion is stored, in the item of "shape", coordinate information representing the contour shape of the lesion is stored, and in the item of "diagnosis", the qualitative diagnosis result of the lesion is stored. In the embodiment, since no lesion is included in all the endoscopic images, each item is set to blank.

[0032] In the item of "image quality", information indicating whether the image has been appropriately captured is stored. "Good" indicates that the image has been appropriately captured, and "bad" indicates that the image has not been appropriately captured. In the item of "shooting time", information indicating the shooting time of the image is stored. The item of "shooting time" may include a frame number.

[0033] FIG. 4 shows the functional blocks of the information processing apparatus 11b. The information processing apparatus 11b has a function of automatically selecting an endoscopic image to be attached to a report of an examination in which no lesion is detected, and includes a communication unit 76, an input unit 78, a processing unit 80, and a storage device 120. The communication unit 76 transmits and receives information such as data and instructions to and from the server device 2, the image analysis device 3, the endoscopic observation device 5, the image storage device 8, and the terminal device 10a via the network 4. The processing unit 80 includes an operation reception unit 82, an acquisition unit 84, a support mode setting unit 98, a priority setting unit 100, an image selection unit 102, a report generation unit 104, and a registration processing unit 106. The acquisition unit 84 includes an image acquisition unit 86, an additional information acquisition unit 88, and a lesion image presence / absence information acquisition unit 90. The storage device 120 includes an image storage unit 122, an additional information storage unit 124, and a priority storage unit 126.

[0034] The information processing apparatus 11b includes a computer, and various functions shown in FIG. 4 are realized by the computer executing a program. The computer includes, as hardware, a memory for loading the program, one or more processors for executing the loaded program, an auxiliary storage device, and other LSIs. The processor is composed of a plurality of electronic circuits including a semiconductor integrated circuit and an LSI. The plurality of electronic circuits may be mounted on one chip or may be mounted on a plurality of chips. The functional blocks shown in FIG. 4 are realized by the cooperation of hardware and software. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

[0035] After the end of the endoscopic examination, the user, who is a doctor, inputs a user ID and a password into the information processing device 11b to log in. When the user logs in, an application for creating an examination report is launched, and a list of completed examinations is displayed on the display device 12b. In this list of completed examinations, examination information such as the patient name, patient ID, examination date and time, and examination items is listed. The user operates an input unit 78 such as a mouse or a keyboard to select an examination to be the subject of report creation. When the operation reception unit 82 receives the selection operation of the examination, the image acquisition unit 86 acquires a plurality of endoscopic images associated with the examination ID of the examination selected by the user from the image storage device 8 and stores them in the image storage unit 122. The additional information acquisition unit 88 acquires additional information associated with the examination ID of the examination selected by the user from the server device 2 and stores it in the additional information storage unit 124. The lesion image presence / absence information acquisition unit 90 acquires lesion image presence / absence information associated with the examination ID of the examination selected by the user from the server device 2. The lesion image presence / absence information is information indicating whether or not the image analysis device 3 has detected an endoscopic image including a lesion to be attached to the report in the endoscopic examination.

[0036] The support mode setting unit 98 sets a mode in which the image selection unit 102 automatically selects an image to be attached to the report according to the lesion image presence / absence information. When the image analysis device 3 does not detect a lesion in the endoscopic examination, the support mode setting unit 98 sets a first support mode in which the image selection unit 102 selects a report-attached image based on the priority order. In the first support mode, the image selection unit 102 selects a predetermined amount of report-attached images from among the plurality of captured endoscopic images based on the priority order set for the additional information. Here, the predetermined amount of report-attached images may mean a predetermined number of still images when the captured endoscopic image is a still image, and may mean a moving image of a predetermined capacity (or a predetermined time) when the captured endoscopic image is a moving image.

[0037] On the one hand, when the image analysis device 3 detects a lesion during an endoscopic examination, the support mode setting unit 98 sets a second support mode in which the image selection unit 102 selects an endoscopic image including the lesion. In the second support mode, the image selection unit 102 selects a report attachment image including the lesion detected by the image analysis device 3 from among the plurality of captured endoscopic images.

[0038] In the following embodiments, the lesion image presence / absence information acquired by the lesion image presence / absence information acquisition unit 90 indicates that the image analysis device 3 did not detect a lesion, and the operation of the processing unit 80 when the support mode setting unit 98 sets the first support mode will be described. During the examination, the user captures an endoscopic image, which is a still image.

[0039] In the first support mode, the image selection unit 102 automatically selects a report attachment image based on the priority set for the additional information. The priority may be set with respect to at least one of the organ, the site, and the imaging timing. The priority setting unit 100 sets the priority based on the user's input and stores it in the priority storage unit 126. For example, the user may select one priority from among a plurality of priority candidates. Since the user can determine the criteria for automatically selecting the report attachment image in the non-lesion-detected examination, a report reflecting the preferences of the user and the hospital facility can be generated. The upper limit number (upper limit amount) of the report attachment images may also be set by the user.

[0040] FIG. 5(a) shows an example of the priority set for the organ. In the priority shown in FIG. 5(a), the priority of "esophagus" is set to the first place, the priority of "stomach" is set to the second place, and the priority of "duodenum" is set to the third place. The image selection unit 102 selects report attachment images one by one in order from the highest priority while referring to the information indicating the organ included in the additional information of each endoscopic image, and when selecting the report attachment image with the lowest priority, it also selects report attachment images in order from the highest priority.

[0041] For example, when the upper limit number of attached report images is set to "5", the image selection unit 102 selects, for the first one, an attached report image including the esophagus, for the second one, an attached report image including the stomach, for the third one, an attached report image including the duodenum, for the fourth one, an attached report image including the esophagus, and for the fifth one, an attached report image including the stomach. As described above, the image selection unit 102 of the embodiment selects attached report images based on the set priority order within the range of the upper limit number.

[0042] FIG. 5(b) shows another example of the priority order set for organs. In the priority order shown in FIG. 5(b), the priority of "stomach" is set to the first, the priority of "duodenum" is set to the second, and the priority of "esophagus" is set to the third. When the priority order shown in FIG. 5(b) is set and the upper limit number of attached report images is set to "5", the image selection unit 102 selects, for the first one, an attached report image including the stomach, for the second one, an attached report image including the duodenum, for the third one, an attached report image including the esophagus, for the fourth one, an attached report image including the stomach, and for the fifth one, an attached report image including the duodenum.

[0043] FIG. 5(c) shows another example of the priority order set for organs. In the priority order shown in FIG. 5(c), the priority of "stomach" is set to the first, the priority of "esophagus" is set to the second, and the priority of "duodenum" is set to the third. When the priority order shown in FIG. 5(c) is set and the upper limit number of attached report images is set to "5", the image selection unit 102 selects, for the first one, an attached report image including the stomach, for the second one, an attached report image including the esophagus, for the third one, an attached report image including the duodenum, for the fourth one, an attached report image including the stomach, and for the fifth one, an attached report image including the esophagus.

[0044] As described above, the priority setting unit 100 sets priorities based on user input. The image selection unit 102 selects report attached images according to the priorities determined by the user, so that the report generation unit 104 described later can generate a report that meets the user's requirements. Note that the priorities regarding the organs shown in FIGS. 5(a) to 5(c) are examples, and the user may freely determine the priorities regarding the organs.

[0045] FIG. 6 shows an example of the priorities set for the parts of the organs. This priority determines the priority of the parts in each organ. According to the priorities shown in FIG. 6, in the "esophagus", the priority of the "neck" is set to 1st, the priority of the "chest" is set to 2nd, and the priority of the "abdomen" is set to 3rd. In the "stomach", the priority of the "vestibule" is set to 1st, the priority of the "corner" is set to 2nd, and the priority of the "body" is set to 3rd. In the "duodenum", the priority of the "descending part" is set to 1st, the priority of the "bulb" is set to 2nd, and the priority of the "accessory papilla" is set to 3rd. Note that the priorities shown in FIG. 6 are examples, and the user may arbitrarily determine the priorities of the parts in each organ.

[0046] The image selection unit 102 may select report attached images based on the priorities set for the organs (for example, the priorities shown in any of FIGS. 5(a) to 5(c)) and the priorities set for the parts. Hereinafter, the operation of the image selection unit 102 for selecting report attached images based on the priority of the organs shown in FIG. 5(a) (hereinafter simply referred to as the "organ priority") and the priority of the parts shown in FIG. 6 (hereinafter simply referred to as the "part priority") will be described. The image selection unit 102 identifies an endoscopic image including the organ specified by the organ priority shown in FIG. 5(a), and when a plurality of endoscopic images including the organ are acquired, selects a report attached image from among the plurality of endoscopic images including the organ based on the part priority shown in FIG. 6.

[0047] The image selection unit 102 identifies the priority of the combination of the organ and the part for selecting the report attached image from the organ priority shown in FIG. 5(a) and the part priority shown in FIG. 6. (Ranking Determination Step 1) When the image selection unit 102 recognizes from the organ ranking shown in Fig. 5(a) that the report-attached image to be selected first is an image including "esophagus", and recognizes from the site ranking shown in Fig. 6 that the priority of "neck" in "esophagus" is the first, it determines that the combination of (esophagus, neck) has the first priority.

[0048] (Ranking Determination Step 2) When the image selection unit 102 recognizes from the organ ranking shown in Fig. 5(a) that the report-attached image to be selected second is an image including "stomach", and recognizes from the site ranking shown in Fig. 6 that the priority of "vestibular part" in "stomach" is the first, it determines that the combination of (stomach, vestibular part) has the second priority.

[0049] (Ranking Determination Step 3) When the image selection unit 102 recognizes from the organ ranking shown in Fig. 5(a) that the report-attached image to be selected third is an image including "duodenum", and recognizes from the site ranking shown in Fig. 6 that the priority of "descending part" in "duodenum" is the first, it determines that the combination of (duodenum, descending part) has the third priority.

[0050] (Ranking Determination Step 4) When the image selection unit 102 recognizes from the organ ranking shown in Fig. 5(a) that the report-attached image to be selected fourth is an image including "esophagus", and recognizes from the site ranking shown in Fig. 6 that the priority of "chest" in "esophagus" is the second, it determines that the combination of (esophagus, chest) has the fourth priority.

[0051] (Ranking Determination Step 5) When the image selection unit 102 recognizes from the organ ranking shown in Fig. 5(a) that the report-attached image to be selected fifth is an image including "stomach", and recognizes from the site ranking shown in Fig. 6 that the priority of "angular part" in "stomach" is the second, it determines that the combination of (stomach, angular part) has the fifth priority.

[0052] (Ranking Determination Step 6) When the image selection unit 102 recognizes that the report-attached image selected as the sixth image from the organ rankings shown in Fig. 5(a) is an image including the "duodenum", and recognizes that the priority of the "bulb" in the "duodenum" is the second from the site rankings shown in Fig. 6, it determines that the combination of (duodenum, bulb) has the sixth priority.

[0053] (Ranking Determination Step 7) When the image selection unit 102 recognizes that the report-attached image selected as the seventh image from the organ rankings shown in Fig. 5(a) is an image including the "esophagus", and recognizes that the priority of the "abdomen" in the "esophagus" is the third from the site rankings shown in Fig. 6, it determines that the combination of (esophagus, abdomen) has the seventh priority.

[0054] (Ranking Determination Step 8) When the image selection unit 102 recognizes that the report-attached image selected as the eighth image from the organ rankings shown in Fig. 5(a) is an image including the "stomach", and recognizes that the priority of the "body" in the "stomach" is the third from the site rankings shown in Fig. 6, it determines that the combination of (stomach, body) has the eighth priority.

[0055] (Ranking Determination Step 9) When the image selection unit 102 recognizes that the report-attached image selected as the ninth image from the organ rankings shown in Fig. 5(a) is an image including the "duodenum", and recognizes that the priority of the "accessory papilla" in the "duodenum" is the third from the site rankings shown in Fig. 6, it determines that the combination of (duodenum, accessory papilla) has the ninth priority.

[0056] Fig. 7 shows the priorities determined from the organ rankings shown in Fig. 5(a) and the site rankings shown in Fig. 6. When the upper limit number is 5, the image selection unit 102 selects, according to the priority shown in FIG. 7, as the first one, a report-attached image including the cervical part of the esophagus, as the second one, a report-attached image including the vestibule part of the stomach, as the third one, a report-attached image including the descending part of the duodenum, as the fourth one, a report-attached image including the thoracic part of the esophagus, and as the fifth one, a report-attached image including the angular part of the stomach. As described above, the image selection unit 102 may select the report-attached images based on the priority derived for the organs and parts within the range of the upper limit number.

[0057] In addition, when there is no endoscopic image including the combination of (organ, part) in the additional information, the image selection unit 102 selects a report-attached image including a part with a lower priority in the same organ. For example, when the image selection unit 102 attempts to select a report-attached image including the descending part of the duodenum, if there is no endoscopic image including the descending part of the duodenum, it selects a report-attached image including the bulb of the duodenum. If there is also no endoscopic image including the bulb of the duodenum, the image selection unit 102 selects a report-attached image including the accessory papilla of the duodenum. In this way, the image selection unit 102 can select the report-attached images of each organ evenly by selecting the report-attached images based on the organ order.

[0058] FIG. 8(a) shows an example of the priority set regarding the shooting order. The priority set regarding the shooting order is used to select one report-attached image from a plurality of endoscopic images including the same organ or the same part when a plurality of endoscopic images including the same organ or the same part are acquired.

[0059] The priority setting unit 100 sets any one of a first priority criterion for preferentially selecting the endoscopic image shot at the first timing, a second priority criterion for preferentially selecting the endoscopic image shot at the intermediate timing, and a third priority criterion for preferentially selecting the endoscopic image shot at the last timing as the priority regarding the shooting order. In the example shown in FIG. 8(a), the first priority criterion is set.

[0060] For example, when the priority for image selection includes the organ priority shown in FIG. 5(a) and the priority of the shooting order shown in FIG. 8(a), the image selection unit 102 refers to the information indicating the shooting time included in the additional information of the endoscopic image including the esophagus, and selects the endoscopic image including the esophagus with the earliest shooting order as the first report attached image.

[0061] FIG. 8(b) shows another example of the priority set for the shooting order. In the example shown in FIG. 8(b), a second priority criterion is set. When the second priority criterion is set, the image selection unit 102 selects, from among a plurality of endoscopic images, the endoscopic image with an intermediate shooting timing as the report attached image.

[0062] FIG. 8(c) shows another example of the priority set for the shooting order. In the example shown in FIG. 8(c), a third priority criterion is set. When the third priority criterion is set, the image selection unit 102 selects, from among a plurality of endoscopic images, the endoscopic image with the latest shooting order as the report attached image. In this way, the image selection unit 102 can narrow down the report attached images to one by taking into account the priority of the shooting order.

[0063] Hereinafter, the procedure for the image selection unit 102 to select the report attached images based on the organ priority shown in FIG. 5(a), the part priority shown in FIG. 6, and the priority of the shooting order shown in FIG. 8(c) will be described. The upper limit number of the report attached images is set to "5". The number of endoscopic images acquired during the examination is 17, and the additional information acquisition unit 88 acquires the additional information of the endoscopic images shown in FIG. 3 and stores it in the additional information storage unit 124.

[0064] (Procedure for selecting the first report attached image) When the image selection unit 102 identifies that the combination of (esophagus, neck) has the highest priority from the organ ranking and the part ranking, it refers to the additional information of each endoscopic image and searches for an endoscopic image with image analysis information of (esophagus, neck). Referring to FIG. 3 here, the endoscopic image with image ID 1 corresponds, and thus the image selection unit 102 selects the endoscopic image with image ID 1 as the first attached image of the report.

[0065] (Procedure for selecting the second attached image of the report) When the image selection unit 102 identifies that the combination of (stomach, vestibule) has the second highest priority from the organ ranking and the part ranking, it refers to the additional information of each endoscopic image and searches for an endoscopic image with image analysis information of (stomach, vestibule). Referring to FIG. 3 here, two endoscopic images with image IDs 13 and 14 correspond. Since the priority order of the shooting order shown in FIG. 8(c) is set as the third priority criterion for preferentially selecting the endoscopic image taken at the last timing, the image selection unit 102 selects the endoscopic image with image ID 14, which has a later shooting time, as the second attached image of the report.

[0066] (Procedure for selecting the third attached image of the report) When the image selection unit 102 identifies that the combination of (duodenum, descending part) has the third highest priority from the organ ranking and the part ranking, it refers to the additional information of each endoscopic image and searches for an endoscopic image with image analysis information of (duodenum, descending part). Referring to FIG. 3, since there is no endoscopic image with image analysis information of (duodenum, descending part), the image selection unit 102 searches for an endoscopic image with image analysis information of "bulb", which has a part ranking one lower in the duodenum. In FIG. 3, the endoscopic image with image ID 15 corresponds.

[0067] When the image selection unit 102 recognizes that "bad", indicating inappropriate shooting, is stored in the "image quality pass / fail" item associated with image ID 15, it determines not to select the endoscopic image of image ID 15 as an attached image for the report. Therefore, the image selection unit 102 searches for an endoscopic image having image analysis information of "accessory papilla" with a site ranking one lower in the duodenum. Referring to FIG. 3 here, two endoscopic images of image IDs 16 and 17 are applicable. Since the priority order of the shooting order shown in FIG. 8(c) is set as the third priority criterion, the image selection unit 102 selects the endoscopic image of image ID 17 with a later shooting time as the third attached image for the report.

[0068] (Procedure for selecting the fourth attached image for the report) When the image selection unit 102 identifies that the combination of (esophagus, chest) has a fourth priority in terms of organ ranking and site ranking, it searches for an endoscopic image having image analysis information of (esophagus, chest) by referring to the additional information of each endoscopic image. Referring to FIG. 3 here, two endoscopic images of image IDs 2 and 3 are applicable.

[0069] The image selection unit 102 refers to the "image quality pass / fail" item associated with image IDs 2 and 3 and recognizes that "bad" is stored in the "image quality pass / fail" item of image ID 2. Therefore, the image selection unit 102 determines not to select the endoscopic image of image ID 2 as an attached image for the report and selects the endoscopic image of image ID 3 as the fourth attached image for the report.

[0070] (Procedure for selecting the fifth attached image for the report) When the image selection unit 102 identifies that the combination of (stomach, corner) has a fifth priority in terms of organ ranking and site ranking, it searches for an endoscopic image having image analysis information of (stomach, corner) by referring to the additional information of each endoscopic image. Referring to FIG. 3 here, four endoscopic images of image IDs 9, 10, 11, and 12 are applicable.

[0071] When the image selection unit 102 refers to the item of "image quality good or bad" associated with image IDs 9, 10, 11, and 12 and recognizes that "bad" is stored in the item of "image quality good or bad" of image ID 9, it determines not to select the endoscopic image of image ID 9 as an attached image for the report. Subsequently, based on the priority order of the shooting order shown in Fig. 8(c), the image selection unit 102 selects the endoscopic image of image ID 12, whose shooting time is the last, as the fifth attached image for the report.

[0072] Through the above selection procedure, the image selection unit 102 selects 5 attached images for the report, which is the upper limit number. By the user setting the priority order in advance, endoscopic images reflecting the user's preferences will be automatically attached to the report.

[0073] Fig. 9 shows an example of an automatically generated report. When the support mode setting unit 98 sets the first support mode, as described above, the image selection unit 102 selects attached images for the report based on the priority order set for the additional information, and the report generation unit 104 automatically generates a report using the selected attached images for the report.

[0074] Fig. 10 shows an example of an automatically generated report. The report generation unit 104 generates a report including information indicating the absence of lesions and the attached images for the report selected by the image selection unit 102, and displays it on the display device 12b. When the image selection unit 102 selects an attached image for the report, the report generation unit 104 may generate a report including the content of the inspection order, the diagnosis content indicating the absence of abnormal findings, and the attached images for the report. The user can confirm the report displayed on the display device 12b and operate a predetermined registration button (not shown), so that the registration processing unit 106 registers the automatically generated report in the server device 2.

[0075] As described above, the present disclosure has been described based on the embodiments. It is understood by those skilled in the art that the embodiments are examples, and various modifications are possible for each combination of their components and each processing process, and such modifications are also within the scope of the present disclosure.

[0076] FIG. 10 shows a modification of the priority order set for the combination of organs and parts. In the embodiment, the priority order is set by the combination of the organ order shown in FIGS. 5(a) to 5(c) and the part order shown in FIGS. 6(a) to 6(b). However, the priority order shown in FIG. 10 is determined for the combination of organs and parts. Hereinafter, the procedure for the image selection unit 102 to select the report-attached images based on the priority order shown in FIG. 10 and the priority order of the shooting order shown in FIG. 8(c) will be described. The upper limit number of the report-attached images is set to "5". The additional information acquisition unit 88 acquires the additional information of the endoscopic image shown in FIG. 3 and stores it in the additional information storage unit 124.

[0077] (Procedure for selecting the first report-attached image) When the image selection unit 102 recognizes that the combination of (esophagus, neck) has the first priority, it refers to the additional information of each endoscopic image and searches for an endoscopic image having the image analysis information of (esophagus, neck). Referring to FIG. 3 here, the endoscopic image with image ID 1 corresponds. Therefore, the image selection unit 102 selects the endoscopic image with image ID 1 as the first report-attached image.

[0078] (Procedure for selecting the second report-attached image) When the image selection unit 102 recognizes that the combination of (esophagus, chest) has the second priority, it refers to the additional information of each endoscopic image and searches for an endoscopic image having the image analysis information of (esophagus, chest). Referring to FIG. 3 here, the two endoscopic images with image IDs 2 and 3 correspond.

[0079] The image selection unit 102 refers to the item of "image quality good or bad" associated with image IDs 2 and 3 and recognizes that "bad" is stored in the item of "image quality good or bad" of image ID 2. Therefore, the image selection unit 102 determines not to select the endoscopic image with image ID 2 as the report-attached image and selects the endoscopic image with image ID 3 as the second report-attached image.

[0080] (Procedure for selecting the third report-attached image) When the image selection unit 102 recognizes that the combination of (esophagus, abdomen) has the third priority, it refers to the additional information of each endoscopic image and searches for an endoscopic image with image analysis information of (esophagus, abdomen). Referring to FIG. 3 here, two endoscopic images with image IDs 4 and 5 are applicable. Since the priority order of the shooting order shown in FIG. 8(c) is set as the third priority criterion for preferentially selecting the endoscopic image taken at the last timing, the image selection unit 102 selects the endoscopic image with image ID 5, which has a later shooting time, as the third report attachment image.

[0081] (Selection procedure for the fourth report attachment image) When the image selection unit 102 recognizes that the combination of (stomach, body) has the fourth priority, it refers to the additional information of each endoscopic image and searches for an endoscopic image with image analysis information of (stomach, body). Referring to FIG. 3 here, three endoscopic images with image IDs 6, 7, and 8 are applicable. Since the priority order of the shooting order shown in FIG. 8(c) is set as the third priority criterion, the image selection unit 102 selects the endoscopic image with image ID 8, which has the last shooting time, as the fourth report attachment image.

[0082] (Selection procedure for the fifth report attachment image) When the image selection unit 102 identifies that the combination of (stomach, corner) has the fifth priority, it refers to the additional information of each endoscopic image and searches for an endoscopic image with image analysis information of (stomach, corner). Referring to FIG. 3 here, four endoscopic images with image IDs 9, 10, 11, and 12 are applicable.

[0083] When the image selection unit 102 refers to the "image quality good or bad" item associated with image IDs 9, 10, 11, and 12 and recognizes that "bad" is stored in the "image quality good or bad" item of image ID 9, it decides not to select the endoscopic image with image ID 9 as a report attachment image. Subsequently, based on the priority order of the shooting order shown in FIG. 8(c), the image selection unit 102 selects the endoscopic image with image ID 12, which has the last shooting time, as the fifth report attachment image.

[0084] Through the above selection procedure, the image selection unit 102 selects five report attached images, which is the upper limit number. By having the user preset the priorities for all combinations of organs and parts in advance, endoscopic images reflecting the user's preferences will be automatically attached to the report.

[0085] FIG. 11 shows a modification example of the priorities set for combinations of organs and parts. In this example, priorities for shooting timings are further associated with combinations of organs and parts. By being able to set priorities for shooting timings for combinations of organs and parts in this way, it becomes possible to select report attached images that more finely reflect the user's preferences.

[0086] In the embodiment, it was explained that the user can set the upper limit number (upper limit quantity) of report attached images. The user may be able to select the upper limit number from 0. When the user sets the upper limit number to 0, when the support mode setting unit 98 sets the first support mode, the image selection unit 102 operates so as not to automatically select report attached images.

[0087] In the embodiment, the case where the endoscopic examination is an upper endoscopic examination was explained, but the endoscopic examination may be a lower endoscopic examination. The priority setting unit 100 may set priorities for each examination item based on the user's input. For example, the priority setting unit 100 may set the priority regarding the shooting timing as the third priority criterion for the report of the upper endoscopic examination, and set the priority regarding the shooting timing as the second priority criterion for the report of the lower endoscopic examination. In this case, the image selection unit 102 selects the report attached images of the upper endoscopic examination based on the third priority criterion, while selecting the report attached images of the lower endoscopic examination based on the second priority criterion. Since the organs targeted by the lower endoscopic examination are long, images captured at an intermediate timing in the lower endoscopic examination are preferable as report attached images representing the state of the organs.

[0088] In the embodiment, the endoscopic observation device 5 transmits the captured image to the image storage device 8. However, in a modified example, the image analysis device 3 may transmit the captured image to the image storage device 8. Also, in the embodiment, the information processing device 11b has the processing unit 80. However, in a modified example, the server device 2 may have the processing unit 80.

Industrial Applicability

[0089] The present disclosure can be used in the technical field of assisting in the creation of reports.

Explanation of Signs

[0090] 1 ··· Medical support system, 2 ··· Server device, 3 ··· Image analysis device, 4 ··· Network, 5 ··· Endoscopic observation device, 6 ··· Display device, 7 ··· Endoscope, 8 ··· Image storage device, 9 ··· Endoscopic system, 10a, 10b ··· Terminal device, 11a, 11b ··· Information processing device, 12a, 12b ··· Display device, 20 ··· Communication unit, 30 ··· Processing unit, 40 ··· Order information acquisition unit, 42 ··· Additional information acquisition unit, 44 ··· Lesion image presence / absence information acquisition unit, 60 ··· Storage device, 62 ··· Order information storage unit, 64 ··· Additional information storage unit, 66 ··· Lesion image presence / absence information storage unit, 76 ··· Communication unit, 78 ··· Input unit, 80 ··· Processing unit, 82 ··· Operation reception unit, 84 ··· Acquisition unit, 86 ··· Image acquisition unit, 88 ··· Additional information acquisition unit, 90 ··· Lesion image presence / absence information acquisition unit, 98 ··· Support mode setting unit, 100 ··· Priority setting unit, 102 ··· Image selection unit, 104 ··· Report generation unit, 106 ··· Registration processing unit, 120 ··· Storage device, 122 ··· Image storage unit, 124 ··· Additional information storage unit, 126 ··· Priority storage unit.

Claims

1. A medical support system comprising one or more processors having hardware, said one or more processors, obtain a plurality of endoscopic images taken during an endoscopic examination, obtain additional information for each of the plurality of endoscopic images, When a computer fails to detect an endoscopic image including a lesion to be attached to a report during an endoscopic examination, a predetermined amount of report attachment images are selected from among the plurality of endoscopic images based on the priority set for the additional information. A medical support system characterized by the above.

2. The one or more processors generate a report including information indicating the absence of a lesion and the report attachment images. The medical support system according to claim 1, characterized by the above.

3. The one or more processors When a computer fails to detect an endoscopic image including a lesion to be attached to a report during an endoscopic examination, the report attachment images are selected based on the priority. When a computer detects an endoscopic image including a lesion to be attached to a report during an endoscopic examination, the report attachment images including the lesion are selected from among the plurality of endoscopic images. The medical support system according to claim 1, characterized by the above.

4. The one or more processors set the priority based on user input. The medical support system according to claim 1, characterized by the above.

5. The one or more processors set the priority for each examination item based on user input. The medical support system according to claim 4, characterized by the above.

6. The additional information includes information indicating an organ included in the endoscopic image, and the one or more processors select the report-attached image based on a priority set for the organ. The medical support system according to claim 1, characterized in that.

7. The additional information includes information indicating a part of an organ included in the endoscopic image, and the one or more processors select the report-attached image based on a priority set for the part of the organ. The medical support system according to claim 6, characterized in that.

8. The additional information includes information indicating the order in which the endoscopic image was taken, and the one or more processors when a plurality of the endoscopic images including the same organ or the same part are acquired, select the report-attached image from among the plurality of the endoscopic images including the organ or the part based on a priority set for the order in which the images were taken. The medical support system according to claim 1, characterized in that.

9. The one or more processors set any one of a first priority criterion for preferentially selecting the endoscopic image taken at the first timing, a second priority criterion for preferentially selecting the endoscopic image taken at an intermediate timing, and a third priority criterion for preferentially selecting the endoscopic image taken at the last timing as the priority for the order in which the images were taken. The medical support system according to claim 8, characterized in that.

10. The one or more processors When a plurality of the endoscopic images including the same organ are acquired, based on the priority set for the part of the organ, select the report attached image from among the plurality of the endoscopic images including the organ. The medical support system according to claim 7, characterized in that.

11. The one or more processors Select the report attached image based on the priority set for the combination of the organ and the part. The medical support system according to claim 7, characterized in that.

12. The one or more processors Set the upper limit amount of the report attached images based on the input of the user, The user can select the upper limit amount from 0 sheets. The medical support system according to claim 1, characterized in that.

13. The endoscopic examination includes an upper endoscopic examination and a lower endoscopic examination, The one or more processors Select the report attached image of the upper endoscopic examination based on the third priority criterion, Select the report attached image of the lower endoscopic examination based on the second priority criterion. The medical support system according to claim 9, characterized in that.

14. The one or more processors Do not select, as the report attached image, the endoscopic image in which information indicating that it has not been appropriately taken is included in the additional information. The medical support system according to claim 1, characterized in that.

15. A report creation support method, Acquire a plurality of endoscopic images taken in an endoscopic examination, Acquire the additional information of each of the plurality of endoscopic images, When a computer fails to detect an endoscopic image including a lesion to be attached to a report during an endoscopic examination, a predetermined amount of report-attached images are selected from among the plurality of endoscopic images based on the priority set for the additional information. A report creation support method characterized by the above.

16. Setting the priority based on user input. The report creation support method according to claim 15, characterized by the above.

17. In a computer, A function of acquiring a plurality of endoscopic images taken during an endoscopic examination, A function of acquiring additional information for each of the plurality of endoscopic images, When another computer fails to detect an endoscopic image including a lesion to be attached to a report during an endoscopic examination, a function of selecting a predetermined amount of report-attached images from among the plurality of endoscopic images based on the priority set for the additional information, A program for realizing the above.

18. In the computer, A function of setting the priority based on user input is realized. The program according to claim 17, characterized by the above.

19. An information processing apparatus comprising one or more processors having hardware, The one or more processors, Acquire a plurality of endoscopic images taken during an endoscopic examination, Acquire additional information for each of the plurality of endoscopic images, When a computer fails to detect an endoscopic image including a lesion to be attached to a report during an endoscopic examination, a predetermined amount of report-attached images are selected from among the plurality of endoscopic images based on the priority set for the additional information. An information processing apparatus characterized by the above.

20. The one or more processors, Setting the priority based on a user input. The information processing apparatus according to claim 19, characterized by the above.

Images