Medical assistance system, report preparation assistance method, and information processing device

Abstract

To provide a technique of assisting a report preparation job of an endoscopy.SOLUTION: An image acquisition unit 86 acquires a plurality of endoscopic images captured during an endoscopy. An additional information acquisition unit 88 acquires additional information added to the plurality of endoscopic images and including information indicating whether or not a lesion has been detected. When an endoscopic image including a lesion is being captured during the endoscopy, an image selection unit 102 selects the endoscopic image including a lesion as a report attachment image from among the plurality of endoscopic images. When an endoscopic image including a lesion is not being captured during the endoscopy, the image selection unit 102 selects the endoscopic image not including a lesion as a report attachment image from among the plurality of endoscopic images based on priority orders set with respect to the additional information.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to a medical support system and a medical support method for supporting report creation. [Background technology]

[0002] During an endoscopic examination, a doctor observes the endoscopic image displayed on the display device, and when he finds a lesion, he operates the endoscope's release switch to capture (save) the endoscopic image capturing the lesion. After the examination is completed, the doctor enters the examination results into a report input screen and selects an endoscopic image to attach to the report from the multiple captured endoscopic images to create the report. Patent Document 1 discloses a report input screen that displays a list of multiple captured endoscopic images as candidate images to attach. Patent Document 2 discloses a system that recognizes the treatment and lesion on the subject from the endoscopic image and selects the endoscopic image to use in the report. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-86274 [Patent Document 2] Japanese Patent Publication No. 2020-81332 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, research and development of computer-aided diagnosis (CAD) systems has progressed. When creating examination reports, it is expected that the burden on doctors in creating reports can be reduced by automatically selecting images in which the CAD system has detected a lesion as an attached image to the report. In examinations in which the CAD system does not detect a lesion, there are no images in which the CAD system has detected a lesion. However, even in such cases, it is desirable to be able to efficiently support the report creation process.

[0005] The present disclosure has been made in consideration of these circumstances, and its purpose is to provide a technology that supports report creation work when a computer such as a CAD system does not detect an endoscopic image containing a lesion that should be attached to the report. [Means for solving the problem]

[0006] A medical support system according to one embodiment of the present invention comprises one or more processors having hardware, which acquire a plurality of endoscopic images taken during an endoscopic examination, acquire additional information attached to the plurality of endoscopic images, including information indicating whether a lesion has been detected, and, when an endoscopic image including a lesion is taken during the endoscopic examination, select an endoscopic image including the lesion from among the plurality of endoscopic images as an image to be attached to the report, and, when an endoscopic image including a lesion is not taken during the endoscopic examination, select an endoscopic image not including a lesion from among the plurality of endoscopic images as an image to be attached to the report based on a priority set for the additional information.

[0007] Another aspect of the report creation support method of the present invention acquires multiple endoscopic images taken during an endoscopic examination, acquires additional information attached to the multiple endoscopic images, including information indicating whether a lesion has been detected, and, when an endoscopic image including a lesion has been taken during the endoscopic examination, selects from the multiple endoscopic images an endoscopic image including the lesion as an image to be attached to the report, and when an endoscopic image including a lesion has not been taken during the endoscopic examination, selects from the multiple endoscopic images an endoscopic image not including the lesion as an image to be attached to the report based on a priority set for the additional information.

[0008] Another aspect of the information processing device of the present invention includes one or more processors having hardware, which acquire a plurality of endoscopic images taken during an endoscopic examination, acquire additional information attached to the plurality of endoscopic images including information indicating whether a lesion has been detected, and, when an endoscopic image including a lesion is taken during the endoscopic examination, select an endoscopic image including the lesion from the plurality of endoscopic images as an image to be attached to the report, and, when an endoscopic image including a lesion is not taken during the endoscopic examination, select an endoscopic image not including a lesion from the plurality of endoscopic images as an image to be attached to the report based on a priority set for the additional information.

[0009] Any combination of the above components, and conversion of the present disclosure into a method, device, system, recording medium, computer program, etc., are also valid aspects of the present disclosure. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating a configuration of a medical support system according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating functional blocks of a server device. [Figure 3] 10A and 10B are diagrams illustrating examples of additional information associated with an endoscopic image. [Figure 4] FIG. 2 is a diagram illustrating functional blocks of the information processing device. [Figure 5]10(a) to 10(c) are diagrams showing examples of priorities set for organs. [Figure 6] FIG. 10 is a diagram showing an example of priorities set for organ parts. [Figure 7] FIG. 10 is a diagram showing priorities determined from organ ranks and site ranks. [Figure 8] 10(a) to 10(c) are diagrams showing examples of priorities set in relation to the order in which images were taken. [Figure 9] FIG. 10 is a diagram illustrating an example of an automatically generated report. [Figure 10] FIG. 10 is a diagram showing a modified example of priorities set for combinations of organs and parts. [Figure 11] FIG. 10 is a diagram showing a modified example of priorities set for combinations of organs and parts. DETAILED DESCRIPTION OF THE INVENTION

[0011] FIG. 1 shows the configuration of a medical support system 1 according to an embodiment. The medical support system 1 is installed in a medical facility such as a hospital where endoscopic examinations are performed. In the medical support system 1, a server device 2, an image analysis device 3, an image storage device 8, an endoscope system 9, and a terminal device 10b are communicably connected via a network 4 such as a LAN (local area network). The endoscope system 9 is installed in an examination room and includes an endoscopic observation device 5 and a terminal device 10a. In the medical support system 1, the server device 2, the image analysis device 3, and the image storage device 8 may be installed outside the medical facility, for example, as a cloud server.

[0012] An endoscope 7, which is inserted into the patient's digestive tract, is connected to the endoscopic observation device 5. The endoscope 7 has a light guide for transmitting illumination light supplied from the endoscopic observation device 5 to illuminate the inside of the digestive tract, and is provided at its tip with an illumination window for emitting the illumination light transmitted by the light guide to living tissue, and an imaging unit for capturing images of the living tissue at a predetermined cycle and outputting image signals to the endoscopic observation device 5. The imaging unit includes a solid-state imaging element (for example, a CCD image sensor or a CMOS image sensor) that converts incident light into an electrical signal.

[0013] The endoscopic observation device 5 performs image processing on the image signal photoelectrically converted by the solid-state image sensor of the endoscope 7 to generate an endoscopic image, which is displayed in real time on the display device 6. The endoscopic observation device 5 may have a function for performing special image processing for purposes such as highlighting, in addition to normal image processing such as A / D conversion and noise removal. The endoscopic observation device 5 generates endoscopic images at a predetermined cycle (for example, 1 / 60 seconds). The endoscopic observation device 5 may be configured with one or more processors having dedicated hardware, or may be configured with one or more processors having general-purpose hardware.

[0014] The doctor observes the endoscopic image displayed on the display device 6 in accordance with the examination procedure. The doctor observes the endoscopic image while moving the endoscope 7, and when the biological tissue to be captured is displayed on the display device 6, he operates the release switch of the endoscope 7. When the release switch is operated, the endoscopic observation device 5 captures (saves) the endoscopic image at the timing when the release switch is operated, and transmits the captured endoscopic image to the image storage device 8 together with information identifying the endoscopic image (image ID). Note that the endoscopic observation device 5 may transmit multiple captured endoscopic images together to the image storage device 8 after the examination is completed. The image storage device 8 records the endoscopic image transmitted from the endoscopic observation device 5, linking it to the examination ID that identifies the endoscopic examination.

[0015] In the embodiments, "photographing" refers to the operation of the solid-state image sensor of the endoscope 7 converting incident light into an electrical signal, and "capture" refers to the operation of saving (recording) an endoscopic image generated by the endoscopic observation device 5. "Photographing" may also include the operation of the endoscopic observation device 5 generating an endoscopic image from the converted electrical signal.

[0016] The terminal device 10a includes an information processing device 11a and a display device 12a and is installed in an examination room. The terminal device 10a may be used by a doctor, a nurse, or the like to check information about the biological tissue being photographed in real time during an endoscopic examination.

[0017] The terminal device 10b is equipped with an information processing device 11b and a display device 12b and is installed in a room other than the examination room. The terminal device 10b is used by a doctor to create a report on an endoscopic examination. The terminal devices 10a and 10b are configured with one or more processors having general-purpose hardware.

[0018] In the medical support system 1 of the embodiment, the endoscopic observation device 5 displays endoscopic images in real time on the display device 6, and also supplies the endoscopic images together with meta information of the images in real time to the image analysis device 3. Here, the meta information includes at least the frame number of the image and information on the time of shooting, and the frame number may be information indicating the number of the frame since the endoscope 7 started shooting.

[0019] The image analysis device 3 is an electronic computer that analyzes endoscopic images, detects lesions contained in the endoscopic images, and qualitatively diagnoses the detected lesions. The image analysis device 3 may be a CAD (computer-aided diagnosis) system with AI (artificial intelligence) diagnostic functions. The image analysis device 3 may be configured with one or more processors having dedicated hardware, or may be configured with one or more processors having general-purpose hardware.

[0020] The image analysis device 3 uses a trained model generated by machine learning using training endoscopic images and information about lesion areas contained in the endoscopic images as training data. Annotation of endoscopic images is performed by an annotator with specialized knowledge, such as a physician. Deep learning techniques such as CNN, RNN, and LSTM may be used for the machine learning. When an endoscopic image is input, this trained model outputs information indicating the imaged organ, information indicating the imaged region, and information about the imaged lesion (lesion information). The lesion information output by the image analysis device 3 includes at least lesion presence / absence information indicating whether or not a lesion is present in the endoscopic image. If a lesion is present, the lesion information may include information indicating the size of the lesion, information indicating the location of the lesion's outline, information indicating the lesion's shape, information indicating the lesion's depth, and a qualitative diagnosis of the lesion. The qualitative diagnosis of the lesion includes the type of lesion. During an endoscopic examination, the image analysis device 3 receives endoscopic images in real time from the endoscopic observation device 5 and outputs information indicating the organ, information indicating the region, and lesion information for each endoscopic image.

[0021] The image analysis device 3 has a function of recognizing the image quality of an endoscopic image. The image analysis device 3 uses image analysis to check whether the image is blurred, shaken, cloudy, or contains residue, and determines whether the image has been captured appropriately. For each endoscopic image, the image analysis device 3 outputs image quality information indicating whether the image has been captured appropriately. Hereinafter, the information indicating the organ, information indicating the site, lesion information, and image quality information output for each endoscopic image will be collectively referred to as "image analysis information."

[0022] When the user operates the release switch (capture operation), the endoscopic observation device 5 provides the frame number, shooting time, and image ID of the captured endoscopic image to the image analysis device 3, along with information indicating that the capture operation has been performed (capture operation information). Upon acquiring the capture operation information, the image analysis device 3 provides the server device 2 with the examination ID, as well as the image ID, frame number, shooting time information, and image analysis information of the provided frame number. Here, the image ID, frame number, shooting time information, and image analysis information constitute "additional information" that expresses the features and properties of the endoscopic image. Upon acquiring the capture operation information, the image analysis device 3 transmits the additional information to the server device 2 along with the examination ID, and the server device 2 records the additional information by linking it to the examination ID.

[0023] When the user finishes the endoscopic examination, he or she operates the examination end button on the endoscopic observation device 5. Information about the operation of the examination end button is supplied to the server device 2 and the image analysis device 3, which then recognize the end of the endoscopic examination. Upon receiving the examination end information, the image analysis device 3 provides the server device 2 with information indicating whether or not an endoscopic image containing a lesion to be attached to the report has been detected during the endoscopic examination, along with the examination ID. Hereinafter, information indicating whether or not an endoscopic image containing a lesion to be attached to the report has been detected will be referred to as "lesion image presence / absence information." For example, if the image analysis device 3 detects at least one endoscopic image containing a lesion during the examination, it may provide the server device 2 with lesion image presence / absence information indicating that an endoscopic image to be attached to the report has been detected. On the other hand, if the image analysis device 3 does not detect any endoscopic images containing a lesion during the examination, it may provide the server device 2 with lesion image presence / absence information indicating that an endoscopic image to be attached to the report has not been detected.

[0024] 2 shows functional blocks of the server device 2. The server device 2 includes a communication unit 20, a processing unit 30, and a storage device 60. The communication unit 20 transmits and receives information such as data and instructions to and from the image analysis device 3, the endoscopic observation device 5, the image storage device 8, the terminal device 10a, and the terminal device 10b via the network 4. The processing unit 30 includes an order information acquisition unit 40, an additional information acquisition unit 42, and a lesion image presence / absence information acquisition unit 44. The storage device 60 includes an order information storage unit 62, an additional information storage unit 64, and a lesion image presence / absence information storage unit 66.

[0025] The server device 2 includes a computer, which executes programs to realize the various functions shown in FIG. 2. The computer includes hardware such as a memory into which the programs are loaded, one or more processors that execute the loaded programs, an auxiliary storage device, and other LSIs. The processor is composed of multiple electronic circuits including semiconductor integrated circuits and LSIs, and the multiple electronic circuits may be mounted on a single chip or multiple chips. The functional blocks shown in FIG. 2 are realized by cooperation between hardware and software. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various ways using hardware alone, software alone, or a combination thereof.

[0026] The order information acquiring unit 40 acquires order information for an endoscopic examination from the hospital information system. For example, before the start of the day's examination work at the medical facility, the order information acquiring unit 40 acquires order information for the day from the hospital information system and stores it in the order information storage unit 62. Before the start of the examination, the endoscopic observation device 5 or the information processing device 11a may read out the order information for the examination to be performed from the order information storage unit 62 and display it on the display device.

[0027] The additional information acquisition unit 42 acquires the examination ID and additional information of the endoscopic image from the image analysis device 3, and stores the additional information in the additional information storage unit 64 in association with the examination ID. The additional information of the endoscopic image includes the image ID, frame number, shooting time information, and image analysis information. For example, if the user performs a capture operation 17 times during an endoscopic examination, the additional information acquisition unit 42 acquires additional information for 17 endoscopic images from the image analysis device 3 and stores it in the additional information storage unit 64. Note that the image IDs are assigned by the endoscopic observation device 5, which assigns image IDs in order of shooting time, starting from 1. Therefore, in this case, image IDs 1 to 17 are assigned to the 17 endoscopic images, respectively.

[0028] Lesion image presence / absence information acquisition unit 44 acquires the examination ID and lesion image presence / absence information from image analysis device 3, and stores the lesion image presence / absence information linked to the examination ID in lesion image presence / absence information storage unit 66. Lesion image presence / absence information acquisition unit 44 may acquire the lesion image presence / absence information from image analysis device 3 after the examination is completed.

[0029] In the embodiment, a technology is proposed in which, when an endoscopic examination is not performed by the image analysis device 3, the information processing device 11b automatically selects an endoscopic image to be attached to the examination report. Therefore, in the embodiment, after the examination is completed, the lesion image presence / absence information acquisition unit 44 acquires, from the image analysis device 3, lesion image presence / absence information indicating that an endoscopic image containing a lesion to be attached to the report was not detected.

[0030] 3 shows an example of additional information linked to a captured endoscopic image. The "organ" field stores information indicating the organ contained in the image, i.e., information indicating the organ photographed, and the "site" field stores information indicating the site of the organ contained in the image, i.e., information indicating the site of the organ photographed.

[0031] In the lesion information, the "presence / absence" field stores information indicating whether or not a lesion has been detected by the image analyzing device 3. As described above, in the embodiment, the image analyzing device 3 has not detected any lesions in the endoscopic examination, and therefore "none" is stored in the "presence / absence" field for all of the endoscopic images with image IDs 1 to 17.

[0032] The "size" field stores information indicating the longest diameter of the bottom surface of the lesion, the "shape" field stores coordinate information representing the contour shape of the lesion, and the "diagnosis" field stores the qualitative diagnosis result of the lesion. In this embodiment, since no lesions are included in any of the endoscopic images, each field is set to blank.

[0033] The "image quality" field stores information indicating whether the image was captured properly, with "good" indicating that the image was captured properly and "poor" indicating that the image was not captured properly. The "capture time" field stores information indicating the capture time of the image. The "capture time" field may also include a frame number.

[0034] FIG. 4 shows functional blocks of the information processing device 11b. The information processing device 11b has a function of automatically selecting endoscopic images to be attached to a report of an examination in which no lesion was detected, and includes a communication unit 76, an input unit 78, a processing unit 80, and a storage device 120. The communication unit 76 transmits and receives information such as data and instructions between the server device 2, the image analysis device 3, the endoscopic observation device 5, the image storage device 8, and the terminal device 10a via the network 4. The processing unit 80 includes an operation reception unit 82, an acquisition unit 84, an assistance mode setting unit 98, a priority setting unit 100, an image selection unit 102, a report generation unit 104, and a registration processing unit 106. The acquisition unit 84 includes an image acquisition unit 86, an additional information acquisition unit 88, and a lesion image presence / absence information acquisition unit 90. The storage device 120 includes an image storage unit 122, an additional information storage unit 124, and a priority storage unit 126.

[0035] The information processing device 11b includes a computer, which executes a program to realize various functions shown in FIG. 4. The computer includes hardware such as a memory into which the program is loaded, one or more processors that execute the loaded program, an auxiliary storage device, and other LSIs. The processor is composed of multiple electronic circuits including semiconductor integrated circuits and LSIs, and the multiple electronic circuits may be mounted on a single chip or multiple chips. The functional blocks shown in FIG. 4 are realized by cooperation between hardware and software. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms using hardware alone, software alone, or a combination thereof.

[0036] After the endoscopic examination is completed, the doctor user logs in to the information processing device 11b by entering his or her user ID and password. When the user logs in, an application for creating an examination report is launched, and a list of performed examinations is displayed on the display device 12b. This list of performed examinations displays examination information such as the patient name, patient ID, examination date and time, and examination items. The user operates the input unit 78, such as a mouse or keyboard, to select the examination for which a report is to be created. When the operation reception unit 82 receives an examination selection operation, the image acquisition unit 86 acquires from the image storage device 8 multiple endoscopic images associated with the examination ID of the examination selected by the user and stores them in the image storage unit 122. The additional information acquisition unit 88 acquires from the server device 2 additional information associated with the examination ID of the examination selected by the user and stores it in the additional information storage unit 124. The lesion image presence / absence information acquisition unit 90 acquires from the server device 2 lesion image presence / absence information associated with the examination ID of the examination selected by the user. The lesion image presence / absence information is information indicating whether or not the image analysis device 3 has detected an endoscopic image containing a lesion that should be attached to the report during the endoscopic examination.

[0037] The assistance mode setting unit 98 sets a mode in which the image selection unit 102 automatically selects images to be attached to the report according to the lesion image presence / absence information. When the image analysis device 3 does not detect a lesion during the endoscopic examination, the assistance mode setting unit 98 sets a first assistance mode in which the image selection unit 102 selects images to be attached to the report based on a priority order. In the first assistance mode, the image selection unit 102 selects a predetermined amount of images to be attached to the report from among the multiple captured endoscopic images based on a priority order set for the additional information. Here, the predetermined amount of images to be attached to the report may mean a predetermined number of still images if the captured endoscopic images are still images, or may mean a predetermined amount (or a predetermined duration) of moving images if the captured endoscopic images are moving images.

[0038] On the other hand, when the image analyzing device 3 detects a lesion during the endoscopic examination, the assistance mode setting unit 98 sets a second assistance mode in which the image selecting unit 102 selects an endoscopic image including the lesion. In the second assistance mode, the image selecting unit 102 selects an image to be attached to the report that includes the lesion detected by the image analyzing device 3 from among the multiple captured endoscopic images.

[0039] In the following embodiment, the operation of the processing unit 80 when the lesion image presence / absence information acquired by the lesion image presence / absence information acquisition unit 90 indicates that the image analysis device 3 has not detected a lesion and the assistance mode setting unit 98 sets the first assistance mode will be described. During the examination, the user captures an endoscopic image, which is a still image.

[0040] In the first assistance mode, the image selection unit 102 automatically selects images to be attached to a report based on priorities set for additional information. Priorities may be set for at least one of organs, body parts, and imaging timing. The priority setting unit 100 sets priorities based on user input and stores them in the priority storage unit 126. For example, the user may select one priority from multiple priority candidate sets. By allowing the user to determine the criteria for automatically selecting images to be attached to a report in lesion-undetected examinations, a report can be generated that reflects the preferences of the user and hospital facility. The user may also set the upper limit (upper limit amount) of images to be attached to a report.

[0041] FIG. 5(a) shows an example of priorities set for organs. In the priorities shown in FIG. 5(a), the "esophagus" has the first priority, the "stomach" has the second priority, and the "duodenum" has the third priority. The image selection unit 102 selects report-attached images from multiple endoscopic images captured by the user according to the priorities, within the upper limit of the number of report-attached images. The image selection unit 102 references information indicating the organs included in the additional information of each endoscopic image, and selects report-attached images one by one in order from the highest priority, and after selecting the report-attached image with the lowest priority, selects report-attached images again in order from the highest priority.

[0042] For example, if the upper limit number of report attachment images is set to "5," the image selection unit 102 selects the report attachment image including the esophagus as the first image, the report attachment image including the stomach as the second image, the report attachment image including the duodenum as the third image, the report attachment image including the esophagus as the fourth image, and the report attachment image including the stomach as the fifth image. As described above, the image selection unit 102 of the embodiment selects report attachment images based on the set priority order within the range of the upper limit number.

[0043] 5(b) shows another example of priorities set for organs. In the priorities shown in FIG. 5(b), the "stomach" has the first priority, the "duodenum" has the second priority, and the "esophagus" has the third priority. When the priorities shown in FIG. 5(b) are set and the upper limit number of report attached images is set to "5," the image selection unit 102 selects a report attached image including the stomach as the first image, a report attached image including the duodenum as the second image, a report attached image including the esophagus as the third image, a report attached image including the stomach as the fourth image, and a report attached image including the duodenum as the fifth image.

[0044] 5(c) shows another example of priorities set for organs. In the priorities shown in FIG. 5(c), the "stomach" has the first priority, the "esophagus" has the second priority, and the "duodenum" has the third priority. When the priorities shown in FIG. 5(c) are set and the upper limit number of report attached images is set to "5," the image selection unit 102 selects the report attached image including the stomach as the first image, the report attached image including the esophagus as the second image, the report attached image including the duodenum as the third image, the report attached image including the stomach as the fourth image, and the report attached image including the esophagus as the fifth image.

[0045] As described above, the priority setting unit 100 sets priorities based on user input. The image selection unit 102 selects images to be attached to a report according to the priorities set by the user, allowing the report generation unit 104, described later, to generate a report that meets the user's needs. Note that the priorities for organs shown in Figures 5(a) to 5(c) are merely examples, and the user may freely set the priorities for organs.

[0046] FIG. 6 shows an example of priorities set for organ parts. These priorities determine the priority of parts in each organ. According to the priorities shown in FIG. 6, in the "esophagus," the "cervical region" has the first priority, the "thorax" has the second priority, and the "abdomen" has the third priority. In the "stomach," the "antrum" has the first priority, the "angular region" has the second priority, and the "body" has the third priority. In the "duodenum," the "descending region" has the first priority, the "bulb" has the second priority, and the "minor papilla" has the third priority. Note that the priorities shown in FIG. 6 are merely an example, and the user may arbitrarily set the priority of parts in each organ.

[0047] The image selection unit 102 may select an image to be attached to a report based on the priority order set for organs (for example, the priority order shown in any one of FIGS. 5(a) to 5(c)) and the priority order set for regions. The following describes the operation of the image selection unit 102 to select an image to be attached to a report based on the priority order for organs shown in FIG. 5(a) (hereinafter simply referred to as "organ order") and the priority order for regions shown in FIG. 6 (hereinafter simply referred to as "region order"). The image selection unit 102 identifies an endoscopic image including an organ identified by the organ order shown in FIG. 5(a), and when multiple endoscopic images including the organ have been acquired, selects an image to be attached to a report from the multiple endoscopic images including the organ based on the region order shown in FIG. 6.

[0048] The image selection unit 102 determines the priority order of the combination of organ and region from which the report attachment image is to be selected, based on the organ order shown in FIG. 5(a) and the region order shown in FIG. (Ranking step 1) When the image selection unit 102 recognizes from the organ ranking shown in Figure 5(a) that the first image to be selected as an attached report is an image including the "esophagus," it recognizes from the part ranking shown in Figure 6 that the "cervix" has the highest priority within the "esophagus," and determines that the combination of (esophagus, cervix) has the highest priority.

[0049] (Ranking step 2) When the image selection unit 102 recognizes from the organ ranking shown in Figure 5(a) that the second report attachment image to be selected is an image containing the "stomach," it recognizes from the site ranking shown in Figure 6 that the "vestibule" has the highest priority within the "stomach," and determines that the combination of (stomach, vestibule) has the second highest priority.

[0050] (Ranking step 3) When the image selection unit 102 recognizes from the organ ranking shown in Figure 5(a) that the third report attachment image to be selected is an image containing the "duodenum," it recognizes from the site ranking shown in Figure 6 that the "descending portion" has the highest priority within the "duodenum," and determines that the combination of (duodenum, descending portion) has the third highest priority.

[0051] (Ranking step 4) When the image selection unit 102 recognizes from the organ ranking shown in Figure 5(a) that the fourth report attachment image to be selected is an image including the "esophagus," it recognizes from the part ranking shown in Figure 6 that the "chest" has the second priority within the "esophagus," and determines that the combination of (esophagus, chest) has the fourth priority.

[0052] (Ranking step 5) When the image selection unit 102 recognizes from the organ ranking shown in Figure 5(a) that the fifth report attachment image to be selected is an image containing the "stomach," it recognizes from the part ranking shown in Figure 6 that the "corner" has the second priority within the "stomach," and determines that the combination of (stomach, corner) has the fifth priority.

[0053] (Ranking step 6) When the image selection unit 102 recognizes from the organ ranking shown in Figure 5(a) that the sixth report attachment image to be selected is an image containing the "duodenum," it recognizes from the site ranking shown in Figure 6 that the "bulb" has the second priority within the "duodenum," and determines that the combination of (duodenum, bulb) has the sixth priority.

[0054] (Ranking step 7) When the image selection unit 102 recognizes from the organ ranking shown in Figure 5(a) that the seventh report attachment image to be selected is an image including the "esophagus," it recognizes from the site ranking shown in Figure 6 that the "abdomen" has the third priority among the "esophagus," and determines that the combination of (esophagus, abdomen) has the seventh priority.

[0055] (Ranking step 8) When the image selection unit 102 recognizes from the organ ranking shown in Figure 5(a) that the eighth report attachment image to be selected is an image containing the "stomach," it recognizes from the part ranking shown in Figure 6 that the priority of "body" within the "stomach" is third, and determines that the combination of (stomach, body) is eighth in priority.

[0056] (Ranking step 9) When the image selection unit 102 recognizes from the organ ranking shown in Figure 5(a) that the ninth report attachment image to be selected is an image containing the "duodenum," it recognizes from the site ranking shown in Figure 6 that the priority of the "minor papilla" within the "duodenum" is third, and determines that the combination of (duodenum, minor papilla) is ninth in priority.

[0057] FIG. 7 shows the priority order determined from the organ order shown in FIG. 5(a) and the site order shown in FIG. When the upper limit number is five, the image selection unit 102 selects, according to the priority order shown in Fig. 7, an image attached to the report including the neck of the esophagus as the first image, an image attached to the report including the antrum of the stomach as the second image, an image attached to the report including the descending portion of the duodenum as the third image, an image attached to the report including the thoracic portion of the esophagus as the fourth image, and an image attached to the report including the angle of the stomach as the fifth image. As described above, the image selection unit 102 may select an image attached to the report based on the priority order derived for organs and parts within the range of the upper limit number.

[0058] If there is no endoscopic image that includes the combination of (organ, region) in the additional information, the image selection unit 102 selects a report attached image that includes a region of the same organ with a lower priority. For example, when the image selection unit 102 attempts to select a report attached image that includes the descending portion of the duodenum, if there is no endoscopic image that includes the descending portion of the duodenum, the image selection unit 102 selects a report attached image that includes the bulb of the duodenum. If there is also no endoscopic image that includes the bulb of the duodenum, the image selection unit 102 selects a report attached image that includes the minor papilla of the duodenum. In this way, the image selection unit 102 can select report attached images for each organ evenly by selecting report attached images based on the organ order.

[0059] 8(a) shows an example of a priority order set for the order of photographing. When multiple endoscopic images including the same organ or the same part are acquired, the priority order set for the order of photographing is used to select one image to attach to a report from the multiple endoscopic images including the organ or part.

[0060] The priority setting unit 100 sets, as the priority order regarding the order of photographing, one of the following: a first priority criterion that gives priority to selecting an endoscopic image photographed at the first timing, a second priority criterion that gives priority to selecting an endoscopic image photographed at an intermediate timing, or a third priority criterion that gives priority to selecting an endoscopic image photographed at the last timing. In the example shown in Fig. 8(a), the first priority criterion is set.

[0061] For example, if the priorities for image selection are set as the organ order shown in Figure 5(a) and the shooting order order shown in Figure 8(a), the image selection unit 102 refers to the information indicating the shooting time included in the additional information of the endoscopic image including the esophagus, and selects the endoscopic image including the esophagus that was shot earliest as the first image to be attached to the report.

[0062] Fig. 8(b) shows another example of priorities set for the order of photographing. In the example shown in Fig. 8(b), a second priority criterion is set. When the second priority criterion is set, the image selection unit 102 selects, from among a plurality of endoscopic images, an endoscopic image taken at an intermediate timing in the photographing order as an image to be attached to the report.

[0063] 8(c) shows another example of priorities set in relation to the order of capture. In the example shown in FIG. 8(c), a third priority criterion is set. When the third priority criterion is set, the image selection unit 102 selects the latest endoscopic image from among multiple endoscopic images as the image to be attached to the report. In this way, the image selection unit 102 can narrow down the image to be attached to the report to one image by taking into account the priority of the order of capture.

[0064] The following describes the procedure by which the image selection unit 102 selects images to be attached to a report based on the organ order shown in Fig. 5(a), the site order shown in Fig. 6, and the imaging order priority shown in Fig. 8(c). The upper limit for the number of images to be attached to a report is set to "5." Seventeen endoscopic images were acquired during the examination, and the additional information acquisition unit 88 acquires additional information for the endoscopic images shown in Fig. 3 and stores it in the additional information storage unit 124.

[0065] (Steps for selecting the image to attach to the first report) When the image selection unit 102 determines from the organ ranking and site ranking that the combination of (esophagus, cervix) has the highest priority, it refers to the additional information of each endoscopic image and searches for an endoscopic image having image analysis information of (esophagus, cervix). Referring to Figure 3, the endoscopic image with image ID 1 matches this, and therefore the image selection unit 102 selects the endoscopic image with image ID 1 as the first image to be attached to the report.

[0066] (Steps for selecting the image to attach to the second report) When the image selection unit 102 determines from the organ ranking and the site ranking that the combination of (stomach, vestibule) has second priority, it refers to the additional information of each endoscopic image and searches for endoscopic images having image analysis information of (stomach, vestibule). Referring to FIG. 3, two endoscopic images with image IDs 13 and 14 fall into this category. Since the priority order of imaging order shown in FIG. 8(c) is set to the third priority criterion, which gives priority to selecting the endoscopic image captured last, the image selection unit 102 selects the endoscopic image with image ID 14, which was captured last, as the second image to be attached to the report.

[0067] (Steps for selecting the image to attach to the third report) When the image selection unit 102 determines from the organ ranking and site ranking that the combination of (duodenum, descending portion) has third priority, it refers to the additional information of each endoscopic image and searches for an endoscopic image having image analysis information of (duodenum, descending portion). Referring to Figure 3, since there is no endoscopic image having image analysis information of (duodenum, descending portion), the image selection unit 102 searches for an endoscopic image having image analysis information of the "bulb" of the duodenum, which is one site lower in site ranking. In Figure 3, the endoscopic image with image ID 15 corresponds.

[0068] The image selection unit 102 references the "Image Quality" field associated with image ID 15 and determines that "poor," indicating inappropriate imaging, is stored. The image selection unit 102 then determines not to select the endoscopic image with image ID 15 as an image to be attached to the report. Therefore, the image selection unit 102 searches for an endoscopic image with image analysis information for the "minor papilla," which is one site lower in the duodenum ranking. Referring to FIG. 3, two endoscopic images with image IDs 16 and 17 are found to be relevant. Because the imaging order priority shown in FIG. 8(c) is set to the third priority standard, the image selection unit 102 selects the endoscopic image with image ID 17, which was captured later, as the third image to be attached to the report.

[0069] (Steps for selecting the image to attach to the fourth report) When the image selection unit 102 determines that the combination of (esophagus, chest) has the fourth priority based on the organ ranking and the site ranking, it refers to the additional information of each endoscopic image and searches for endoscopic images that have image analysis information of (esophagus, chest). Referring to Figure 3, two endoscopic images with image IDs 2 and 3 match this.

[0070] The image selection unit 102 references the "image quality" field associated with image IDs 2 and 3 and recognizes that "poor" is stored in the "image quality" field for image ID 2. Therefore, the image selection unit 102 determines not to select the endoscopic image with image ID 2 as an image to be attached to the report, and selects the endoscopic image with image ID 3 as the fourth image to be attached to the report.

[0071] (5th report attachment image selection procedure) When the image selection unit 102 determines that the combination of (stomach, angulus) is ranked fifth based on the organ ranking and the site ranking, it refers to the additional information of each endoscopic image and searches for endoscopic images that have image analysis information of (stomach, angulus). Referring to Figure 3, four endoscopic images with image IDs 9, 10, 11, and 12 match this criteria.

[0072] The image selection unit 102 refers to the "image quality" field associated with image IDs 9, 10, 11, and 12, and when it recognizes that "poor" is stored in the "image quality" field for image ID 9, it decides not to select the endoscopic image with image ID 9 as an image to be attached to the report. Next, based on the priority order of image capture shown in Figure 8(c), the image selection unit 102 selects the endoscopic image with image ID 12, which was captured last, as the fifth image to be attached to the report.

[0073] Through the above selection procedure, the image selection unit 102 selects up to five images to attach to the report. By allowing the user to set the priority order in advance, endoscopic images that reflect the user's preferences will be automatically attached to the report.

[0074] 9 shows an example of an automatically generated report. When the assistance mode setting unit 98 sets the first assistance mode, the image selection unit 102 selects an image to attach to the report based on the priority set for the additional information, as described above, and the report generation unit 104 automatically generates a report using the selected image to attach to the report.

[0075] 10 shows an example of an automatically generated report. The report generation unit 104 generates a report including information indicating that no lesions are found and the report attachment image selected by the image selection unit 102, and displays the report on the display device 12b. When the image selection unit 102 selects the report attachment image, the report generation unit 104 may generate a report including the contents of the test order, the diagnosis indicating that no abnormal findings are found, and the report attachment image. The user may check the report displayed on the display device 12b and operate a predetermined registration button (not shown), which causes the registration processing unit 106 to register the automatically generated report in the server device 2.

[0076] The present disclosure has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of the respective components and the respective treatment processes, and that such modifications are also within the scope of the present disclosure.

[0077] FIG. 10 shows a modified example of the priority order set for combinations of organs and regions. In the embodiment, the priority order is set by combining the organ order shown in FIGS. 5(a) to 5(c) and the region order shown in FIGS. 6(a) to 6(b), but the priority order shown in FIG. 10 is determined for combinations of organs and regions. Below, the procedure by which the image selection unit 102 selects images to be attached to a report based on the priority order shown in FIG. 10 and the priority order of imaging order shown in FIG. 8(c) will be described. The upper limit of the number of images to be attached to a report is set to "5." The additional information acquisition unit 88 acquires additional information for the endoscopic image shown in FIG. 3 and stores it in the additional information storage unit 124.

[0078] (Steps for selecting the image to attach to the first report) When the image selection unit 102 recognizes that the combination of (esophagus, cervix) has the highest priority, it refers to the additional information of each endoscopic image and searches for an endoscopic image having image analysis information of (esophagus, cervix). Referring to Figure 3, the endoscopic image with image ID 1 matches this, and therefore the image selection unit 102 selects the endoscopic image with image ID 1 as the first image to be attached to the report.

[0079] (Steps for selecting the image to attach to the second report) When the image selection unit 102 recognizes that the combination of (esophagus, chest) has second priority, it refers to the additional information of each endoscopic image and searches for endoscopic images that have image analysis information of (esophagus, chest). Referring to Figure 3, two endoscopic images with image IDs 2 and 3 match this.

[0080] The image selection unit 102 references the "image quality" field associated with image IDs 2 and 3 and recognizes that "poor" is stored in the "image quality" field for image ID 2. Therefore, the image selection unit 102 determines not to select the endoscopic image with image ID 2 as an image to be attached to the report, and selects the endoscopic image with image ID 3 as the second image to be attached to the report.

[0081] (Steps for selecting the image to attach to the third report) When the image selection unit 102 recognizes that the combination of (esophagus, abdomen) has the third priority, it refers to the additional information of each endoscopic image and searches for endoscopic images having image analysis information of (esophagus, abdomen). Referring to FIG. 3, two endoscopic images with image IDs 4 and 5 fall into this category. Since the priority order of imaging order shown in FIG. 8(c) is set to the third priority standard, which gives priority to selecting the endoscopic image captured last, the image selection unit 102 selects the endoscopic image with image ID 5, which was captured last, as the third image to be attached to the report.

[0082] (Steps for selecting the image to attach to the fourth report) When the image selection unit 102 recognizes that the combination of (stomach, body) has the fourth highest priority, it refers to the additional information of each endoscopic image and searches for endoscopic images that have image analysis information of (stomach, body). Referring to FIG. 3, three endoscopic images with image IDs 6, 7, and 8 fall into this category. Since the priority of the shooting order shown in FIG. 8(c) is set to the third priority standard, the image selection unit 102 selects the endoscopic image with image ID 8, which was shot last, as the fourth image to be attached to the report.

[0083] (5th report attachment image selection procedure) When the image selection unit 102 determines that the combination of (stomach, angulus) is ranked fifth in priority, it refers to the additional information of each endoscopic image and searches for endoscopic images that have image analysis information of (stomach, angulus). Referring to Figure 3, four endoscopic images with image IDs 9, 10, 11, and 12 match this criteria.

[0084] The image selection unit 102 refers to the "image quality" field associated with image IDs 9, 10, 11, and 12, and when it recognizes that "poor" is stored in the "image quality" field for image ID 9, it decides not to select the endoscopic image with image ID 9 as an image to be attached to the report. Next, based on the priority order of image capture shown in Figure 8(c), the image selection unit 102 selects the endoscopic image with image ID 12, which was captured last, as the fifth image to be attached to the report.

[0085] Through the above selection procedure, the image selection unit 102 selects up to five images to attach to the report. By allowing the user to set the priorities for all combinations of organs and parts in advance, endoscopic images that reflect the user's preferences will be automatically attached to the report.

[0086] Figure 11 shows a modified example of the priority order set for combinations of organs and body parts. In this example, the priority order of the timing of imaging is also associated with each combination of organs and body parts. By being able to set the priority order of imaging timing for each combination of organs and body parts in this way, it becomes possible to select images to attach to reports that reflect the user's preferences in more detail.

[0087] In the embodiment, it has been described that the user can set an upper limit (upper limit amount) of images to be attached to a report. The user may be able to select the upper limit from 0. By setting the upper limit to 0, when the assistance mode setting unit 98 sets the first assistance mode, the image selection unit 102 operates so as not to automatically select images to be attached to a report.

[0088] In the embodiment, the case where the endoscopic examination is an upper endoscopy has been described, but the endoscopic examination may also be a lower endoscopy. The priority setting unit 100 may set a priority for each examination item based on user input. For example, the priority setting unit 100 may set a priority regarding the timing of image capture as a third priority criterion for the report of an upper endoscopy, and set a priority regarding the timing of image capture as a second priority criterion for the report of a lower endoscopy. In this case, the image selection unit 102 selects images to be attached to the report of an upper endoscopy based on the third priority criterion, while selecting images to be attached to the report of a lower endoscopy based on the second priority criterion. Because the organs targeted in a lower endoscopy are long, images captured at intermediate times during a lower endoscopy are preferable as images to be attached to the report, representing the condition of the organ.

[0089] In the embodiment, the endoscopic observation device 5 transmits the captured image to the image storage device 8, but in a modified example, the image analysis device 3 may transmit the captured image to the image storage device 8. Furthermore, in the embodiment, the information processing device 11b has the processing unit 80, but in a modified example, the server device 2 may have the processing unit 80. [Explanation of symbols]

[0090] 1 Medical support system, 2 Server device, 3 Image analysis device, 4 Network, 5 Endoscopic observation device, 6 Display device, 7 Endoscope, 8 Image storage device, 9 Endoscopic system, 10a, 10b Terminal device, 11a, 11b Information processing device, 12a, 12b Display device, 20 Communication unit, 30 Processing unit, 40 Order information acquisition unit, 42 Additional information acquisition unit, 44 Lesion image presence / absence information acquisition unit, 60 Storage device, 62 Order information storage unit, 64 ···Additional information storage unit, 66···Lesion image presence / absence information storage unit, 76···Communication unit, 78···Input unit, 80···Processing unit, 82···Operation reception unit, 84···Acquisition unit, 86···Image acquisition unit, 88···Additional information acquisition unit, 90···Lesion image presence / absence information acquisition unit, 98···Assistance mode setting unit, 100···Priority setting unit, 102···Image selection unit, 104···Report generation unit, 106···Registration processing unit, 120···Storage device, 122···Image storage unit, 124···Additional information storage unit, 126···Priority order storage unit.

Claims

1. 1. A medical support system, comprising: one or more processors having hardware; The one or more processors: Acquire multiple endoscopic images taken during an endoscopic examination, acquiring additional information added to the plurality of endoscopic images, the additional information including information indicating whether a lesion has been detected; When an endoscopic image including a lesion is captured during an endoscopic examination, the endoscopic image including the lesion is selected from the plurality of endoscopic images as an image to be attached to the report; When an endoscopic image including a lesion is not captured during the endoscopic examination, an endoscopic image including no lesion is selected as an image to be attached to the report from among the plurality of endoscopic images based on the priority order set for the additional information. A medical support system characterized by:

2. The one or more processors: generating a report including the report attachment image; 2. The medical support system according to claim 1.

3. The one or more processors: setting the priority based on user input; 2. The medical support system according to claim 1.

4. The one or more processors: setting the priority for each test item based on a user input; 4. The medical support system according to claim 3.

5. the additional information includes information indicating an organ included in the endoscopic image, The one or more processors: selecting the images to be attached to the report based on a priority set for the organs; 2. The medical support system according to claim 1.

6. the additional information includes information indicating a region of an organ included in the endoscopic image, The one or more processors: selecting the report attachment images based on a set priority for organ parts; 6. The medical support system according to claim 5.

7. the additional information includes information indicating the order in which the endoscopic images were captured, The one or more processors: When a plurality of endoscopic images including the same organ or the same region are acquired, the image to be attached to the report is selected from the plurality of endoscopic images including the organ or the region based on a priority order set for the order in which they were captured.

2. The medical support system according to claim 1.

8. The one or more processors: a first priority criterion for preferentially selecting the endoscopic image captured at the first timing, a second priority criterion for preferentially selecting the endoscopic image captured at an intermediate timing, or a third priority criterion for preferentially selecting the endoscopic image captured at the last timing, is set as the priority order regarding the order of capturing; 8. The medical support system according to claim 7.

9. The one or more processors: When a plurality of endoscopic images including the same organ have been acquired, the image to be attached to the report is selected from the plurality of endoscopic images including the organ based on the priority order set for the part of the organ.

7. The medical support system according to claim 6.

10. The one or more processors: selecting an image to be attached to the report based on a priority order established for the combination of organ and part; 7. The medical support system according to claim 6.

11. The one or more processors: setting an upper limit for the number of images attached to the report based on user input; The user can select the upper limit from 0.

2. The medical support system according to claim 1.

12. Endoscopy includes upper endoscopy and lower endoscopy. The one or more processors: selecting the report attachment images of the upper endoscopy examination based on the third priority criterion; selecting the report attachment images of the lower endoscopy examination based on the second priority criterion; 9. The medical support system according to claim 8.

13. The one or more processors: The endoscopic image in which the additional information includes information indicating that the image was not properly captured is not selected as an image to be attached to the report.

2. The medical support system according to claim 1.

14. A report creation support method, comprising: Acquire multiple endoscopic images taken during an endoscopic examination, acquiring additional information added to the plurality of endoscopic images, the additional information including information indicating whether a lesion has been detected; When an endoscopic image including a lesion is captured during an endoscopic examination, the endoscopic image including the lesion is selected from the plurality of endoscopic images as an image to be attached to the report; When an endoscopic image including a lesion is not captured during the endoscopic examination, an endoscopic image including no lesion is selected as an image to be attached to the report from among the plurality of endoscopic images based on the priority order set for the additional information. A report creation support method comprising:

15. setting the priority based on user input; 15. The report creation support method according to claim 14.

16. On the computer, The function to acquire multiple endoscopic images taken during endoscopic examinations, a function of acquiring additional information added to the plurality of endoscopic images, the additional information including information indicating whether a lesion has been detected; a function of selecting an endoscopic image including a lesion as an image to be attached to a report from among a plurality of endoscopic images when the endoscopic image includes a lesion during an endoscopic examination; a function of selecting an endoscopic image that does not include a lesion as an image to be attached to a report from among the plurality of endoscopic images based on a priority order set for the additional information when no endoscopic image including a lesion is captured during an endoscopic examination; A program to achieve this.

17. The computer, realizing a function of setting the priority based on a user input; 17. The program according to claim 16.

18. An information processing device, comprising one or more processors having hardware, The one or more processors: Acquire multiple endoscopic images taken during an endoscopic examination, acquiring additional information added to the plurality of endoscopic images, the additional information including information indicating whether a lesion has been detected; When an endoscopic image including a lesion is captured during an endoscopic examination, the endoscopic image including the lesion is selected from the plurality of endoscopic images as an image to be attached to the report; When an endoscopic image including a lesion is not captured during the endoscopic examination, an endoscopic image including no lesion is selected as an image to be attached to the report from among the plurality of endoscopic images based on the priority order set for the additional information.

1. An information processing device comprising:

19. The one or more processors: setting the priority based on user input; 19. The information processing apparatus according to claim 18,

Images