Medical support system, report creation support method and information processing device

Abstract

An image acquisition unit 86 acquires user captured images that are captured through a capturing operation performed by a user, and computer captured images that are captured by a computer and that include a lesion. An image specification unit 102 specifies, as an attachment candidate image, a computer captured image that includes a lesion not included in a user captured image. A display screen generation unit 100 generates a selection screen that is for selecting an image to be attached to a report and that includes the user captured images and the attachment candidate images, and displays the selection screen on a display device 12b.

Description

[Technical field]

[0001] The present disclosure relates to a medical support system and a medical support method for supporting report creation. [Background technology]

[0002] In an endoscopic examination, a doctor observes an endoscopic image displayed on a display device, and when he finds a lesion, he operates the release switch of the endoscope to capture (save) an endoscopic image of the lesion. After the examination is completed, the doctor inputs the examination results into a report input screen, and selects an endoscopic image to be attached to the report from among the multiple captured endoscopic images, and creates a report. Patent Document 1 discloses a report input screen that displays a list of multiple captured endoscopic images as candidate images to be attached. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-86274 A Summary of the Invention [Problem to be solved by the invention]

[0004] Conventionally, the endoscopic images displayed in a list on the report input screen are limited to those captured by the doctor through the capture operation (release switch operation). Therefore, when the doctor performs the endoscopic examination in a short time so as not to burden the patient, it is not possible to attach images that the doctor forgot to capture to the report. It is possible to use a computer-aided diagnosis (CAD) system, which has been researched in recent years, to display images detected by the CAD system as attachment candidate images on the report input screen. However, if there are a large number of images detected by the CAD system, it will increase the effort for the doctor to select the images to attach to the report from the report input screen.

[0005] The present disclosure has been made in view of the above circumstances, and has as its objective to provide a medical support technology capable of efficiently displaying images captured by a computer such as a CAD system. [Means for solving the problem]

[0006] In order to solve the above problems, a medical support system according to an embodiment of the present invention includes one or more processors having hardware. The one or more processors acquire a first image captured by a user's capture operation and a computer-captured image including a lesion captured by a computer, identify a computer-captured image including a lesion not included in the first image as a second image, and generate a selection screen for selecting an image to be attached to a report, the selection screen including the first image and the second image.

[0007] Another aspect of the present invention is a medical support method, which includes obtaining a first image captured by a user's capture operation, obtaining a computer-captured image including a lesion captured by a computer, identifying a computer-captured image including a lesion not included in the first image as a second image, and generating a selection screen for selecting an image to be attached to a report, the selection screen including the first image and the second image.

[0008] Any combination of the above components, and conversion of the present disclosure into a method, device, system, recording medium, computer program, etc. are also valid aspects of the present disclosure. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing a configuration of a medical support system according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating functional blocks of the server device. [Diagram 3] FIG. 13 is a diagram showing an example of information associated with a capture image. [Figure 4] FIG. 2 is a diagram illustrating functional blocks of the information processing device. [Diagram 5] FIG. 13 is a diagram showing an example of a selection screen for selecting an endoscopic image. [Figure 6] FIG. 13 is a diagram showing an example of a report creation screen for inputting inspection results. [Figure 7] FIG. 13 is a diagram showing a table stored in a priority storage unit. [Figure 8] FIG. 13 is a diagram showing another example of a selection screen for selecting an endoscopic image. [Figure 9] FIG. 13 is a diagram showing an example of a selection screen when all computer-captured images are displayed. [Figure 10] FIG. 13 is a diagram showing another example of information associated with a captured image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] FIG. 1 shows a configuration of a medical support system 1 according to an embodiment. The medical support system 1 is provided in a medical facility such as a hospital where endoscopic examinations are performed. In the medical support system 1, a server device 2, an image analysis device 3, an image storage device 8, an endoscope system 9, and a terminal device 10b are communicatively connected via a network 4 such as a LAN (local area network). The endoscope system 9 is provided in an examination room, and has an endoscopic observation device 5 and a terminal device 10a. In the medical support system 1, the server device 2, the image analysis device 3, and the image storage device 8 may be provided outside the medical facility, for example, as a cloud server.

[0011] An endoscope 7 inserted into the digestive tract of a patient is connected to the endoscopic observation device 5. The endoscope 7 has a light guide for transmitting illumination light supplied from the endoscopic observation device 5 to illuminate the inside of the digestive tract, and is provided at its tip with an illumination window for emitting the illumination light transmitted by the light guide to the living tissue, and an imaging unit for photographing the living tissue at a predetermined period and outputting an imaging signal to the endoscopic observation device 5. The imaging unit includes a solid-state imaging element (e.g., a CCD image sensor or a CMOS image sensor) that converts incident light into an electrical signal.

[0012] The endoscopic observation device 5 performs image processing on the imaging signal photoelectrically converted by the solid-state imaging element of the endoscope 7 to generate an endoscopic image, which is displayed in real time on the display device 6. The endoscopic observation device 5 may have a function of performing special image processing for the purpose of highlighting or the like, in addition to normal image processing such as A / D conversion and noise removal. The endoscopic observation device 5 generates an endoscopic image at a predetermined cycle (for example, 1 / 60 seconds). The endoscopic observation device 5 may be configured with one or more processors having dedicated hardware, or may be configured with one or more processors having general-purpose hardware.

[0013] The doctor observes the endoscopic image displayed on the display device 6 according to the examination procedure. The doctor observes the endoscopic image while moving the endoscope 7, and operates the release switch of the endoscope 7 when a lesion is displayed on the display device 6. When the release switch is operated, the endoscopic observation device 5 captures (saves) the endoscopic image at the timing when the release switch is operated, and transmits the captured endoscopic image to the image storage device 8 together with information (image ID) that identifies the endoscopic image. Note that the endoscopic observation device 5 may transmit a plurality of captured endoscopic images together to the image storage device 8 after the examination is completed. The image storage device 8 records the endoscopic image transmitted from the endoscopic observation device 5 by linking it to an examination ID that identifies the endoscopic examination. The endoscopic images stored in the image storage device 8 are used by the doctor to create an examination report.

[0014] The terminal device 10a includes an information processing device 11a and a display device 12a, and is installed in an examination room. The terminal device 10a is used by doctors, nurses, etc. to check information about a lesion in real time during an endoscopic examination. The information processing device 11a acquires information about a lesion from the server device 2 and / or the image analysis device 3 during an endoscopic examination, and displays it on the display device 12a. For example, the size of the lesion, the depth of invasion of the lesion, and a qualitative diagnosis result of the lesion, etc., which are image-analyzed by the image analysis device 3, may be displayed on the display device 12a.

[0015] The terminal device 10b includes an information processing device 11b and a display device 12b, and is installed in a room other than the examination room. The terminal device 10b is used when a doctor creates a report on an endoscopic examination. The terminal devices 10a and 10b are configured with one or more processors having general-purpose hardware.

[0016] In the medical support system 1 of the embodiment, the endoscopic observation device 5 displays an endoscopic image on the display device 6 in real time, and supplies the endoscopic image together with meta information of the image to the image analysis device 3 in real time. Here, the meta information includes at least the frame number of the image and shooting time information, and the frame number is information indicating the number of the frame since the endoscope 7 started shooting. In other words, the frame number may be a serial number indicating the shooting order, and for example, the frame number of the first endoscopic image shot is set to "1", the frame number of the second endoscopic image shot is set to "2", etc.

[0017] The image analysis device 3 is an electronic calculator (computer) that analyzes an endoscopic image, detects a lesion contained in the endoscopic image, and performs a qualitative diagnosis of the detected lesion. The image analysis device 3 may be a CAD (computer-aided diagnosis) system having an AI (artificial intelligence) diagnostic function. The image analysis device 3 may be configured with one or more processors having dedicated hardware, or may be configured with one or more processors having general-purpose hardware.

[0018] The image analysis device 3 uses a trained model generated by machine learning using the endoscopic image for learning and information on the lesion area included in the endoscopic image as teacher data. The annotation work of the endoscopic image is performed by an annotator with specialized knowledge such as a doctor, and the machine learning may use CNN, RNN, LSTM, etc., which are types of deep learning. When an endoscopic image is input, this trained model outputs information indicating the captured organ, information indicating the captured site, and information on the captured lesion (lesion information). The lesion information output by the image analysis device 3 includes at least lesion presence / absence information indicating whether or not a lesion is included in the endoscopic image. If a lesion is included, the lesion information may include information indicating the size of the lesion, information indicating the position of the outline of the lesion, information indicating the shape of the lesion, information indicating the depth of the lesion, and a qualitative diagnosis result of the lesion. The qualitative diagnosis result of the lesion includes the type of the lesion. During the endoscopic examination, the image analysis device 3 receives the endoscopic image from the endoscopic observation device 5 in real time, and outputs information indicating the organ, information indicating the site, and lesion information for each endoscopic image. Hereinafter, the information indicating the organ, information indicating the site, and lesion information will be collectively referred to as "image meta information."

[0019] The image analysis device 3 of the embodiment has a function of timing the time that a doctor (hereinafter, sometimes referred to as "user") has observed a lesion. When a user operates a release switch to capture an endoscopic image, the image analysis device 3 measures the time from when the lesion included in the endoscopic image was first included in a previous endoscopic image until the release switch was operated as the time that the user observed the lesion. In other words, the image analysis device 3 specifies the time from when the lesion was first photographed until the user performed a capture operation (release switch operation) as the observation time of the lesion. In the embodiment, "photographing" refers to the operation of the solid-state image sensor of the endoscope 7 converting incident light into an electrical signal, and "capture" refers to the operation of saving (recording) the endoscopic image generated by the endoscopic observation device 5.

[0020] When the user performs a capture operation, the endoscopic observation device 5 provides the frame number, shooting time, and image ID of the captured endoscopic image together with information indicating that the capture operation has been performed (capture operation information) to the image analysis device 3. Upon acquiring the capture operation information, the image analysis device 3 specifies the observation time of the lesion, and provides the image ID, frame number, shooting time information, observation time of the lesion, and image meta information of the provided frame number to the server device 2. The server device 2 records the frame number, shooting time information, observation time of the lesion, and image meta information in association with the image ID of the endoscopic image.

[0021] The image analysis device 3 of the embodiment has a function of automatically capturing an endoscopic image when a lesion is detected in the endoscopic image. When one identical lesion is included in a 10-second endoscopic video, the image analysis device 3 may automatically capture an endoscopic image when the lesion is first detected, and may not automatically capture the image even if the same lesion is detected from a subsequent endoscopic image. The image analysis device 3 may finally capture one endoscopic image including the detected lesion, and may select one endoscopic image in which the lesion is most clearly captured after capturing multiple endoscopic images including the same lesion, and discard the other captured endoscopic images. Hereinafter, to clarify the subject of capture, an endoscopic image captured by a user through a capture operation may be called a "user-captured image," and an endoscopic image automatically captured by the image analysis device 3 may be called a "computer-captured image."

[0022] When the image analysis device 3 acquires a computer-captured image, it identifies the observation time of the lesion included in the computer-captured image afterwards. The image analysis device 3 may identify the time during which the lesion is photographed as the observation time of the lesion. For example, if the lesion is included in a video for 10 seconds (i.e., if the video is photographed for 10 seconds and displayed on the display device 12a), the image analysis device 3 may identify the observation time of the lesion as 10 seconds. Therefore, the image analysis device 3 measures the time from when the lesion comes into the frame and begins to be photographed to when the lesion goes out of the frame and ends to identify this as the observation time of the lesion. The image analysis device 3 provides the computer-captured image to the server device 2 together with the frame number of the computer-captured image, photographing time information, the observation time of the lesion, and image meta information.

[0023] When the user finishes the endoscopic examination, he or she operates an examination end button on the endoscopic observation device 5. Operation information of the examination end button is supplied to the server device 2 and the image analysis device 3, and the server device 2 and the image analysis device 3 recognize the end of the endoscopic examination.

[0024] 2 shows functional blocks of the server device 2. The server device 2 includes a communication unit 20, a processing unit 30, and a storage device 60. The communication unit 20 transmits and receives information such as data and instructions between the image analysis device 3, the endoscopic observation device 5, the image storage device 8, the terminal device 10a, and the terminal device 10b via the network 4. The processing unit 30 includes a first information acquisition unit 40, a second information acquisition unit 42, an image ID setting unit 44, and an image transmission unit 46. The storage device 60 includes an order information storage unit 62, a first information storage unit 64, and a second information storage unit 66. The order information storage unit 62 stores information on endoscopic examination orders.

[0025] The server device 2 includes a computer, which executes a program to realize various functions shown in Fig. 2. The computer includes hardware such as a memory into which the program is loaded, one or more processors that execute the loaded program, an auxiliary storage device, and other LSIs. The processor is composed of multiple electronic circuits including a semiconductor integrated circuit and an LSI, and the multiple electronic circuits may be mounted on one chip or multiple chips. The functional blocks shown in Fig. 2 are realized by cooperation between hardware and software, and therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms by hardware alone, software alone, or a combination thereof.

[0026] The first information acquisition unit 40 acquires the image ID, frame number, shooting time information, observation time, and image meta information of the user capture image from the image analysis device 3, and stores them in the first information storage unit 64 together with information indicating that the image is a user capture image. For example, if the user performs a capture operation seven times during an endoscopic examination, the first information acquisition unit 40 acquires the image ID, frame number, shooting time information, observation time, and image meta information of the seven user capture images from the image analysis device 3, and stores them in the first information storage unit 64 together with information indicating that the image is a user capture image. The first information storage unit 64 may store information indicating that the image is a user capture image, frame number, shooting time information, observation time, and image meta information, in association with the image ID. The image ID is assigned to the user capture image by the endoscopic observation device 5, and the endoscopic observation device 5 assigns the image IDs in order of the shooting time, starting from 1. Therefore, in this case, the seven user capture images are assigned image IDs 1 to 7, respectively.

[0027] The second information acquisition unit 42 acquires the computer-captured image, the frame number of the computer-captured image, the photographing time information, the observation time, and the image meta information from the image analysis device 3, and stores them in the second information storage unit 66 together with information indicating that the image is a computer-captured image. At this point, since the image ID is not assigned to the computer-captured image, the image ID setting unit 44 may set the image ID according to the photographing time to the computer-captured image. Specifically, the image ID setting unit 44 sets the image ID of the computer-captured image so as not to overlap with the image ID of the user-captured image. As described above, when image IDs 1 to 7 are assigned to seven user-captured images, the image ID setting unit 44 may assign image IDs to the computer-captured images in order of the photographing time, starting from 8. When the image analysis device 3 performs seven automatic captures (i.e., when a total of seven lesions are detected in the endoscopic examination and seven endoscopic images are automatically captured), the image ID setting unit 44 sets the image IDs in order of the earliest photographing time, starting from 8, and therefore, in this case, image IDs 8 to 14 are set to the seven computer-captured images. The second information storage section 66 stores, in association with the image ID, information indicating that the image is a computer capture image, the frame number, the photographing time information, the observation time, and the image meta information.

[0028] The image transmission unit 46 transmits the computer-captured images together with the assigned image IDs to the image storage device 8. As a result, the image storage device 8 stores both the user-captured images with image IDs 1 to 7 captured during the endoscopic examination and the computer-captured images with image IDs 8 to 14.

[0029] 3 shows an example of information associated with a capture image. Information relating to user-captured images with image IDs 1 to 7 is stored in the first information storage unit 64, and information relating to computer-captured images with image IDs 8 to 14 is stored in the second information storage unit 66.

[0030] The "image type" field stores information indicating whether the image is a user-captured image or a computer-captured image. The "organ" field stores information indicating the organ contained in the image, i.e., the organ photographed, and the "site" field stores information indicating the site of the organ contained in the image, i.e., the site of the organ photographed.

[0031] Among the lesion information, the "presence / absence" field stores information indicating whether or not a lesion has been detected by the image analysis device 3. Since all computer-captured images contain lesions, "presence" is stored in the "presence" field of the computer-captured images. The "size" field stores information indicating the longest diameter of the bottom surface of the lesion, the "shape" field stores coordinate information expressing the contour shape of the lesion, and the "diagnosis" field stores the qualitative diagnosis result of the lesion. The "observation time" field stores the observation time derived by the image analysis device 3. The "photographing time" field stores information indicating the time the image was photographed. The "photographing time" field may also include a frame number.

[0032] 4 shows the functional blocks of the information processing device 11b. The information processing device 11b has a function of selecting an endoscopic image to be displayed on a report input screen, and includes a communication unit 76, an input unit 78, a processing unit 80, and a storage device 120. The communication unit 76 transmits and receives information such as data and instructions between the server device 2, the image analysis device 3, the endoscopic observation device 5, the image storage device 8, and the terminal device 10a via the network 4. The processing unit 80 includes an operation reception unit 82, an acquisition unit 84, a display screen generation unit 100, an image specification unit 102, and a registration processing unit 110, and the acquisition unit 84 includes an image acquisition unit 86 and an information acquisition unit 88. The storage device 120 includes an image storage unit 122, an information storage unit 124, and a priority storage unit 126.

[0033] The information processing device 11b includes a computer, which executes a program to realize various functions shown in Fig. 4. The computer includes hardware such as a memory into which the program is loaded, one or more processors that execute the loaded program, an auxiliary storage device, and other LSIs. The processor is composed of multiple electronic circuits including a semiconductor integrated circuit and an LSI, and the multiple electronic circuits may be mounted on one chip or multiple chips. The functional blocks shown in Fig. 4 are realized by cooperation between hardware and software, and therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms by hardware alone, software alone, or a combination thereof.

[0034] After the endoscopic examination is completed, the doctor user logs in by inputting a user ID and password into the information processing device 11b. When the user logs in, an application for creating an examination report is started, and a list of performed examinations is displayed on the display device 12b. This list of performed examinations displays examination information such as the patient name, patient ID, examination date and time, and examination items in a list, and the user operates the input unit 78 such as a mouse or keyboard to select the examination for which a report is to be created. When the operation reception unit 82 receives an operation to select an examination, the image acquisition unit 86 acquires multiple endoscopic images linked to the examination ID of the selected examination from the image storage device 8, and stores them in the image storage unit 122.

[0035] In the embodiment, the image acquisition unit 86 acquires user-captured images with image IDs 1 to 7 captured by a user's capture operation and computer-captured images with image IDs 8 to 14 including a lesion captured by the image analysis device 3, and stores them in the image storage unit 122. The display screen generation unit 100 generates a selection screen for selecting an endoscopic image to be attached to a report, and displays it on the display device 12b.

[0036] As described above, the number of computer-captured images is equal to the number of lesions detected by the image analysis device 3. In the embodiment, seven endoscopic images are automatically captured by the image analysis device 3, but depending on the endoscopic examination, it is assumed that the image analysis device 3 detects tens to hundreds of lesions and automatically captures tens to hundreds of endoscopic images. In such a case, if all the automatically captured endoscopic images are displayed on the selection screen, it is not preferable because it requires a lot of effort for the user to select. Therefore, in the embodiment, the image specification unit 102 narrows down the computer-captured images to be displayed on the selection screen from among multiple computer-captured images. Hereinafter, the computer-captured images displayed on the selection screen will be referred to as "attachment candidate images."

[0037] The information acquiring unit 88 acquires information linked to the capture image from the storage device 60 of the server device 2 and stores it in the information storage unit 124. The image identifying unit 102 refers to the information linked to the capture image and identifies a computer capture image including a lesion not included in the user capture image as a "candidate image to be attached." By the image identifying unit 102 identifying a computer capture image including a lesion not included in the user capture image as a candidate image to be attached, computer capture images including the same lesion included in the user capture image are prevented from being displayed in duplicate on the selection screen.

[0038] 5 shows an example of a selection screen for selecting an endoscopic image to be attached to a report. The selection screen constitutes a part of the report input screen. The selection screen for endoscopic images is displayed on the display device 12b with the recorded image tab 54a selected. The upper part of the selection screen displays information on the patient's name, patient ID, date of birth, examination item, examination date, and performing doctor. This information is included in the examination order information and may be obtained from the server device 2.

[0039] The display screen generating unit 100 generates a selection screen including the user-captured image and the narrowed-down computer-captured images (candidate images to be attached), and displays the selection screen on the display device 12b. Since the lesions included in the computer-captured images (candidate images to be attached) do not overlap with the lesions included in the user-captured images, the user can efficiently select images to be attached to the report.

[0040] The display screen generating unit 100 generates a selection screen having a first area 50 for displaying user-captured images and a second area 52 for displaying computer-captured images (candidate images to be attached) separately provided, and displays the selection screen on the display device 12b. The display screen generating unit 100 displays the user-captured images with image IDs 1 to 7 in the first area 50 in the order in which they were captured, and displays the computer-captured images with image IDs 8, 10, 13, and 14 in the second area 52 in the order in which they were captured. By display device 12b displaying the user-captured images and the candidate images to be attached on the same screen, the user can efficiently select images to be attached to the report.

[0041] The process of narrowing down computer captured images will be described below. The image specification unit 102 specifies a computer capture image including a lesion not included in the user capture image as a "candidate image to be attached" based on the information on the capture image stored in the information storage unit 124 (see FIG. 3). The image specification unit 102 determines whether or not there is a user capture image including the same organ as the organ included in the computer capture image based on the information indicating the organ included in the user capture images of image IDs 1 to 7 and the information indicating the organ included in the computer capture image, and if there is no user capture image including the same organ as the organ included in the computer capture image, the image specification unit 102 may specify the computer capture image as a candidate image to be attached. If there is no user capture image including the same organ as the organ included in the computer capture image, it is certain that the lesion included in the computer capture image is not included in the user capture image, so the image specification unit 102 may specify the computer capture image as a candidate image to be attached.

[0042] The image specification unit 102 determines whether or not there is a user capture image including the same part as the part included in the computer capture image based on the information indicating the part of the organ included in the user capture images of image IDs 1 to 7 and the information indicating the part of the organ included in the computer capture image, and if there is no user capture image including the same part as the part included in the computer capture image, the image specification unit 102 may specify the computer capture image as a candidate image to be attached. If there is no user capture image including the same part as the part included in the computer capture image, it is certain that the lesion included in the computer capture image is not included in the user capture image, so the image specification unit 102 may specify the computer capture image as a candidate image to be attached.

[0043] The image specification unit 102 may determine whether or not there is a user capture image including a lesion of substantially the same size as the size of the lesion included in the computer capture image based on the information indicating the size of the lesion included in the user capture images of image IDs 1 to 7 and the information indicating the size of the lesion included in the computer capture image, and may specify the computer capture image as a candidate image for attachment if there is no user capture image including a lesion of substantially the same size as the size of the lesion included in the computer capture image. If there is no user capture image including a lesion of the same size as the size of the lesion included in the computer capture image, it is certain that the lesion included in the computer capture image is not included in the user capture image, so the image specification unit 102 may specify the computer capture image as a candidate image for attachment.

[0044] The image specification unit 102 determines whether or not there is a user capture image including a lesion having substantially the same shape as the shape of the lesion included in the computer capture image based on information indicating the shape of the lesion included in the user capture images of image IDs 1 to 7 and information indicating the shape of the lesion included in the computer capture image, and if there is no user capture image including a lesion having substantially the same shape as the shape of the lesion included in the computer capture image, the image specification unit 102 may specify the computer capture image as a candidate image to be attached. If there is no user capture image including a lesion having the same shape as the shape of the lesion included in the computer capture image, it is certain that the lesion included in the computer capture image is not included in the user capture image, so the image specification unit 102 may specify the computer capture image as a candidate image to be attached.

[0045] The image specification unit 102 may determine whether or not there is a user capture image including a lesion of substantially the same type as the type of lesion included in the computer capture image based on the information indicating the type of lesion included in the user capture images of image IDs 1 to 7 and the information indicating the type of lesion included in the computer capture image, and may specify the computer capture image as a candidate image for attachment if there is no user capture image including a lesion of substantially the same type as the type of lesion included in the computer capture image. If there is no user capture image including a lesion of the same type as the type of lesion included in the computer capture image, it is certain that the lesion included in the computer capture image is not included in the user capture image, so the image specification unit 102 may specify the computer capture image as a candidate image for attachment.

[0046] The image specification unit 102 determines whether or not there is a user capture image including an organ, a site, a lesion, a size, a shape, and a type of the organ included in the user capture image of image ID 1 to 7, based on the information indicating the organ, the site, the size, the shape, and a type of the lesion included in the computer capture image, and if there is no user capture image including an organ, a site, and a lesion, which are substantially the same as the organ, the site, and the lesion included in the computer capture image, the image specification unit 102 may specify the computer capture image as a candidate image to be attached. If there is no user capture image including an organ, a site, and a lesion, which are substantially the same as the organ, the site, and the lesion included in the computer capture image, it is certain that the lesion included in the computer capture image is not included in the user capture image, so the image specification unit 102 may specify the computer capture image as a candidate image to be attached.

[0047] As described above, the image specification unit 102 specifies computer-captured images including lesions not included in the user-captured images of image IDs 1 to 7 as candidate images to be attached. With reference to FIG. 3, the image specification unit 102 determines that the lesion included in the computer-captured image of image ID 9 is the same as the lesion included in the user-captured image of image ID 3, the lesion included in the computer-captured image of image ID 11 is the same as the lesion included in the user-captured image of image ID 5, and the lesion included in the computer-captured image of image ID 12 is the same as the lesion included in the user-captured image of image ID 6. Therefore, the image specification unit 102 determines not to include the computer-captured images of image IDs 9, 11, and 12 in the selection screen, and determines the computer-captured images of image IDs 8, 10, 13, and 14 as candidate images to be attached. After the above narrowing down is performed, the display screen generation unit 100 displays the user-captured images of image IDs 1 to 7 in the first area 50, and displays the computer-captured images of image IDs 8, 10, 13, and 14 in the second area 52.

[0048] A check box is provided on the endoscopic image displayed on the selection screen, and when the user operates the mouse to place the mouse pointer on the check box and right-click, the endoscopic image is selected as an image to be attached to the report. On the selection screen, the operation reception unit 82 receives an operation by the user to select a user-captured image or a candidate image to be attached. When the user places the mouse pointer on an endoscopic image and right-clicks, the endoscopic image can be enlarged and displayed, and the user may determine whether to attach the image to the report by looking at the enlarged endoscopic image.

[0049] 5, check marks indicating that the images have been selected are displayed in the check boxes for the user-captured image with image ID 3 and the computer-captured images with image IDs 13 and 14. When the user operates the temporary save button using the input unit 78, the registration processing unit 110 provisionally registers the selected endoscopic images with image IDs 3, 13, and 14 as report attachment images in the image storage unit 122. After selecting the attachment images, the user selects the report tab 54b to display a report input screen on the display device 12b.

[0050] FIG. 6 shows an example of a report creation screen for inputting the examination results. The report creation screen constitutes a part of the report input screen. When the report tab 54b is selected, the display screen generating unit 100 generates a report creation screen and displays it on the display device 12b. The report creation screen is composed of two areas, an attached image display area 56 for displaying attached images is arranged on the left side, and an input area 58 for the user to input the examination results is arranged on the right side. In this example, the endoscopic images with image IDs 3, 13, and 14 are selected as attached images and are displayed in the attached image display area 56.

[0051] An upper limit may be set for the number of computer-captured images displayed in the second area 52 shown in Fig. 5. Since the computer-captured images displayed in the second area 52 are automatically captured by the image analysis device 3 and are not captured by the user, if the number of images displayed is too large, the user has to make more effort to select images. Therefore, when the number of computer-captured images including a lesion not included in the user-captured images exceeds a predetermined first upper limit, the image specification unit 102 may specify computer-captured images equal to or less than the first upper limit as candidate images to be attached based on the priority order set according to the type of lesion.

[0052] 7 shows an example of a table stored in the priority storage unit 126. The priority storage unit 126 stores a table that defines the correspondence between the qualitative diagnosis result indicating the type of lesion and the priority order. In this example, with regard to the qualitative diagnosis of the large intestine (type of lesion), the priority order is defined as follows: colon cancer has the first priority, malignant polyps has the second priority, malignant melanoma has the third priority, and non-neoplastic polyps has the fourth priority. Below, a case will be described where the upper limit of the number of images to be displayed in the second area 52 is set to three (first upper limit number=3).

[0053] In the example, computer-captured images with image IDs 8, 10, 13, and 14 are identified as images containing lesions not included in the user-captured images. The qualitative diagnosis results for each computer-captured image are shown below. Image ID 8: Malignant melanoma Image ID 10: Non-neoplastic polyp Image ID 13: Malignant polyp Image ID 14: Malignant melanoma

[0054] The image specification unit 102 specifies three or less computer-captured images as candidate images to be attached based on the priority order set according to the type of lesion. In the embodiment, the number of computer-captured images including lesions not included in the user-captured image is four, which exceeds the first upper limit number (three), so the image specification unit 102 needs to narrow down the number of computer-captured images to three or less based on the priority order. The priority order corresponding to the qualitative diagnosis result of each computer-captured image is shown below.

[0055] Image ID 8: Malignant melanoma 3rd place Image ID 10: Non-neoplastic polyp 4th place Image ID 13: Malignant polyp 2nd place Image ID 14: Malignant melanoma 3rd place

[0056] The image specification unit 102 may exclude the computer-captured image with image ID 10 and specify the computer-captured images with image IDs 8, 13, and 14 as candidate images to be attached based on the priority order corresponding to the qualitative diagnosis result of each image. In this manner, an upper limit on the number of images to be displayed in the second area 52 is set, and the candidate images to be attached are selected based on the priority order set according to the qualitative diagnosis, so that the display screen generation unit 100 can preferentially display computer-captured images including important lesions in the second area 52.

[0057] As described above, even if the attachment candidate images are specified based on the priority order, the number of attachment candidate images may exceed the first upper limit. For example, if the upper limit of the number of images to be displayed in the second area 52 is set to two (first upper limit=2), three attachment candidate images are specified in the above example, which still exceeds the first upper limit. Therefore, the image specification unit 102 may select computer-captured images, further taking into account the observation time, so that the number of attachment candidate images is equal to or less than the first upper limit.

[0058] First, when the number of computer-captured images including a lesion not included in the user-captured image exceeds a first upper limit number (two), the image specification unit 102 specifies computer-captured images to be candidates for attachment candidate images based on the priority order. Here, the computer-captured images with image IDs 8, 13, and 14 are specified as candidates for attachment candidate images. When the number of computer-captured images specified as candidates for attachment candidate images (three) exceeds the first upper limit number, the image specification unit 102 may remove computer-captured images that have been observed by the user for a short time from the candidates for attachment candidate images, and specify computer-captured images that are equal to or less than the first upper limit number as attachment candidate images.

[0059] Since the priority of the image with image ID 13 is second and the priority of the images with image IDs 8 and 14 is third, the image specification unit 102 determines that the image with image ID 13 is a candidate image for attachment, and then compares the observation times of the images with image IDs 8 and 14. Here, the observation time of the image with image ID 8 is 16 seconds, and the observation time of the image with image ID 14 is 12 seconds. Since it is expected that the image with the longer observation time attracted the user's attention during the inspection, the image specification unit 102 removes the computer-captured image with image ID 14, which has a shorter observation time, from the candidates for the candidate image for attachment, and specifies the computer-captured image with image ID 8 as the candidate image for attachment. Therefore, the image specification unit 102 may specify the computer-captured images with image IDs 8 and 13 as the candidate images for attachment.

[0060] Fig. 8 shows another example of a selection screen for selecting an endoscopic image to be attached to a report. Compared to the selection screen shown in Fig. 5, the number of computer-captured images displayed in the second area 52 is limited to two. By limiting the number of images displayed in the second area 52, the user can efficiently select images to be attached to the report. Note that a switching button 70 for displaying all computer-captured images may be provided in the second area 52.

[0061] 9 shows an example of a selection screen when all computer-captured images are displayed. A switching button 70 is used to switch between a mode in which a limited number of computer-captured images are displayed and a mode in which all computer-captured images are displayed. By displaying all computer-captured images in the second area 52, the user can view an endoscopic image that includes all detected lesions.

[0062] The display screen generating unit 100 may display the user captured image and the candidate images to be attached on the same screen in the first display mode, and may display only the candidate images to be attached in the second display mode without displaying the user captured image. This mode switching may be performed by an operation button other than the switching button 70. The display screen generating unit 100 generates the selection screen in various modes, so that the user can select an image to be attached to the report from the selection screen including the capture image that the user desires.

[0063] 10 shows another example of information associated with a capture image. In this example, information about the user-captured image with image ID 1 is stored in the first information storage unit 64, and information about the computer-captured images with image IDs 2 to 7 is stored in the second information storage unit 66.

[0064] 10, the user-captured image with image ID 1 and the computer-captured images with image IDs 2 to 5 include non-neoplastic polyps in the ascending colon of the large intestine. When the total number of one or more user-captured images and one or more computer-captured images including a predetermined type of lesion (in this embodiment, non-neoplastic polyps) in the same area exceeds a predetermined second upper limit number, the image specifying unit 102 does not specify the one or more computer-captured images as attachment candidate images.

[0065] When the number of captured images including non-neoplastic polyps in the same location exceeds a second upper limit (for example, 4), it is assumed that the user intentionally did not capture non-neoplastic polyps. In such a case, since it is not preferable to display images not captured by the user as candidate images to be attached, the image specification unit 102 does not select computer-captured images as candidate images to be attached when the total number of user-captured images and computer-captured images including a predetermined type of lesion (non-neoplastic polyp) in the same location exceeds a predetermined second upper limit. In the example of FIG. 10, it is preferable that the image specification unit 102 specifies computer images with image IDs 6 and 7 as candidate images to be attached, and does not specify computer-captured images with image IDs 2 to 5 as candidate images to be attached.

[0066] The present disclosure has been described above based on a number of embodiments. These embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each processing process, and that such modifications are also within the scope of the present disclosure. In the embodiments, the endoscopic observation device 5 transmits a user-captured image to the image storage device 8, but in a modified example, the image analysis device 3 may transmit a user-captured image to the image storage device 8. In addition, in the embodiments, the information processing device 11b has the image specification unit 102, but in a modified example, the server device 2 may have the image specification unit 102. [Industrial Applicability]

[0067] The present disclosure can be used in the technical field of supporting report creation. [Explanation of symbols]

[0068] REFERENCE SIGNS LIST 1 medical support system, 2 server device, 3 image analysis device, 4 network, 5 endoscopic observation device, 6 display device, 7 endoscope, 8 image storage device, 9 endoscopic system, 10a, 10b terminal device, 11a, 11b information processing device, 12a, 12b display device, 20 communication unit, 30 processing unit, 40 first information acquisition unit, 42 second information acquisition unit, 44 image ID setting unit, 46 image transmission unit, 50 first area, 52 second area, 54a recorded image tab, 54b ··Report tab, 56···Attached image display area, 58···Input area, 60···Storage device, 62···Order information storage section, 64···First information storage section, 66···Second information storage section, 70···Switching button, 76···Communication section, 78···Input section, 80···Processing section, 82···Operation reception section, 84···Acquisition section, 86···Image acquisition section, 88···Information acquisition section, 100···Display screen generation section, 102···Image identification section, 110···Registration processing section, 120···Storage device, 122···Image storage section, 124···Information storage section, 126···Priority storage section.

Claims

1. 1. A medical support system, comprising: one or more processors having hardware; The one or more processors: Obtaining a first image captured by a user's capture operation and a computer-captured image including the lesion and captured by a computer; identifying the computer-captured image containing a lesion not included in the first image as a second image; generating a selection screen for selecting an image to be attached to a report, the selection screen including the first image and the second image; A medical support system comprising:

2. The one or more processors: accepting an operation by a user to select the first image or the second image on the selection screen; 2. The medical support system according to claim 1.

3. The one or more processors: determining whether or not there is a first image including an organ that is the same as an organ included in the computer-captured image, based on information indicating an organ included in the first image and information indicating an organ included in the computer-captured image; identifying the computer-captured image as the second image if there is no first image that includes the same organ as the organ included in the computer-captured image; 2. The medical support system according to claim 1.

4. The one or more processors: determining whether or not there is a first image including a part that is the same as a part included in the computer-captured image, based on information indicating a part of an organ included in the first image and information indicating a part of an organ included in the computer-captured image; identifying the computer-captured image as the second image if there is no first image that includes the same part as the part included in the computer-captured image; 2. The medical support system according to claim 1.

5. The one or more processors: determining whether or not there is a first image including a lesion having the same size as the lesion included in the computer-captured image based on information indicating a size of the lesion included in the first image and information indicating a size of the lesion included in the computer-captured image; identifying the computer-captured image as the second image if no first image contains a lesion of the same size as the lesion contained in the computer-captured image; 2. The medical support system according to claim 1.

6. The one or more processors: determining whether or not there is a first image including a lesion having the same shape as the shape of the lesion included in the computer-captured image based on information indicating the shape of the lesion included in the first image and information indicating the shape of the lesion included in the computer-captured image; identifying the computer-captured image as the second image if there is no first image that includes a lesion having the same shape as the lesion included in the computer-captured image; 2. The medical support system according to claim 1.

7. The one or more processors: determining whether or not there is a first image including a lesion of the same type as the lesion included in the computer-captured image based on information indicating the type of lesion included in the first image and information indicating the type of lesion included in the computer-captured image; identifying the computer-captured image as the second image if there is no first image containing a lesion of the same type as the lesion contained in the computer-captured image; 2. The medical support system according to claim 1.

8. The one or more processors: determining whether or not there is a first image including an organ, a site, and a lesion that are the same as those included in the computer-captured image, based on information included in the first image that indicates an organ, a site, a size of a lesion, a shape of a lesion, and a type of a lesion, and information included in the computer-captured image that indicates an organ, a site, a lesion that are the same as those included in the computer-captured image; identifying the computer-captured image as the second image if there is no first image that includes the same organ, site, or lesion as the organ, site, or lesion included in the computer-captured image; 2. The medical support system according to claim 1.

9. The one or more processors: when the number of the computer-captured images including a lesion not included in the first image exceeds a predetermined first upper limit number, identifying the computer-captured images not exceeding the first upper limit number as the second images based on a priority order set according to the type of lesion; 2. The medical support system according to claim 1.

10. The one or more processors: obtaining information indicative of a time at which a user observed a lesion included in the computer-captured image; when the number of the candidate second images exceeds the first upper limit number even if the computer-captured images to be the candidates for the second image are identified based on the priority order, the computer-captured images that have been observed for a short time by the user are removed from the candidates for the second image, and the computer-captured images that are equal to or less than the first upper limit number are identified as the second images; The medical support system according to claim 9 .

11. The one or more processors: When a total number of the one or more first images including a predetermined type of lesion in the same site and the one or more computer-captured images exceeds a predetermined second upper limit number, the computer-captured images are not identified as the second images.

2. The medical support system according to claim 1.

12. The one or more processors: generating the selection screen having a first area for displaying the first image and a second area for displaying the second image, the selection screen being provided separately; 2. The medical support system according to claim 1.

13. The one or more processors: Displaying the first image and the second image on the same screen. The medical support system according to claim 12 .

14. The one or more processors: In a first display mode, the first image and the second image are displayed on the same screen; In a second display mode, the second image is displayed and the first image is not displayed.

2. The medical support system according to claim 1.

15. acquiring a first image captured by a user's capture operation; obtaining a computer-captured image including the lesion; identifying the computer-captured image containing a lesion not included in the first image as a second image; generating a selection screen for selecting an image to be attached to a report, the selection screen including the first image and the second image; A report creation support method comprising:

16. In the selection screen, an operation by a user to select the first image or the second image is accepted.

16. The report creation support method according to claim 15.

17. On the computer, A function of acquiring a first image captured by a user's capture operation; acquiring a computer-captured image including the lesion and captured by a separate computer; identifying the computer-captured image as a second image that includes a lesion not included in the first image; generating a selection screen for selecting an image to be attached to a report, the selection screen including the first image and the second image; A program to achieve this.

18. A computer comprising: realizing a function of accepting an operation by a user to select the first image or the second image on the selection screen; 18. The program according to claim 17 .

19. An information processing device, comprising one or more processors having hardware, The one or more processors: Obtaining a first image captured by a user's capture operation and a computer-captured image including the lesion and captured by a computer; identifying the computer-captured image containing a lesion not included in the first image as a second image; generating a selection screen for selecting an image to be attached to a report, the selection screen including the first image and the second image; 23. An information processing apparatus comprising:

20. The one or more processors: accepting an operation by a user to select the first image or the second image on the selection screen; 20. The information processing apparatus according to claim 19,

Images