Medical assistance device, medical assistance system, and medical assistance method

Abstract

Provided are a medical assistance device, a medical assistance system, and a medical assistance method that make it possible to input a report by a simple operation while shortening a search time with a simple configuration. The medical assistance device: selects observation information associated with an image ID, from an observation information database in which items of a doctor's observation information are associated with image IDs, in order of closeness of feature amounts in annotation information; and displays the observation information associated with the image ID on a display unit as recommended observation information for a lesion, in order of closeness of the feature amounts in the annotation information.

Description

Medical support device, medical support system, and medical support method

[0001] The present disclosure relates to a medical support device, a medical support system, and a medical support method that provide medical support when interpreting and creating reports on endoscopic images captured by an endoscope.

[0002] Conventionally, in systems that manage content such as documents, images, music, and videos, a technology has been known in which annotations added to content are used to calculate a value indicating the strength of relevance to the content, and then used to rank content during content searches and recommend content (see, for example, Patent Document 1).

[0003] Furthermore, in the medical field, a technology is known that aims to improve usability when annotating images of biological subjects (see, for example, Patent Document 2). This technology identifies a similar region that is similar to the first region in a second image that captures a region different from the first region in the first image or a region of the subject that includes at least a portion of the region captured in the first image, based on information about the first region in a first annotation made by a user on the first image, and displays a second annotation in a second region in the first image that corresponds to the similar region.

[0004] JP 2011-227633 A International Publication No. 2021 / 125305

[0005] In the medical field using endoscopes, a large number of images are taken when examining a subject. Therefore, in the medical field, when searching for similar images similar to an annotated image from a group of images as in Patent Documents 1 and 2, large resources are required for the CPU and memory, and therefore a technology that can shorten the search time with a simple configuration has been desired.

[0006] In addition, medical professionals such as doctors who perform endoscopy or interpret images want to input test reports for subjects while referring to previous findings made by other doctors, but they have to input the information while looking at the reference reports, which makes the process complicated.

[0007] The present disclosure has been made in consideration of the above, and aims to provide a medical support device, a medical support system, and a medical support method that have a simple configuration, shorten search time, and enable report input with simple operations.

[0008] In order to solve the above-mentioned problems and achieve the object, a medical support device according to the present disclosure is a medical support device including a processor, wherein the processor, in response to an external operation, assigns annotations according to lesions to endoscopic images of a patient under examination as annotation information, calculates a ratio of the annotation information to the endoscopic image and each of feature quantities in the annotation information, and selects a group of similar images having similar annotation information and similar ratios as a first priority from an annotation database that records annotation IDs that identify annotation information when annotations are assigned to multiple past images from previous examinations, in association with the type of annotation information and the ratio. The image information database extracts the plurality of past images as a group of prior images, records the annotation ID and the feature of each of the plurality of past images in association with the image ID that identifies each of the plurality of past images, searches for an image that matches the annotation ID in the first priority image group, and sequentially selects the image IDs in order of proximity to the feature in the annotation information; the image information database records the doctor's finding information in association with the image IDs, selects the finding information associated with the image IDs in order of proximity to the feature in the annotation information, and outputs the finding information associated with the image IDs to a display unit in order of proximity to the feature in the annotation information as recommended finding information for the lesion.

[0009] In addition, in the medical support device according to the present disclosure, the annotation information has a shape that specifies a lesion area in the endoscopic image.

[0010] In addition, in the medical support device according to the present disclosure, the annotation information is a shape that specifies a boundary of a lesion in the endoscopic image.

[0011] In addition, in the medical support device according to the present disclosure, in the above disclosure, the annotation information has a shape that specifies a point of interest of a lesion in the endoscopic image or a vicinity of the point of interest.

[0012] In addition, in the medical support device according to the present disclosure, in the above disclosure, the processor extracts the group of similar images as a first priority group of images from the annotation database based on the type of annotation information and the ratio.

[0013] In the medical support device according to the present disclosure, the recommended finding information includes the finding information and doctor information.

[0014] In addition, in the medical support device according to the present disclosure, in the above disclosure, the processor displays the endoscopic image and the recommended finding information on the display unit, and the recommended finding information includes the finding information and disease name for each doctor.

[0015] Further, a medical support system according to the present disclosure includes an endoscope capable of capturing images of the inside of a subject's body, and a medical support device capable of communicating with the endoscope, the medical support device including a processor, which, in response to an external operation, adds annotations according to a lesion to an endoscopic image of a patient to be examined as annotation information, calculates a ratio of the annotation information to the endoscopic image and each of feature amounts in the annotation information, and retrieves an annotation database that records an annotation ID that identifies annotation information when annotating each of a plurality of past images obtained by a previous examination, in association with a type of annotation information and the ratio, and The method extracts a group of similar images that match the annotation ID in the first priority image group as a first priority image group, searches an image information database in which the annotation ID and the feature amount of each of the plurality of past images are associated with an image ID that identifies each of the plurality of past images, and sequentially selects the image IDs in order of proximity to the feature amount in the annotation information, selects the finding information associated with the image IDs in order of proximity to the feature amount in the annotation information from a finding information database in which the doctor's finding information is associated with the image IDs, and outputs the finding information associated with the image IDs to a display unit as recommended finding information for the lesion in order of proximity to the feature amount in the annotation information.

[0016] In addition, in the medical support system according to the present disclosure, the annotation information has a shape that specifies a lesion area in the endoscopic image.

[0017] In addition, in the medical support system according to the present disclosure, the annotation information is a shape that specifies a boundary of a lesion in the endoscopic image.

[0018] In addition, in the medical support system according to the present disclosure, the annotation information has a shape that specifies a point of interest of a lesion in the endoscopic image or a vicinity of the point of interest.

[0019] In addition, in the medical support system according to the present disclosure, in the above disclosure, the processor extracts the group of similar images as a first priority group of images from the annotation database based on the type of annotation information and the ratio.

[0020] In addition, in the medical support system according to the present disclosure, the recommended finding information includes the finding information and doctor information.

[0021] In addition, in the medical support system according to the present disclosure, the processor displays the endoscopic image and the recommended finding information on the display unit, and the recommended finding information includes the finding information and disease name for each doctor.

[0022] Furthermore, a medical support method according to the present disclosure is a medical support method executed by a medical support device including a processor, wherein the processor, in response to an external operation, adds annotations according to a lesion to an endoscopic image of a patient under examination as annotation information, calculates a ratio of the annotation information to the endoscopic image and each of feature quantities in the annotation information, and selects a group of similar images having similar annotation information and similar ratios from an annotation database that records annotation IDs that identify annotation information when annotations are added to each of a plurality of past images from previous examinations in association with the type of annotation information and the ratio, and selects a group of similar images having similar annotation information and similar ratios as a first priority image group. and searches for an image ID in the first priority image group that matches the annotation ID from an image information database that records an image ID identifying each of the plurality of past images in association with the annotation ID and a feature amount of each of the plurality of past images, and sequentially selects the image IDs in order of proximity to the feature amount in the annotation information; and selects the finding information associated with the image ID from a finding information database that records a doctor's finding information in association with the image IDs in order of proximity to the feature amount in the annotation information, and outputs the finding information associated with the image IDs to a display unit as recommended finding information for the lesion in order of proximity to the feature amount in the annotation information.

[0023] In addition, in the medical support method according to the present disclosure, the annotation information has a shape that specifies a lesion area in the endoscopic image.

[0024] Further, in the medical support method according to the present disclosure, in the above disclosure, the annotation information is a shape that specifies a boundary of a lesion in the endoscopic image.

[0025] Further, in the medical support method according to the present disclosure, in the above disclosure, the annotation information has a shape that specifies a point of interest of a lesion in the endoscopic image or a vicinity of the point of interest.

[0026] In addition, in the medical support method according to the present disclosure, in the above disclosure, the processor extracts the group of similar images as a first priority group of images from the annotation database based on the type of annotation information and the ratio.

[0027] In addition, in the medical support method according to the present disclosure, the recommended finding information includes the finding information and doctor information.

[0028] Further, in the medical support method according to the present disclosure, in the above disclosure, the processor displays the endoscopic image and the recommended finding information on the display unit, and the recommended finding information includes the finding information and disease name for each doctor.

[0029] According to the present disclosure, it is possible to achieve the effect of shortening search time with a simple configuration and allowing reports to be input with simple operations.

[0030] FIG. 1 is a diagram illustrating an overall configuration of a medical support system according to an embodiment. FIG. 2 is a block diagram illustrating a functional configuration of a medical support device according to an embodiment. FIG. 3 is a flowchart illustrating an outline of processing executed by the medical support device according to an embodiment. FIG. 4 is a diagram illustrating an example of an imaging / report screen displayed on a display unit included in the medical support device according to an embodiment. FIG. 5 is a diagram illustrating an example of an image displayed on a display unit included in the medical support device according to an embodiment. FIG. 6 is a diagram illustrating an example of annotation information. FIG. 7 is a diagram illustrating another example of annotation information. FIG. 8 is a diagram illustrating another example of annotation information. FIG. 9 is a diagram illustrating another example of annotation information. FIG. 10 is a diagram illustrating another example of annotation information. FIG. 11 is a diagram illustrating another example of annotation information. FIG. 12 is a diagram illustrating an example of processing performed by each of a type discrimination unit, a ratio calculation unit, and a feature amount calculation unit on annotation information. FIG. 13 is a diagram illustrating another example of processing performed by each of a type discrimination unit, a ratio calculation unit, and a feature amount calculation unit on annotation information. FIG. 14 is a diagram illustrating another example of processing performed by each of the type discrimination unit, the ratio calculation unit, and the feature amount calculation unit on annotation information. FIG. 15 is a diagram illustrating another example of processing performed by each of the type discrimination unit, the ratio calculation unit, and the feature amount calculation unit on annotation information. FIG. 16 is a diagram illustrating another example of processing performed by each of the type discrimination unit, the ratio calculation unit, and the feature amount calculation unit on annotation information. FIG. 17 is a diagram illustrating another example of processing performed by each of the type discrimination unit, the ratio calculation unit, and the feature amount calculation unit on annotation information. FIG. 18 is a diagram schematically illustrating extraction content extracted from the annotation information DB by an extraction unit included in a medical support device according to an embodiment. FIG. 19 is a diagram schematically illustrating selection content searched for and selected by a search unit included in a medical support device according to an embodiment. FIG. 20 is a diagram schematically illustrating selection content searched for and selected by a selection unit included in a medical support device according to an embodiment. FIG. 21 is a diagram illustrating an example of an image displayed by a display unit included in a medical support device according to an embodiment. FIG. 22 is a diagram illustrating an example of an image displayed on a display unit included in the medical support device according to one embodiment.

[0031] Hereinafter, embodiments for carrying out the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the following embodiments. Furthermore, each drawing referred to in the following description merely shows a schematic representation of the shape, size, and positional relationship to the extent that the contents of the present disclosure can be understood. In other words, the present disclosure is not limited to only the shape, size, and positional relationship exemplified in each drawing. Furthermore, in the description of the drawings, the same parts are denoted by the same reference numerals.

[0032] [Overall Configuration of Medical Support System] Fig. 1 is a diagram showing the overall configuration of a medical support system according to one embodiment. The medical support system 1 shown in Fig. 1 includes an endoscope 2 and a medical support device 10 connected via a network N100.

[0033] The endoscope 2 is inserted into the esophagus through the mouth of a human or animal subject, or into the large intestine through the anus of the subject, to continuously capture images of the inside of the subject, and displays the captured image data in chronological order, captures images of the inside of the subject to generate endoscopic images, and performs treatments or biopsies inside the subject using treatment tools.

[0034] The medical support device 10 acquires and manages endoscopic images corresponding to the image data captured by the endoscope 2, creates reports of the interpretation results for the subject, and accepts and creates inputs such as the subject's diagnosis and conference based on the diagnostic results of biological cells obtained by biopsy during observation of the subject.

[0035] [Functional Configuration of Medical Support Device] Next, the functional configuration of the medical support device 10 will be described. Fig. 2 is a block diagram showing the functional configuration of the medical support device 10. The medical support device 10 includes a communication unit 11, a display unit 12, an input unit 13, a recording unit 14, an image information database 15 (hereinafter simply referred to as "image information DB 15"), an annotation information database 16 (hereinafter simply referred to as "annotation information DB 16"), a finding information database 17 (hereinafter simply referred to as "finding information DB 17"), and a control unit 18.

[0036] The communication unit 11, under the control of the control unit 18, receives endoscopic image data generated by the endoscope 2 via the network N100 and outputs the data to the control unit 18. The communication unit 11 is configured using a communication module or the like that is capable of Wi-Fi (Wireless Fidelity) (registered trademark) or the like.

[0037] The display unit 12 displays various information related to the medical support device 10 and endoscopic images under the control of the control unit 18. The display unit 12 is configured using a liquid crystal display, an organic electroluminescent display (EL display), or the like.

[0038] The input unit 13 is configured using a mouse, keyboard, switches, a touch panel, etc., and receives various inputs and outputs them to the control unit 18.

[0039] The recording unit 14 is configured using a non-volatile memory, a volatile memory, a hard disk drive (HDD), a solid state drive (SSD), etc., and records various information related to the medical support device 10. The recording unit 14 also has a program recording unit 141 that records various programs executed by the medical support device 10.

[0040] The image information DB 15 is constructed using an HDD or SSD, etc., and records an image ID that identifies each of the multiple endoscopic images (past images) captured by the endoscope 2, in association with an annotation ID, features, and endoscopic images (image data).

[0041] The annotation information DB 16 is configured using a HDD, an SSD, or the like. The annotation information DB 16 stores annotation IDs that identify annotation information when annotations are added to each of multiple past images from previous examinations, in association with the type of annotation information and the proportion of the annotation information for each of the multiple past images. Here, the past images are endoscopic images generated by the endoscope 2 when a user or another doctor or the like previously performed an examination on a subject using the endoscope 2. The proportion of the annotation information for the past images is the proportion of the area of ​​the annotation information to the display area of ​​the past images. For example, the proportion of the annotation information for the past images is the value obtained by dividing the size of the display area of ​​the past image (display pixels) by the size of the area to which the annotation information is added (area pixels).

[0042] The finding information DB 17 is configured using a HDD, an SSD, etc. The finding information DB 17 records the doctor's finding information in association with an image ID that identifies each of a plurality of endoscopic images captured by the endoscope 2.

[0043] The control unit 18 is realized using a processor having hardware such as a CPU (Central Processing Unit), ASIC (Application Specific Integrated Circuit), or FPGA (Field-Programmable Gate Array), and a memory serving as a temporary storage area used by the processor. The control unit 18 comprehensively controls each unit constituting the medical support device 10. The control unit 18 includes an acquisition unit 181, a display control unit 182, an annotation assignment unit 183, a type determination unit 184, a ratio calculation unit 185, a feature calculation unit 186, an extraction unit 187, a search unit 188, a selection unit 189, and a recording control unit 190.

[0044] The acquisition unit 181 acquires endoscopic images (image data) from the endoscope 2 via the communication unit 11 and records the acquired endoscopic images in the image information DB 15 .

[0045] The display control unit 182 displays on the display unit 12 an imaging / report screen on which the surgeon interprets the subject's image using the endoscopic image captured by the endoscope 2 and reports and inputs findings. The display control unit 182 also displays on the display unit 12 information on recommended findings for the lesion.

[0046] The annotation providing unit 183 provides annotation information to the endoscopic image based on an annotation operation performed by the user on the endoscopic image displayed on the display unit 12 by operating the input unit 13 .

[0047] The type determination unit 184 determines the type of annotation information that has been added to the endoscopic image by the annotation addition unit 183. Specifically, the type determination unit 184 determines whether the type of annotation information is any of the following types: a shape that specifies the range of a lesion in the endoscopic image, a shape that specifies the boundary of a lesion in the endoscopic image, and a shape that specifies a point of interest of a lesion or the vicinity of the point of interest in the endoscopic image.

[0048] The ratio calculation unit 185 determines the type of annotation information that the annotation assignment unit 183 has assigned to the endoscopic image P1.

[0049] The feature amount calculation unit 186 calculates the feature amount in the annotation information that the annotation assignment unit 183 has assigned to the endoscopic image P1.

[0050] Based on the type of annotation information determined by the type determination unit 184 and the ratio calculated by the ratio calculation unit 185, the extraction unit 187 extracts, from the multiple annotation information recorded in the annotation information DB 16, annotation IDs that approximate the type of annotation information determined by the type determination unit 184 and the ratio calculated by the ratio calculation unit 185 as a first priority image group.

[0051] The search unit 188 searches the image information DB 15 for features of the image IDs associated with each of the multiple annotation IDs extracted by the extraction unit 187, and selects multiple candidate endoscopic images as a second priority image group in order of features closest to the features included in the annotation information W1 of the endoscopic image P1 selected by the user.

[0052] The selection unit 189 selects, from the finding information DB 17 , the finding information associated with the image ID of each of the endoscopic images in the second priority image group searched by the search unit 188 .

[0053] The recording control unit 190 associates the endoscopic image P1 with the finding information selected or input by the user, the doctor's name that identifies the user, and the annotation information, and records the image in each of the image information DB15, annotation information DB16, and finding information DB17.

[0054] [Processing of Medical Support Device] Next, a description will be given of the processing executed by the medical support device 10. Fig. 3 is a flowchart showing an outline of the processing executed by the medical support device 10.

[0055] 3 , first, the acquisition unit 181 acquires an endoscopic image (image data) from the endoscope 2 via the communication unit 11 (step S101). In this case, the acquisition unit 181 records (stores) the endoscopic image acquired from the endoscope 2 in the image information DB 15.

[0056] Next, the display control unit 182 displays an imaging / report screen on the display unit 12, which allows the surgeon to interpret the subject's image using the endoscopic image captured by the endoscope 2 and report and input the findings (step S102).

[0057] Fig. 4 is a diagram showing an example of an imaging / report screen displayed by the display unit 12. The imaging / report screen M1 shown in Fig. 4 includes an image display area R1 that displays a plurality of endoscopic images, and a report input area R2 in which the operator inputs the results of his / her interpretation of the subject as a report.

[0058] After step S102, if a user such as a doctor operates the input unit 13 to select an endoscopic image for interpretation and report input from among the multiple endoscopic images displayed by the display unit 12 in the image display area R1 (step S103: Yes), the display control unit 182 causes the user to operate the input unit 13 to display the endoscopic image on the display unit 12 (step S104). After step S104, the medical support device 10 proceeds to step S105, which will be described later. On the other hand, if a user such as a doctor does not operate the input unit 13 to select an endoscopic image for interpretation and report input from among the multiple endoscopic images displayed by the display unit 12 in the image display area R1 (step S103: No), the medical support device 10 proceeds to step S118, which will be described later.

[0059] Fig. 5 is a diagram showing an example of an image displayed by the display unit 12. As shown in Fig. 5, the image editing screen M2 includes an endoscopic image P1 selected by the user operating the input unit 13, and a plurality of annotation tool areas G1 for adding annotation information to the endoscopic image P1.

[0060] As shown in FIG. 5, the user selects a desired annotation tool from the annotation tool area G1 via the input unit 13 and performs an annotation operation to add annotation information to the endoscopic image P1.

[0061] 3, the description of step S105 and subsequent steps will be continued. In step S105, if the user has performed an annotation operation on the endoscopic image displayed on the display unit 12 by operating the input unit 13 (step S105: Yes), the medical support device 10 proceeds to step S106, which will be described later. On the other hand, if the user has not performed an annotation operation on the endoscopic image displayed on the display unit 12 by operating the input unit 13 (step S105: No), the medical support device 10 returns to step S104.

[0062] In step S106, the annotation assigning unit 183 assigns annotation information to the endoscopic image P1 based on the annotation operation performed by the user on the endoscopic image P1 displayed on the display unit 12 by operating the input unit 13 (step S106).

[0063] Here, the annotation information added by the annotation adding unit 183 will be described in detail.

[0064] 6 is a diagram showing an example of annotation information. As shown in Fig. 6, when a user operates the input unit 13 to select a circular template annotation tool and attach it to an endoscopic image P1, the annotation attaching unit 183 attaches annotation information W1 to the area of ​​the endoscopic image P1 where the annotation tool is attached.

[0065] 7 is a diagram showing another example of annotation information. As shown in Fig. 7, when a user operates the input unit 13 to draw a circle on an endoscopic image P1 using a free curve, the annotation assigning unit 183 assigns annotation information W2 to the area of ​​the endoscopic image P1 drawn using the free curve.

[0066] 6 and 7, the shape of the annotation is approximately circular, but is not limited to this and may be a polygon, such as a triangle, a rectangle, etc. In the case of a tumor or polyp in a subject, the user assigns annotation information using a circle or a polygon.

[0067] 8 is a diagram showing another example of annotation information. As shown in Fig. 8, when a user operates the input unit 13 to draw a circle on the endoscopic image P1 using straight lines, the annotation assigning unit 183 assigns annotation information to a region of the endoscopic image P1 that is sandwiched between the straight lines W3.

[0068] 9 is a diagram showing another example of annotation information. As shown in Fig. 9, when a user operates the input unit 13 to draw a circle on the endoscopic image P1 using a curved line, the annotation assigning unit 183 assigns annotation information to a region of the endoscopic image P1 that includes the curved line W4.

[0069] 8 and 9, the shape of the annotation is a line, but is not limited to this and may be a free-form curve, a curve, etc. When specifying the boundary between a lesion area and a normal area in a subject, the user assigns annotation information using a straight line, a free-form curve, a curve, etc.

[0070] 10 is a diagram showing another example of annotation information. As shown in Fig. 10, when a user operates the input unit 13 to select an annotation tool and attach an arrow annotation to an endoscopic image P1, the annotation attaching unit 183 attaches annotation information W5 to the area of ​​the endoscopic image P1 where the annotation is attached.

[0071] 11 is a diagram showing another example of annotation information. As shown in Fig. 11 , when a user operates the input unit 13 to select an annotation tool and attach multiple arrow annotations to an endoscopic image P1, the annotation attaching unit 183 attaches annotation information W6 to the area of ​​the endoscopic image P1 at the tip of the arrow to which the multiple annotation tools are attached.

[0072] In addition, in FIGS. 10 and 11, the annotation has an arrow shape, but is not limited to this and can be applied to designating multiple points or a range around a point.

[0073] 3, the description of step S107 and subsequent steps will be continued. In step S107, the type determining unit 184 determines the type of annotation information that the annotation providing unit 183 has provided to the endoscopic image P1.

[0074] Next, the ratio calculation unit 185 calculates the ratio of the annotation information assigned to the endoscopic image P1 by the annotation assignment unit 183 to the endoscopic image P1 (step S108).

[0075] Thereafter, the feature amount calculation unit 186 calculates the feature amount in the annotation information that the annotation adding unit 183 added to the endoscopic image P1 (step S109).

[0076] Here, the processing of each of the type determination unit 184, the ratio calculation unit 185, and the feature amount calculation unit 186 will be described in detail.

[0077] FIG. 12 is a diagram illustrating an example of processing performed by the type determination unit 184, the ratio calculation unit 185, and the feature amount calculation unit 186 on annotation information.

[0078] As shown in FIG. 12 , the type discrimination unit 184 first determines the type of annotation information W1 as a circle (○) for specifying the area of ​​the lesion in the endoscopic image P1. Furthermore, the proportion calculation unit 185 calculates the proportion of the area of ​​the annotation information W1 in the endoscopic image P1 (display area) as 8%. Furthermore, the feature calculation unit 186 calculates the feature of the annotation information W1 as "xxxxxx" using a well-known feature calculation technique. Here, the feature may be, for example, a luminance distribution, a color appearance rate, a distribution, a type, a positional relationship of an object, an edge, or the like. Furthermore, the feature calculation method may use, for example, a Haar-Like feature, a HOG feature, or a SIFT feature.

[0079] FIG. 13 is a diagram illustrating another example of processing performed by the type determination unit 184, the ratio calculation unit 185, and the feature amount calculation unit 186 on annotation information.

[0080] As shown in FIG. 13 , the type discrimination unit 184 first discriminates the type of annotation information W2 as a circle (◯) for specifying a range for specifying a lesion area in the endoscopic image P1. In this case, when the annotation information W2 is a free curve, if the start point and end point are connected, the type discrimination unit 184 discriminates the annotation information W2 as a circle (◯) for specifying a range for specifying a lesion area. Furthermore, the proportion calculation unit 185 calculates the proportion of the area of ​​the annotation information W1 in the endoscopic image P1 as 11%. Furthermore, the feature calculation unit 186 calculates the feature of the annotation information W2 as "xxxxxx" using a well-known feature calculation technique.

[0081] FIG. 14 is a diagram illustrating another example of processing performed by the type determination unit 184, the ratio calculation unit 185, and the feature amount calculation unit 186 on annotation information.

[0082] As shown in Fig. 14 , first, the type discrimination unit 184 discriminates the type of annotation information W3 as a boundary, which is a shape that specifies the boundary of a lesion in the endoscopic image P1. Furthermore, if annotation information W3 specifying multiple boundaries is present in the endoscopic image P1, the proportion calculation unit 185 calculates the proportion of the annotation information W3 occupying the endoscopic image P1, using the range connecting the multiple boundaries as a designated region (designated range) for specifying the lesion area. In this case, as shown in Fig. 14 , the proportion calculation unit 185 calculates the proportion of the annotation information W3 to be 13%. Furthermore, the feature calculation unit 186 calculates the feature of the annotation information W3 as "xxxxxx" using a well-known feature calculation technique.

[0083] FIG. 15 is a diagram illustrating another example of processing performed by the type determination unit 184, the ratio calculation unit 185, and the feature amount calculation unit 186 on annotation information.

[0084] As shown in Fig. 15 , first, the type discrimination unit 184 discriminates the type of annotation information W4 as a boundary, which is a shape that specifies the boundary of a lesion in the endoscopic image P1. Furthermore, if annotation information W4 specifying multiple boundaries is present in the endoscopic image P1, the proportion calculation unit 185 calculates the proportion of the annotation information W4 in the endoscopic image P1, using an area including the boundaries as a designated area for specifying the lesion range. In this case, as shown in Fig. 15 , the proportion calculation unit 185 calculates the proportion of the annotation information W4 as 4%. Furthermore, the feature calculation unit 186 calculates the feature of the annotation information W4 as "xxxxxx" using a well-known feature calculation technique.

[0085] FIG. 16 is a diagram illustrating another example of processing performed by the type determination unit 184, the ratio calculation unit 185, and the feature amount calculation unit 186 on annotation information.

[0086] As shown in Fig. 16 , first, the type discrimination unit 184 discriminates the type of annotation information W5 as an arrow, which is a shape that designates a point of interest of a lesion or the vicinity of the point of interest in the endoscopic image P1. Furthermore, the proportion calculation unit 185 calculates the proportion of the annotation information W5 in the endoscopic image P1, using a predetermined area pointed to by the arrow in the annotation information W5 as a designated area that designates the lesion range. In this case, as shown in Fig. 16 , the proportion calculation unit 185 calculates the proportion of the annotation information W5 to be 8%. Furthermore, the feature calculation unit 186 calculates the feature of the annotation information W5 as "xxxxxx" using a well-known feature calculation technique.

[0087] FIG. 17 is a diagram illustrating another example of processing performed by the type determination unit 184, the ratio calculation unit 185, and the feature amount calculation unit 186 on annotation information.

[0088] As shown in Fig. 17 , first, the type discrimination unit 184 discriminates the type of annotation information W6 as a plurality of arrows, which are shaped to designate a point of interest or the vicinity of the point of interest of a lesion in the endoscopic image P1. Furthermore, the proportion calculation unit 185 calculates the proportion of the annotation information W6 in the endoscopic image P1, using a predetermined region pointed to by the plurality of arrows in the annotation information W6 as a designated region. In this case, as shown in Fig. 17 , the proportion calculation unit 185 calculates the proportion of the annotation information W6 to be 3%. Furthermore, the feature calculation unit 186 calculates the feature of the annotation information W6 as "xxxxxx" using a well-known feature calculation technique.

[0089] 3 , the description of step S110 and subsequent steps will be continued. In step S110, the extraction unit 187 extracts, from the plurality of pieces of annotation information recorded in the annotation information DB 16, annotation IDs that approximate the type of annotation information determined by the type determination unit 184 and the ratio calculated by the ratio calculation unit 185, as a first priority image group, based on the type of annotation information determined by the type determination unit 184 and the ratio calculated by the ratio calculation unit 185.

[0090] 18 is a diagram schematically illustrating the extraction content extracted by the extraction unit 187 from the annotation information DB 16. As shown in Fig. 18, the annotation information DB 16 stores an annotation table T1 in which annotation IDs are associated with the types and proportions of annotation information.

[0091] 18 , when the type of annotation information determined by the type determination unit 184 is a circle ("◯") and the percentage calculated by the percentage calculation unit 185 is "8%" (see, for example, the above-mentioned FIG. 12 ), the extraction unit 187 extracts annotation IDs whose percentages are close to "8%," for example, within ±2%, from the annotation table T1. For example, in the case shown in FIG. 18 , the extraction unit 187 extracts annotation IDs "3," "5," and "7" from the annotation table T1, which correspond to the percentages "10%, "9%," and "8%," respectively.

[0092] Note that, when the type of annotation information determined by the type determination unit 184 is a circle ("○") and there are multiple annotation IDs in the annotation table T1 that are close to the ratio calculated by the ratio calculation unit 185, the extraction unit 187 extracts a predetermined number of annotation IDs from the annotation table T1, for example, 10. Of course, when there are not a predetermined number of annotation IDs in the annotation table T1 that are close to the ratio calculated by the ratio calculation unit 185, the extraction unit 187 may gradually increase the value that approximates the ratio calculated by the ratio calculation unit 185, so that the number of annotation IDs extracted from the annotation table T1 becomes a predetermined number. Furthermore, when there are a predetermined number or more annotation IDs in the annotation table T1 that are close to the ratio calculated by the ratio calculation unit 185, the extraction unit 187 may gradually decrease the value that approximates the ratio calculated by the ratio calculation unit 185, so that the number of annotation IDs extracted from the annotation table T1 becomes a predetermined number.

[0093] 3, the description of step S111 and subsequent steps will be continued. In step S111, the search unit 188 searches the image information DB 15 for feature amounts of the image IDs associated with each of the multiple annotation IDs extracted by the extraction unit 187, and searches for multiple candidate endoscopic images as a second priority image group in order of feature amounts closest to the feature amount included in the annotation information W1 of the endoscopic image P1 selected by the user.

[0094] Fig. 19 is a diagram illustrating the selection contents searched and selected by the search unit 188. As shown in Fig. 18, the image information DB 15 stores an image information table T2 in which an image ID is associated with an annotation ID and a feature amount. Note that the numerical values ​​of the feature amounts in the image information table T2 are merely examples and may be changed as appropriate depending on the feature amount extraction method.

[0095] 18 , the search unit 188 sequentially determines the degree of similarity of the feature amounts "12344," "55555," and "31234" of the image IDs associated with the multiple annotation IDs "3," "5," and "7" extracted by the extraction unit 187 from endoscopic images having feature amounts close to the numerical values ​​of the feature amounts included in the annotation information of the endoscopic image P1 selected by the user, and selects multiple candidate endoscopic images as a second priority image group in descending order of feature amount. More specifically, the search unit 188 searches for the image ID "10234" as the first candidate, the image ID "23173" as the second candidate, and the image ID "15467" as the third candidate.

[0096] In this way, the search unit 188 searches for a plurality of endoscopic images having feature amounts close to the numerical values ​​of the feature amounts included in the annotation information of the endoscopic image P1 selected by the user, and searches for these as a second priority image group from the first priority image group extracted by the extraction unit 187 using a filter that uses the annotation information W1 from the image information DB 15. As a result, the medical support device 10 can reduce the amount of processing required for feature-based searches compared to determining the degree of coincidence with the feature amounts of all endoscopic images recorded in the image information DB 15, and can shorten the processing time without improving the performance of the control unit 18.

[0097] 3, the description of step S112 and subsequent steps will be continued. In step S112, the selection unit 189 selects, from the finding information DB 17, finding information associated with the image ID of each of the endoscopic images in the second priority image group searched by the search unit 188.

[0098] Fig. 20 is a diagram illustrating the selection contents searched for and selected by the selection unit 189. As shown in Fig. 20, the finding information DB 17 stores a finding information data table T3 in which an image ID, a lesion name, and a doctor's name are associated with a lesion ID.

[0099] 20 , the selection unit 189 selects, from the finding information data table T3 recorded in the finding information DB 17, finding information associated with the image IDs "10234," "15467," and "23173" of the endoscopic images in the second priority image group searched for by the search unit 188. For example, in the case of the first candidate image ID "10234" in the second priority image group searched for by the search unit 188, the selection unit 189 selects, from the finding information data table T3, the lesion name "adenoma (Ip)" and the doctor name "Dr. AAA" as finding information.

[0100] 3, the description of step S113 and subsequent steps will be continued. In step S113, the display control unit 182 links the multiple pieces of finding information selected by the selection unit 189 to the annotation information W1 of the endoscopic image P1 as recommended finding information for the lesion in order of similarity of feature amounts in the annotation information W1, and displays them on the display unit 12.

[0101] Fig. 21 is a diagram showing an example of an image displayed by the display unit 12. As shown in Fig. 21 , the display control unit 182 displays, on the display unit 12, recommended finding information Q1, which is a compilation of multiple pieces of finding information selected by the selection unit 189, linked to annotation information W1 of the endoscopic image P1.

[0102] The recommended finding information Q1 displays a plurality of pieces of finding information selected by the selection unit 189 in order of similarity to the feature amount of the annotation information W1. Furthermore, the recommended finding information Q1 includes disease names resulting from findings by other doctors and disease names previously entered by the user regarding findings.

[0103] Returning to FIG. 3 , the description of step S114 and subsequent steps will be continued. In step S114, if the user operates the input unit 13 to select desired finding information from the recommended finding information Q1 displayed by the display unit 12 (step S114: Yes), the medical support device 10 proceeds to step S115, which will be described later. For example, the user determines that candidate 2, early colorectal cancer (Ip), which was previously input by another surgeon (another doctor), is appropriate from the recommended finding information Q1 (see FIG. 21 ) displayed by the display unit 12, and selects candidate 2, early colorectal cancer (Ip). On the other hand, if the user does not operate the input unit 13 to select desired finding information from the recommended finding information Q1 displayed by the display unit 12 (step S114: No), the medical support device 10 proceeds to step S116, which will be described later.

[0104] In step S115, the display control unit 182 displays the finding information selected by the user operating the input unit 13 in the finding input area.

[0105] Fig. 22 is a diagram showing an example of an image displayed by the display unit 12. As shown in Fig. 22, the display control unit 182 displays finding information selected by the user through operation of the input unit 13 in the finding input area. In this case, as shown in Fig. 22, when candidate 2, early colorectal cancer (Ip), is selected from the recommended finding information Q1 (see Fig. 21) displayed by the display unit 12, the display control unit 182 displays early colorectal cancer (Ip) in the finding input area R100 on the interpretation point report screen M100. After step S116, the medical support device 10 proceeds to step S117.

[0106] In step S116, the display control unit 182 displays in the finding input area the finding information input by the user operating the input unit 13. After step S116, the medical support device 10 proceeds to step S117.

[0107] Next, the recording control unit 190 associates the endoscopic image P1 with the finding information selected or input by the user, the doctor's name that identifies the user, and the annotation information, and records them in each of the image information DB15, annotation information DB16, and finding information DB17 (step S117).

[0108] Thereafter, the control unit 18 determines whether or not the user will end the input of the finding information for the endoscopic image P1 by operating the input unit 13 (step S118). If the control unit 18 determines that the user will end the input of the finding information for the endoscopic image P1 by operating the input unit 13 (step S118: Yes), the medical support device 10 ends this processing. On the other hand, if the control unit 18 determines that the user will not end the input of the finding information for the endoscopic image P1 without operating the input unit 13 (step S118: No), the medical support device 10 returns to step S102.

[0109] According to the embodiment described above, the search unit 188 searches the first priority image group extracted by the extraction unit 187 using a filter that uses the annotation information W1 from the image information DB 15 for multiple endoscopic images having feature amounts close to the numerical values ​​of the feature amounts included in the annotation information of the endoscopic image P1 selected by the user, and searches for these as a second priority image group. As a result, the medical support device 10 can reduce the amount of processing required for feature-based searches compared to when determining the degree of coincidence with the feature amounts of all endoscopic images recorded in the image information DB 15, and can shorten the processing time without improving the performance of the control unit 18.

[0110] Furthermore, according to one embodiment, the display control unit 182 links the multiple pieces of finding information selected by the selection unit 189 to the annotation information W1 of the endoscopic image P1 as recommended finding information for the lesion in order of similarity of feature amounts in the annotation information W1, and displays them on the display unit 12. As a result, the user can shorten search time with a simple configuration and input a report with simple operations.

[0111] Furthermore, according to one embodiment, the display control unit 182 displays on the display unit 12 recommended finding information Q1 including the disease name resulting from findings by other doctors and the disease name previously input by the user when the user made findings. This makes it easy to input findings from reading for a lesion annotated on the endoscopic image P1 while referring to the findings of experienced doctors and the user's own findings from the past. Furthermore, in the case of a new doctor, the findings of experienced doctors can be used as a reference, making it possible to utilize this information for training new doctors on reading images.

[0112] Various inventions can be formed by appropriately combining multiple components disclosed in the medical support system according to the embodiment of the present disclosure described above. For example, some components may be omitted from all the components described in the medical support system according to the embodiment of the present disclosure described above. Furthermore, the components described in the medical support system according to the embodiment of the present disclosure described above may be appropriately combined.

[0113] Furthermore, in the medical support system according to an embodiment of the present disclosure, the above-described "unit" can be read as "means" or "circuit," etc. For example, a control unit can be read as control means or a control circuit.

[0114] In addition, the program to be executed by the medical support system according to one embodiment of the present disclosure is provided as file data in an installable or executable format recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, a DVD (Digital Versatile Disk), a USB medium, or a flash memory.

[0115] Furthermore, the program executed by the medical support system according to one embodiment of the present disclosure may be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network.

[0116] In the description of the flowcharts in this specification, expressions such as "first," "then," and "continue" are used to clearly indicate the order of processing between steps, but the order of processing required to implement the present invention is not uniquely determined by these expressions. In other words, the order of processing in the flowcharts described in this specification can be changed within a consistent range. Furthermore, programs are not limited to those consisting of simple branching processing, and branching can be achieved by comprehensively determining more judgment items.

[0117] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be implemented in other forms that have undergone various modifications and improvements based on the knowledge of those skilled in the art, including the aspects described in the disclosure of the present invention.

[0118] REFERENCE SIGNS LIST 1 Medical support system 2 Endoscope 10 Medical support device 11 Communication unit 12 Display unit 13 Input unit 14 Recording unit 15 Image information database 16 Annotation information database 17 Findings information database 18 Control unit 141 Program recording unit 181 Acquisition unit 182 Display control unit 183 Annotation assignment unit 184 Type discrimination unit 185 Ratio calculation unit 186 Feature calculation unit 187 Extraction unit 188 Search unit 189 Selection unit 190 Recording control unit

Claims

1. A medical support device including a processor, wherein the processor, in response to an external operation, assigns annotations according to lesions as annotation information to endoscopic images of a patient under examination, calculates the ratio of the annotation information to the endoscopic image and each of the feature amounts in the annotation information, extracts a group of similar images having similar annotation information and similar ratios as a first priority image group from an annotation database that records annotation IDs that identify annotation information when annotations were assigned to each of a plurality of past images from previous examinations, in association with the type of annotation information and the ratio, and searches an image information database that records image IDs that identify each of the plurality of past images in association with the annotation IDs and feature amounts of each of the plurality of past images for images that match the annotation IDs in the first priority image group, and sequentially selects the image IDs in order of similarity in the feature amounts in the annotation information, a medical support device that selects the finding information associated with the image ID from a finding information database that records doctor's finding information in association with the image ID in order of proximity of feature amounts in the annotation information, and outputs the finding information associated with the image ID to a display unit as recommended finding information for the lesion in order of proximity of feature amounts in the annotation information.

2. A medical support device according to claim 1, wherein the annotation information is in a form that specifies the range of a lesion in the endoscopic image.

3. A medical support device according to claim 1, wherein the annotation information is a shape that specifies the boundary of a lesion in the endoscopic image.

4. A medical support device according to claim 1, wherein the annotation information has a shape that specifies a point of interest or a vicinity of a point of interest of a lesion in the endoscopic image.

5. A medical support device according to claim 1, wherein the processor extracts the group of similar images from the annotation database as a first priority group of images based on the type of annotation information and the ratio.

6. A medical support device according to claim 1, wherein the recommended finding information includes the finding information and doctor information.

7. A medical support device according to claim 1, wherein the processor displays the endoscopic image and the recommended finding information on the display unit, and the recommended finding information includes the finding information and disease name for each doctor.

8. A medical support device comprising: an endoscope capable of capturing images of the inside of a subject's body; and a medical support device capable of communicating with said endoscope, wherein said medical support device comprises a processor, which, in response to an external operation, adds annotations according to lesions to endoscopic images of a patient under examination as annotation information; calculates the ratio of said annotation information to said endoscopic image and each of the feature amounts in said annotation information; extracts a group of similar images having similar annotation information and said ratio as a first priority image group from an annotation database that records annotation IDs that identify annotation information when annotations are added to each of a plurality of past images from which examinations have been performed, in association with the type of annotation information and the ratio; searches an image information database that records image IDs that identify each of the plurality of past images in association with the annotation IDs and the feature amounts of each of the plurality of past images for images that match the annotation ID in the first priority image group, and sequentially selects said image IDs in order of similarity in the feature amount in the annotation information; a medical support system that selects the finding information associated with the image ID from a finding information database that records doctor's finding information in association with the image ID in order of proximity of feature amounts in the annotation information, and outputs the finding information associated with the image ID to a display unit as recommended finding information for the lesion in order of proximity of feature amounts in the annotation information.

9. A medical support system according to claim 8, wherein the annotation information is a shape that specifies the range of a lesion in the endoscopic image.

10. A medical support system according to claim 8, wherein the annotation information is a shape that specifies the boundary of a lesion in the endoscopic image.

11. A medical support system according to claim 8, wherein the annotation information is in a form that specifies a point of interest of a lesion in the endoscopic image or the vicinity of the point of interest.

12. A medical support system according to claim 8, wherein the processor extracts the group of similar images from the annotation database as a first priority group of images based on the type of annotation information and the ratio.

13. A medical support system according to claim 8, wherein the recommended finding information includes the finding information and doctor information.

14. A medical support system as described in claim 8, wherein the processor displays the endoscopic image and the recommended finding information on the display unit, and the recommended finding information includes the finding information and disease name for each doctor.

15. A medical support method executed by a medical support device having a processor, wherein the processor, in response to an external operation, assigns annotations according to lesions as annotation information to endoscopic images of a patient under examination, calculates the ratio of the annotation information to the endoscopic image and each of the feature amounts in the annotation information, extracts a group of similar images having similar annotation information and similar ratios as a first priority image group from an annotation database that records annotation IDs that identify annotation information when annotations are assigned to each of a plurality of past images from previous examinations, in association with the type of annotation information and the ratio, and searches an image information database that records image IDs that identify each of the plurality of past images in association with the annotation IDs and feature amounts of each of the plurality of past images for images that match the annotation IDs in the first priority image group, and sequentially selects the image IDs in order of similarity in the feature amount in the annotation information, a medical support method comprising: selecting the finding information associated with the image ID from a finding information database that records doctor's finding information in association with the image ID in order of proximity of feature amounts in the annotation information; and outputting the finding information associated with the image ID to a display unit as recommended finding information for the lesion in order of proximity of feature amounts in the annotation information.

16. A medical support method according to claim 15, wherein the annotation information is a shape that specifies the range of a lesion in the endoscopic image.

17. A medical support method according to claim 15, wherein the annotation information is a shape that specifies the boundary of a lesion in the endoscopic image.

18. A medical support method according to claim 15, wherein the annotation information has a shape that specifies a point of interest or the vicinity of a point of interest of a lesion in the endoscopic image.

19. A medical support method according to claim 15, wherein the processor extracts the group of similar images from the annotation database as a first priority group of images based on the type of annotation information and the ratio.

20. A medical support method according to claim 15, wherein the recommended finding information includes the finding information and doctor information.

Images