Medical support device, endoscope, medical support method, and program

Abstract

To provide a medical support device, an endoscope, a medical support method, and a program capable of contributing to causing a user, etc. to observe a region that has been observed in a luminal organ through an endoscope scope by the user, etc. while the endoscope scope is inserted into the luminal organ, through the endoscope scope again without omission.SOLUTION: A medical support device acquires a first medical image and a second medical image while an endoscope scope is inserted into a luminal organ. A processor outputs the first medical image and / or the second medical image when a degree of similarity between a first partial region and a second partial region exceeds a threshold. The output first medical image is an image expressed in a mode by which the first partial region and the first other region can be discriminated. The output second medical image is an image expressed in a mode by which the second partial region and the second other region can be discriminated.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present disclosure relates to a medical support device, an endoscope, a medical support method, and a program. [Background technology]

[0002] Patent Document 1 discloses an image processing device that may include an image input unit, a parameter calculation unit, and a display control unit. In the image processing device described in Patent Document 1, the image input unit inputs a first image and a second image captured at different times. The image input unit also inputs a first image captured with a first observation light and a second image captured with a second observation light different from the first observation light. The parameter calculation unit calculates parameters for aligning the first image and the second image. The image generation unit applies the parameters to the first image to generate an aligned first image. The display control unit sequentially displays the input first image and the generated aligned first image on a display device.

[0003] Patent Document 2 discloses an image processing device including an image acquisition unit, a recognition unit, and a matching unit. In the image processing device described in Patent Document 2, the image acquisition unit acquires a first image and a second image including a specific site in a subject, which are captured at different times in a time series. The recognition unit recognizes the specific site in each image acquired by the image acquisition unit, and recognizes a plurality of first specific site candidates in the first image and a plurality of second specific site candidates in the second image. The matching unit matches each of the first specific site candidates with each of the second specific site candidates based on the difference between the feature amount of each of the plurality of first specific site candidates and the feature amount of each of the plurality of second specific site candidates, and identifies a correspondence relationship that represents the same specific site in the first image and the second image.

[0004] Patent Document 3 discloses a magnetic resonance imaging apparatus including a magnetic field generating unit, a detecting unit, and a processing unit. In the magnetic resonance imaging apparatus described in Patent Document 3, the magnetic field generating unit generates a uniform static magnetic field in a space accommodating a subject, and a gradient magnetic field superimposed on the static magnetic field. The detecting unit irradiates a high-frequency magnetic field to the subject and detects a nuclear magnetic resonance signal generated from the subject. The processing unit converts the detected nuclear magnetic resonance signal into an image. The processing unit also acquires past examination information from past images obtained at different dates and times or with different devices, determines current examination information using the acquired examination information, and displays the current image acquired based on the determined examination information in a state in which the past image can be compared. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2019 / 167623 [Patent Document 2] JP 2017-097836 A [Patent Document 3] JP 2016-209336 A Summary of the Invention

[0006] One embodiment of the present disclosure provides a medical support device, an endoscope, a medical support method, and a program that can help a user, etc. to thoroughly observe again through an endoscope an area observed within a tubular organ by a user, etc. through an endoscope scope while the endoscope scope is inserted into the tubular organ. [Means for solving the problem]

[0007] A first aspect of the present disclosure is a medical support device including a processor, the processor acquiring a first medical image obtained by imaging the inside of a tubular organ with an endoscopic scope while the endoscopic scope is inserted into the tubular organ, and a second medical image obtained temporally after the first medical image by imaging the inside of the tubular organ with the endoscopic scope, and when a similarity between a first partial region that is a part of the first medical image and a second partial region that is a part of the second medical image exceeds a threshold, the processor executing a first output process to output the first medical image and / or a second output process to output the second medical image, wherein the first medical image output by executing the first output process is an image in which the first partial region and a first other region that is another region in the first medical image are represented in a manner in which they are distinguishable, and the second medical image output by executing the second output process is an image in which the second partial region and a second other region that is another region in the second medical image are represented in a manner in which they are distinguishable.

[0008] A second aspect of the present disclosure is a medical support device according to the first aspect, in which a first medical image output by executing a first output process is an image in which a first partial region is emphasized more than a first other region, and a second medical image output by executing a second output process is an image in which a second partial region is emphasized more than a second other region.

[0009] A third aspect of the present disclosure is a medical support device according to the first or second aspect, wherein the first medical image is an image obtained during an insertion process in which an endoscopic scope is inserted into a tubular organ, and the second medical image is an image obtained during a removal process in which the endoscopic scope is removed from the tubular organ.

[0010] A fourth aspect of the present disclosure is a medical support device relating to any one of the first to third aspects, wherein the first partial region is at least one of a plurality of first divided regions obtained by dividing a first medical image according to a first rule, and the second partial region is at least one of a plurality of second divided regions obtained by dividing a second medical image according to the first rule.

[0011] A fifth aspect of the present disclosure is a medical support device according to the fourth aspect, in which a processor derives a map indicating a distribution of similarities between a plurality of first partitioned regions and a plurality of second partitioned regions based on a first medical image and a second medical image, and extracts, from the plurality of first partitioned regions and the plurality of second partitioned regions, first partitioned regions and second partitioned regions whose similarity exceeds a threshold value according to the derived map, as first partial regions and second partial regions whose similarity exceeds the threshold value.

[0012] A sixth aspect of the present disclosure is a medical support device according to the fifth aspect, in which the processor derives the map using AI.

[0013] A seventh aspect of the present disclosure is the medical support device according to the fifth or sixth aspect, in which the map is derived in response to a first instruction given from outside.

[0014] An eighth aspect of the present disclosure is a medical support device relating to any one of the first to third aspects, wherein the first partial region is at least one of a plurality of third divided regions obtained by dividing the first region, which is a part of the image in the first medical image, in accordance with a second rule, and the second partial region is at least one of a plurality of fourth divided regions obtained by dividing the second region, which is a part of the image in the second medical image, in accordance with the second rule.

[0015] A ninth aspect of the present disclosure is a medical support device according to the eighth aspect, in which a processor derives a similarity between each of the multiple third partitioned regions and each of the multiple fourth partitioned regions based on each of the multiple third partitioned regions and each of the multiple fourth partitioned regions, and extracts from the multiple third partitioned regions and the multiple fourth partitioned regions those third partitioned regions and fourth partitioned regions whose similarity exceeds a threshold as first partial regions and second partial regions whose similarity exceeds the threshold.

[0016] A tenth aspect of the present disclosure is a medical support device according to the ninth aspect, in which the processor derives the similarity using AI.

[0017] An eleventh aspect of the present disclosure is the medical support device according to the tenth aspect, in which the similarity is derived in response to a second instruction given from outside.

[0018] A twelfth aspect of the present disclosure is a medical support device according to any one of the eighth to eleventh aspects, wherein each of the first and second areas is a feature area recognized by performing object recognition processing as an area having a predetermined characteristic.

[0019] A thirteenth aspect of the present disclosure is a medical support device according to the twelfth aspect, wherein the feature region is a region depicting a lesion, a marked region, a region depicting an organ, and / or a region depicting a treatment tool.

[0020] A fourteenth aspect of the present disclosure is the medical support device according to the twelfth or thirteenth aspect, in which an object recognition AI is used for the object recognition process.

[0021] A fifteenth aspect of the present disclosure is a medical support device relating to any one of the twelfth to fourteenth aspects, in which a first medical image is stored in a first storage area on the condition that the first area is recognized as a characteristic area by performing object recognition processing.

[0022] A sixteenth aspect of the present disclosure is a medical support device according to any one of the first to fifteenth aspects, wherein the first medical image is stored in the second storage area in accordance with a third instruction given from outside.

[0023] A 17th aspect of the present disclosure is a medical support device relating to any one of the first to sixteenth aspects, wherein the first medical image and the second medical image are images obtained by imaging an area within a tubular organ where the same type of light is irradiated using an endoscopic scope.

[0024] An 18th aspect of the present disclosure is a medical support device relating to any one of the 1st to 17th aspects, in which a first medical image output by executing a first output process is displayed on a first screen, and a second medical image output by executing a second output process is displayed on a second screen.

[0025] A 19th aspect of the present disclosure is a medical support device comprising a processor, wherein the processor acquires a first medical image obtained by imaging the inside of a tubular organ with an endoscopic scope while the endoscopic scope is inserted into the tubular organ, and a second medical image obtained temporally after the first medical image by imaging the inside of the tubular organ with the endoscopic scope, and when a degree of difference between a first partial region that is a part of the first medical image and a second partial region that is a part of the second medical image is less than a threshold, performs a first output process to output the first medical image and / or a second output process to output the second medical image, wherein the first medical image output by executing the first output process is an image in which the first partial region and a first other region that is another region in the first medical image are represented in a manner in which they are distinguishable, and the second medical image output by executing the second output process is an image in which the second partial region and a second other region that is another region in the second medical image are represented in a manner in which they are distinguishable.

[0026] A twentieth aspect of the present disclosure is an endoscope including a medical support device according to any one of the first to nineteenth aspects and an endoscope scope.

[0027] A 21st aspect of the present disclosure is a medical support method including obtaining a first medical image obtained by imaging the inside of a tubular organ with an endoscopic scope while the endoscopic scope is inserted into the tubular organ, and a second medical image obtained temporally after the first medical image by imaging the inside of the tubular organ with the endoscopic scope, and executing a first output process to output the first medical image and / or a second output process to output the second medical image when a similarity between a first partial region that is a part of the first medical image and a second partial region that is a part of the second medical image exceeds a threshold, wherein the first medical image output by executing the first output process is an image in which the first partial region and a first other region that is another region in the first medical image are represented in a distinguishable manner, and the second medical image output by executing the second output process is an image in which the second partial region and a second other region that is another region in the second medical image are represented in a distinguishable manner.

[0028] A 22nd aspect of the present disclosure is a program for causing a computer to execute medical support processing, the medical support processing including acquiring, while an endoscope scope is inserted into a hollow organ, a first medical image obtained by imaging the inside of the hollow organ with an endoscope scope, and a second medical image obtained temporally after the first medical image by imaging the inside of the hollow organ with the endoscope scope, and detecting a similarity between a first partial region that is a part of the first medical image and a second partial region that is a part of the second medical image. The program includes executing a first output process to output a first medical image and / or a second output process to output a second medical image when the degree of discrimination exceeds a threshold, wherein the first medical image output by executing the first output process is an image in which the first partial region and a first other region that is another region in the first medical image are represented in a manner that is distinguishable from each other, and the second medical image output by executing the second output process is an image in which the second partial region and a second other region that is another region in the second medical image are represented in a manner that is distinguishable from each other. [Brief description of the drawings]

[0029] [Figure 1] FIG. 1 is a conceptual diagram showing an example of an embodiment in which an endoscope is used. [Diagram 2] 1 is a conceptual diagram showing an example of an overall configuration of an endoscope. [Diagram 3] 2 is a block diagram showing an example of a hardware configuration of an electrical system of the endoscope; [Figure 4] 2 is a block diagram showing an example of main functions of a processor included in the medical support device, and an example of information stored in a storage. [Diagram 5] FIG. 4 is a conceptual diagram showing an example of processing contents of a recognition unit and a control unit. [Figure 6] 11 is a conceptual diagram showing an example of processing contents of a recognition unit and a control unit in an insertion process. FIG. [Figure 7] 13 is a conceptual diagram showing an example of processing contents of a recognition unit and a control unit in a removal process. FIG. [Figure 8] 13 is a conceptual diagram showing an example of a process for deriving a similarity between a lesion shown in an insertion step frame and a lesion shown in a removal step frame. FIG. [Figure 9] 13 is a conceptual diagram showing an example of processing by the control unit using a removal step frame and map when a high similarity distribution region is obtained, and processing by the control unit using an insertion step frame and map. FIG. [Figure 10] 13 is a conceptual diagram showing an example of processing performed by a control unit using a frame with an insertion tag and a frame with a removal tag. FIG. [Figure 11A] 13 is a flowchart showing an example of the flow of a medical support process. [Figure 11B] This is a continuation of the flowchart shown in FIG. 11A. [Figure 12] FIG. 11 is a conceptual diagram showing a first modified example of the processing content by the control unit according to the first embodiment. [Figure 13] FIG. 13 is a conceptual diagram showing a second modified example of the processing content by the control unit according to the first embodiment. [Figure 14] FIG. 13 is a conceptual diagram showing a third modified example of the processing content by the control unit according to the first embodiment. [Figure 15]FIG. 13 is a conceptual diagram showing a fourth modified example of the processing content by the control unit according to the first embodiment. [Figure 16] FIG. 13 is a conceptual diagram showing a fifth modified example of the processing content by the control unit according to the first embodiment. [Figure 17] FIG. 13 is a conceptual diagram showing a sixth modified example of the processing content by the control unit according to the first embodiment. [Figure 18] 13 is a conceptual diagram showing an example of the processing contents of a recognition unit and a control unit in an insertion step when an endoscope according to a second embodiment is used. FIG. [Figure 19] FIG. 13 is a conceptual diagram showing an example of processing contents for deriving a similarity between a lesion included in an insertion step frame and a lesion included in a removal step frame when an endoscope according to a second embodiment is used; [Figure 20] FIG. 11 is a conceptual diagram showing an example of processing by the control unit using the lesion area and map in a removal process and processing by the control unit using the lesion area and map in an insertion process when a high similarity distribution area is obtained by using an endoscope according to a second embodiment; [Figure 21] A conceptual diagram showing an example of a series of processes in which a processor included in a computer issues a processing execution request to an external device via a network, the external device executes processing in response to the processing execution request, and the processor included in the computer receives the processing result from the external device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] Hereinafter, examples of embodiments of a medical support device, an endoscope, a medical support method, and a program according to the present disclosure will be described with reference to the accompanying drawings.

[0031] First, the terms used in the following description will be explained.

[0032] CPU is an abbreviation for "Central Processing Unit". GPU is an abbreviation for "Graphics Processing Unit". GPGPU is an abbreviation for "General-Purpose computing on Graphics Processing Units". APU is an abbreviation for "Accelerated Processing Unit". TPU is an abbreviation for "Tensor Processing Unit". RAM is an abbreviation for "Random Access Memory". NVM is an abbreviation for "Non-volatile memory". EEPROM is an abbreviation for "Electrically Erasable Programmable Read-Only Memory". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". SSD is an abbreviation for "Solid State Drive". USB is an abbreviation for "Universal Serial Bus". HDD is an abbreviation for "Hard Disk Drive". EL is an abbreviation for "Electro-Luminescence". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". AI is an abbreviation for "Artificial Intelligence". BLI is an abbreviation for "Blue Light Imaging". LCI is an abbreviation for "Linked Color Imaging". I / F is an abbreviation for "Interface". SSL is an abbreviation for "Sessile Serrated Lesion". LAN is an abbreviation for "Local Area Network". WAN is an abbreviation for "Wide Area Network".5G is an abbreviation for "5th Generation Mobile Communication System."

[0033] In the following description, a coded processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Also, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU, a GPU, a GPGPU, an APU, and a TPU.

[0034] In the following description, a signed memory is a memory, such as a RAM, in which information is temporarily stored and which is used as a working memory by a processor.

[0035] In the following description, a storage with a symbol is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory, magnetic disks, and magnetic tapes. Another example of storage is cloud storage.

[0036] In the following embodiments, the external I / F with a symbol controls the transmission and reception of various information between multiple devices connected to each other. An example of the external I / F is a USB interface. A communication I / F including a communication processor and an antenna may be applied to the external I / F. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G, Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0037] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0038] [First embodiment] Fig. 1 is a conceptual diagram showing an example of an embodiment in which an endoscope 10 is used. As shown in Fig. 1, the endoscope 10 is used by a doctor 12 in an endoscopic examination or the like. The endoscopic examination is assisted by a staff member such as a nurse 14.

[0039] The endoscope 10 is communicatively connected to a communication device (not shown), and information obtained by the endoscope 10 is transmitted to the communication device. Examples of the communication device include a server that manages various information such as electronic medical records, a personal computer, and / or a taglet terminal. The communication device receives the information transmitted from the endoscope 10 and executes processing using the received information (e.g., processing to store the information in an electronic medical record, etc.).

[0040] The endoscope 10 includes an endoscope scope 16, a display device 18, a light source device 20, a control device 22, and a medical support device 24. In the first embodiment, the endoscope 10 is an example of an "endoscope" according to the present disclosure, and the endoscope scope 16 is an example of an "endoscope scope" according to the present disclosure.

[0041] The endoscope 10 is a modality for performing medical treatment on a large intestine 28 contained in the body of a subject 26 (e.g., a patient) using an endoscope scope 16. In this first embodiment, the large intestine 28 is an object to be observed by a doctor 12.

[0042] The endoscope 16 is used by the doctor 12 and inserted into a hollow organ of the subject 26. In the present first embodiment, the endoscope 16 is inserted into the large intestine 28 of the subject 26. The large intestine 28 is an example of a "hollow organ" according to the present disclosure.

[0043] The endoscope 10 causes the endoscope scope 16 inserted into the large intestine 28 of the subject 26 to capture images of the inside of the large intestine 28 of the subject 26, and also performs various medical procedures on the large intestine 28 as necessary.

[0044] The endoscope 10 captures an image of the inside of the large intestine 28 of the subject 26, thereby acquiring and outputting an image showing the state inside the large intestine 28. In the present first embodiment, the endoscope 10 is an endoscope having an optical imaging function for capturing an image of reflected light obtained by irradiating the inside of the large intestine 28 with light 30 and reflecting it off a wall 32 of the large intestine 28.

[0045] Although an endoscopic examination of the large intestine 28 is illustrated here, this is merely one example, and the present disclosure also applies to an endoscopic examination of a hollow organ such as the esophagus, stomach, duodenum, or trachea.

[0046] The light source device 20, the control device 22, and the medical support device 24 are installed on a wagon 34. The wagon 34 has a plurality of tables arranged vertically, and the medical support device 24, the control device 22, and the light source device 20 are installed from the lower table to the upper table. In addition, the display device 18 is installed on the top table of the wagon 34.

[0047] The control device 22 controls the entire endoscope 10. The medical support device 24, under the control of the control device 22, performs various types of image processing on the images obtained by imaging the intestinal wall 32 by the endoscope scope 16.

[0048] The display device 18 displays various information including images. Examples of the display device 18 include a liquid crystal display and an EL display. Instead of the display device 18 or together with the display device 18, a taglet terminal with a display may be used.

[0049] The display device 18 displays a screen 35. The screen 35 includes a plurality of display areas. The plurality of display areas are arranged side by side within the screen 35. In the example shown in FIG. 1, a first display area 36 and a second display area 38 are shown as an example of the plurality of display areas. The size of the first display area 36 is larger than the size of the second display area 38. The first display area 36 is used as a main display area, and the second display area 38 is used as a sub display area. Note that the size relationship between the first display area 36 and the second display area 38 is not limited thereto, and may be any size relationship that fits within the screen 35. In the first embodiment, the first display area 36 is an example of a "second screen" according to the present disclosure, and the second display area 38 is an example of a "first screen" according to the present disclosure.

[0050] An endoscopic moving image 39 is displayed in the first display area 36. The endoscopic moving image 39 is a moving image acquired by imaging an intestinal wall 32 in the large intestine 28 of the subject 26 with the endoscope scope 16. In the example shown in Fig. 1, a moving image showing the intestinal wall 32 is shown as an example of the endoscopic moving image 39.

[0051] The intestinal wall 32 shown in the endoscopic video 39 includes a lesion 42 (e.g., one lesion 42 in the example shown in FIG. 1) as an area of ​​interest (i.e., an area to be observed) gazed upon by the doctor 12, and the doctor 12 can visually recognize the appearance of the intestinal wall 32 including the lesion 42 through the endoscopic video 39. In this first embodiment, the lesion 42 is an example of a "lesion" according to the present disclosure.

[0052] There are various types of lesions 42, and examples of the types of lesions 42 include neoplastic polyps and non-neoplastic polyps. Examples of the types of neoplastic polyps include adenomatous polyps (e.g., SSL). Examples of the types of non-neoplastic polyps include hamartomatous polyps, hyperplastic polyps, and inflammatory polyps. Note that the types exemplified here are types that are assumed in advance as types of lesions 42 when an endoscopic examination is performed on the large intestine 28, and the types of lesions 42 will differ if the organ in which the endoscopic examination is performed is different.

[0053] In this first embodiment, for ease of explanation, an example is given in which one lesion 42 is captured in the endoscopic video 39, but the present disclosure is not limited to this, and the present disclosure also applies when multiple lesions 42 are captured in the endoscopic video 39.

[0054] In this first embodiment, a lesion 42 is illustrated, however, this is merely one example, and the area of ​​interest (i.e., the area to be observed) gazed upon by the doctor 12 may be an organ (e.g., the duodenal papilla), a mark, an artificial treatment tool (e.g., an artificial clip), or a treated area (e.g., an area where traces remain after the removal of a polyp, etc.), etc.

[0055] The image displayed in the first display area 36 is one frame 40 included in a moving image that includes a plurality of frames 40 arranged in time series. In other words, the plurality of frames 40 arranged in time series are displayed in the first display area 36 at a default frame rate (e.g., several tens of frames per second).

[0056] An example of a moving image displayed in the first display area 36 is a moving image of a live view system. The live view system is merely one example, and a moving image that is temporarily stored in a memory or the like and then displayed, such as a moving image of a post-view system, may also be used. Furthermore, each frame included in a moving image for recording stored in a memory or the like may be reproduced and displayed on the screen 35 (for example, the first display area 36) as an endoscopic moving image 39.

[0057] Within the screen 35, the second display area 38 is adjacent to the first display area 36, ​​and is displayed at the lower right when viewed from the front within the screen 35. The display position of the second display area 38 may be anywhere within the screen 35 of the display device 18, but it is preferable that it is displayed at a position where it can be contrasted with the endoscopic moving image 39.

[0058] The second display area 38 displays auxiliary information 44 that assists the doctor 12 in making medical decisions during an endoscopic examination. Examples of the auxiliary information 44 include various types of information related to the subject 26 into whose body the endoscope 16 is inserted, and / or various types of information obtained by performing medical support processing, which will be described later. In the example shown in FIG. 1, a reference image 44A is displayed in the second display area 38. The reference image 44A is an image referred to by the doctor 12, and is obtained by performing medical support processing, which will be described later.

[0059] The endoscopic examination of the large intestine 28 includes an insertion step in which the endoscope 16 is inserted into the large intestine 28, and a removal step in which the endoscope 16 is removed from the large intestine 28. In the insertion step, the endoscope 16 is inserted into the large intestine 28 along an insertion path 29A. For example, the insertion path 29A refers to a path from the anus (not shown) to the ileocecal portion 28A. In the removal step, the endoscope 16 is removed from the large intestine 28 along a removal path 29B. For example, the removal path 29B refers to a path from the ileocecal portion 28A to the anus. In the insertion step, when the endoscope 16 reaches the ileocecal portion 28A of the large intestine 28 along the insertion path 29A, the insertion step is switched to a removal step, and the endoscope 16 is removed from the large intestine 28 along the removal path 29B. In the insertion step, the endoscope 10 captures an image of the inside of the large intestine 28. In the removal step, the endoscope 10 captures an image of the inside of the large intestine 28, as in the insertion step. Furthermore, during the removal process, a medical procedure (eg, identification, marking, resection, and / or hemostasis, etc.) is performed by the endoscope 10.

[0060] Fig. 2 is a conceptual diagram showing an example of the overall configuration of the endoscope 10. As shown in Fig. 2, the endoscope scope 16 includes an operation section 46 and an insertion section 48. The insertion section 48 is partially curved by operating the operation section 46. The insertion section 48 is inserted into the large intestine 28 (see Fig. 1) while curving in accordance with the shape of the large intestine 28 (see Fig. 1) in accordance with the operation of the operation section 46 by the doctor 12 (see Fig. 1).

[0061] The distal end 50 of the insertion section 48 is provided with a camera 52, an illumination device 54, and a treatment tool opening 56. The camera 52 and the illumination device 54 are provided on the distal end surface 50A of the distal end 50. Note that, although an example in which the camera 52 and the illumination device 54 are provided on the distal end surface 50A of the distal end 50 is given here, this is merely one example, and the camera 52 and the illumination device 54 may be provided on the side surface of the distal end 50, so that the endoscope 16 is configured as a side-viewing scope.

[0062] The camera 52 is inserted into the body cavity of the subject 26 to capture an image of an observation target region. In the first embodiment, the camera 52 captures an image of the inside of the subject 26 (for example, inside the large intestine 28) to obtain an endoscopic moving image 39. An example of the camera 52 is a CMOS camera. However, this is merely an example, and other types of cameras such as a CCD camera may also be used.

[0063] The illumination device 54 has illumination windows 54A and 54B. The illumination device 54 irradiates light 30 (see FIG. 1) through the illumination windows 54A and 54B. Examples of the light 30 irradiated from the illumination device 54 include visible light (e.g., white light) and non-visible light (e.g., near-infrared light). The illumination device 54 also irradiates special light through the illumination windows 54A and 54B. Examples of the special light include light for BLI and / or light for LCI. The camera 52 captures an image of the inside of the large intestine 28 by an optical method while the light 30 is irradiated inside the large intestine 28 by the illumination device 54.

[0064] The treatment tool opening 56 is an opening for allowing a treatment tool 58 to protrude from the distal end portion 50. The treatment tool opening 56 is also used as a suction port for sucking blood, internal waste, and the like, and as a delivery port for delivering fluid.

[0065] A treatment tool insertion port 60 is formed in the operation section 46, and the treatment tool 58 is inserted into the insertion section 48 from the treatment tool insertion port 60. The treatment tool 58 passes through the insertion section 48 and protrudes to the outside from the treatment tool opening 56. In the example shown in FIG. 2, a puncture needle is shown as the treatment tool 58 protruding from the treatment tool opening 56. Here, a puncture needle is shown as the treatment tool 58, but this is merely one example, and the treatment tool 58 may be a grasping forceps, a papillotomy knife, a snare, a catheter, a guidewire, a cannula, and / or a puncture needle with a guide sheath, etc.

[0066] The endoscope 16 is connected to a light source device 20 and a control device 22 via a universal cord 62. A medical support device 24 and a reception device 64 are connected to the control device 22. In addition, a display device 18 is connected to the medical support device 24. That is, the control device 22 is connected to the display device 18 via the medical support device 24.

[0067] In this example, the medical support device 24 is shown as an external device for expanding the functions performed by the control device 22, and therefore an example is shown in which the control device 22 and the display device 18 are indirectly connected via the medical support device 24, but this is merely one example. For example, the display device 18 may be directly connected to the control device 22. In this case, for example, the function of the medical support device 24 may be installed in the control device 22, or the control device 22 may be equipped with a function to cause a server (not shown) to execute the same process as the process executed by the medical support device 24 (for example, the medical support process described later) and to receive and use the results of the process by the server.

[0068] The reception device 64 receives instructions from the doctor 12 and outputs the received instructions as an electrical signal to the control device 22. Examples of the reception device 64 include a keyboard, a mouse, a touch panel, a foot switch, a microphone, and / or a remote control device.

[0069] The control device 22 controls the light source device 20, exchanges various signals with the camera 52, and exchanges various signals with the medical support device 24.

[0070] The light source device 20 emits light under the control of the control device 22, and supplies the light to the illumination device 54. A light guide is built into the illumination device 54, and the light supplied from the light source device 20 passes through the light guide and is irradiated from illumination windows 54A and 54B. The control device 22 causes the camera 52 to capture an image, acquires an endoscopic video image 39 (see FIG. 1) from the camera 52, and outputs it to a predetermined output destination (for example, the medical support device 24).

[0071] The medical support device 24 performs various types of image processing on the endoscopic moving image 39 input from the control device 22 to provide support for medical care (here, as an example, an endoscopic examination). The medical support device 24 outputs the endoscopic moving image 39 that has been subjected to various types of image processing to a predetermined output destination (for example, the display device 18).

[0072] Although an example in which the endoscopic moving image 39 output from the control device 22 is output to the display device 18 via the medical support device 24 has been described here, this is merely one example. For example, the control device 22 and the display device 18 may be connected, and the endoscopic moving image 39 that has been subjected to image processing by the medical support device 24 may be displayed on the display device 18 via the control device 22.

[0073] Fig. 3 is a block diagram showing an example of a hardware configuration of an electrical system of the endoscope 10. As shown in Fig. 3, the control device 22 includes a computer 66, a bus 68, and an external I / F 70. The computer 66 includes a processor 72, a memory 74, and a storage 76. The processor 72, the memory 74, the storage 76, and the external I / F 70 are connected to the bus 68. The processor 72 controls the entire control device 22. The memory 74 and the storage 76 are used by the processor 72.

[0074] The external I / F 70 is responsible for transmitting and receiving various types of information between the processor 72 and one or more devices (hereinafter, also referred to as “first external devices”) that exist outside the control device 22.

[0075] The camera 52 is connected to the external I / F 70 as one of the first external devices, and the external I / F 70 is responsible for transmitting and receiving various types of information between the camera 52 and the processor 72. The processor 72 controls the camera 52 via the external I / F 70. The processor 72 also acquires, via the external I / F 70, an endoscopic moving image 39 (see FIG. 1) obtained by the camera 52 capturing an image of the inside of the large intestine 28 (see FIG. 1).

[0076] The light source device 20 is connected to the external I / F 70 as one of the first external devices, and the external I / F 70 controls the exchange of various information between the light source device 20 and the processor 72. The light source device 20 supplies light to the illumination device 54 under the control of the processor 72. The illumination device 54 irradiates the light supplied from the light source device 20.

[0077] A reception device 64 is connected to the external I / F 70 as one of the first external devices, and the processor 72 acquires instructions accepted by the reception device 64 via the external I / F 70 and executes processing according to the acquired instructions.

[0078] The medical support device 24 includes a computer 78 and an external I / F 80. The computer 78 includes a processor 82, a memory 84, and a storage 86. The processor 82, the memory 84, the storage 86, and the external I / F 80 are connected to a bus 88. In the first embodiment, the medical support device 24 is an example of a "medical support device" according to the present disclosure, the computer 78 is an example of a "computer" according to the present disclosure, and the processor 82 is an example of a "processor" according to the present disclosure.

[0079] The hardware configuration of the computer 78 (i.e., the processor 82, the memory 84, and the storage 86) is basically the same as the hardware configuration of the computer 66, so a description of the hardware configuration of the computer 78 will be omitted here.

[0080] The external I / F 80 controls the exchange of various information between the processor 82 and one or more devices (hereinafter, also referred to as “second external devices”) that exist outside the medical support device 24.

[0081] The control device 22 is connected to the external I / F 80 as one of the second external devices. In the example shown in Fig. 3, the external I / F 70 of the control device 22 is connected to the external I / F 80. The external I / F 80 controls the exchange of various information between the processor 82 of the medical support device 24 and the processor 72 of the control device 22. For example, the processor 82 acquires endoscopic moving images 39 (see Fig. 1) from the processor 72 of the control device 22 via the external I / Fs 70 and 80, and performs various image processing on the acquired endoscopic moving images 39.

[0082] The display device 18, which serves as one of the second external devices, is connected to the external I / F 80. The processor 82 controls the display device 18 via the external I / F 80 to cause the display device 18 to display various information (e.g., an endoscopic moving image 39 that has been subjected to various image processing, etc.).

[0083] During an endoscopic examination, the doctor 12 checks the endoscopic moving image 39 via the display device 18 and determines whether or not medical treatment is required for the lesion 42 shown in the endoscopic moving image 39, and performs medical treatment on the lesion 42 if necessary.

[0084] In a typical endoscopic examination, when the doctor 12 finds a lesion 42 via the screen 35 (see FIG. 1) during the insertion process (see FIG. 1), the doctor 12 performs a medical procedure on the lesion 42 found during the insertion process during the removal process. In this case, the doctor 12 needs to visually identify all of the lesions 42 found during the insertion process again during the removal process.

[0085] For example, if the number of lesions 42 discovered during the insertion process is one, the doctor 12 can almost always find the lesion 42 discovered during the insertion process during the removal process. However, if the number of lesions 42 discovered during the insertion process reaches multiple, the doctor 12 may miss some of the multiple lesions 42 discovered during the insertion process during the removal process. The greater the number of lesions 42 discovered during the insertion process, the higher the possibility that the doctor 12 will miss a lesion 42 discovered during the insertion process during the removal process.

[0086] In order to ensure that medical treatment is performed on each and every lesion 42 discovered during endoscopic examination, it is extremely important that all lesions 42 discovered during the insertion process are visually identified again during the removal process, regardless of the number of lesions 42 discovered during the insertion process.

[0087] In view of such circumstances, in the first embodiment, as an example shown in Fig. 4, medical support processing is performed by a processor 82 of the medical support device 24. Note that Fig. 4 is a block diagram showing an example of main functions of the processor 82 included in the medical support device 24 and an example of information stored in a storage 86.

[0088] A medical support program 90 is stored in the storage 86. The medical support program 90 is an example of a "program" according to the present disclosure. The processor 82 reads the medical support program 90 from the storage 86 and executes the read medical support program 90 on the memory 84 to perform medical support processing. The medical support processing is realized by the processor 82 operating as a recognition unit 82A and a control unit 82B in accordance with the medical support program 90 executed on the memory 84.

[0089] The storage 86 stores a recognition model 92 and a similarity derivation model 94. As will be described in detail later, the recognition model 92 is used by the recognition unit 82A, and the similarity derivation model 94 is used by the control unit 82B.

[0090] Fig. 5 is a conceptual diagram showing an example of the processing contents of the recognition unit 82A and the control unit 82B. As shown in Fig. 5, the recognition unit 82A and the control unit 82B acquire, from the camera 52, each of a plurality of frames 40 in time series included in the endoscopic moving image 39 generated by capturing images by the camera 52 at an imaging frame rate (e.g., several tens of frames / second), in a frame-by-frame manner in time series.

[0091] The control unit 82B outputs the endoscopic moving image 39 to the display device 18. For example, the control unit 82B displays the endoscopic moving image 39 as a live view image in the first display area 36. That is, every time the control unit 82B acquires a frame 40 from the camera 52, the control unit 82B displays the acquired frame 40 in sequence in the first display area 36 according to a display frame rate (for example, several tens of frames per second). The control unit 82B also displays auxiliary information 44 in the second display area 38. For example, the control unit 82B also updates the display content of the second display area 38 (for example, auxiliary information 44) in accordance with the display content of the first display area 36.

[0092] The recognition unit 82A uses the endoscopic video 39 acquired from the camera 52 to recognize a lesion 42 in the endoscopic video 39. That is, the recognition unit 82A sequentially performs a recognition process 96 on each of a plurality of frames 40 in a time series included in the endoscopic video 39 acquired from the camera 52, thereby recognizing the lesion 42 appearing in the frame 40. For example, the recognition unit 82A recognizes the geometric characteristics (e.g., position and shape, etc.) of the lesion 42, the type of the lesion 42, and the type of the lesion 42 (e.g., pedunculated, subpedunculated, sessile, surface elevated, surface flat, surface depressed, etc.). In the first embodiment, the recognition process 96 is an example of the "object recognition process" of the present disclosure.

[0093] The recognition process 96 is performed by the recognition unit 82A on the acquired frame 40 each time the frame 40 is acquired. The recognition process 96 is a process for recognizing the lesion 42 using an AI-based method. Here, the recognition process 96 is a process using a recognition model 92. The recognition model 92 is a trained model for object recognition using an AI-based bounding box method. Here, the recognition model 92 is an example of the "AI for object recognition" of the present disclosure.

[0094] The recognition model 92 is optimized by performing machine learning on the neural network using first training data. The first training data is a data set including a plurality of data (i.e., data for a plurality of frames) in which the first example data and the first answer data are associated with each other.

[0095] The first example data is an image assuming frame 40. An example of a first image assuming frame 40 is an image obtained by actually capturing an image of the inside of the large intestine with a camera. An example of a second image assuming frame 40 is a virtually created image. The first correct answer data is correct answer data (i.e., annotations) for the first example data. Here, annotations that identify the geometric characteristics, type, and model of the lesion depicted in the image used as the first example data are used as an example of the first correct answer data.

[0096] The recognition unit 82A acquires a frame 40 from the camera 52, and inputs the acquired frame 40 to the recognition model 92. As a result, each time a frame 40 is input, the recognition model 92 recognizes a lesion 42 appearing in the input frame 40, and outputs a recognition result 98.

[0097] The recognition result 98 includes lesion presence / absence information 97. The lesion presence / absence information 97 is information indicating whether or not a lesion 42 is captured in the frame 40 input to the recognition model 92. Furthermore, when a lesion 42 is captured in the frame 40 input to the recognition model 92, the recognition result 98 includes geometric property information 99, a lesion location map 100, lesion feature information 101, and the like.

[0098] The geometric property information 99 is information (e.g., coordinates) capable of identifying the shape and position of the lesion 42 within the frame 40. The lesion position map 100 is a map capable of identifying the position of the lesion 42 within the frame 40. The lesion feature information 101 is information capable of identifying the type and model of the lesion 42 appearing in the frame 40 that has been input to the recognition model 92.

[0099] The geometric characteristics of lesion location map 100 (e.g., shape and size of the outer contour) correspond to the geometric characteristics of frame 40 (e.g., shape and size of the outer contour). Lesion location map 100 includes a bounding box BB. The bounding box BB is a rectangular frame (e.g., a rectangular frame circumscribing an image area showing lesion 42) that can identify the position recognized by recognition model 92 as the position within frame 40 of lesion 42 captured in frame 40. Geometric characteristic information 99 is assigned to the geometric center of bounding box BB (e.g., the center of gravity of bounding box BB).

[0100] In the example shown in FIG. 5, the lesion location map 100 is displayed on the screen 35 (for example, the second display area 38) as the reference image 44A by the control unit 82B. In this case, the lesion location map 100 displayed on the screen 35 is updated according to the display frame rate applied to the first display area 36. That is, the display of the lesion location map 100 in the second display area 38 (i.e., the display of the bounding box BB) is updated in synchronization with the display timing of the endoscopic moving image 39 displayed in the first display area 36. With this configuration, the doctor 12 can grasp the general position of the lesion 42 in the endoscopic moving image 39 displayed in the first display area 36 by referring to the lesion location map 100 displayed in the second display area 38 while observing the endoscopic moving image 39 displayed in the first display area 36.

[0101] The control unit 82B may display the lesion characteristic information 101 on the screen 35 (for example, the second display area 38). The control unit 82B may also display a bounding box BB or an identifier (for example, a mark) in place of the bounding box BB in the first display area 36. In this case, for example, the control unit 82B may specify the shape and position of the lesion 42 within the frame 40 from the geometric property information 99, and display the bounding box BB or an identifier in place of the bounding box BB at the specified shape and position.

[0102] Fig. 6 is a conceptual diagram showing an example of the processing contents of the recognition unit 82A and the control unit 82B in the insertion step. As shown in Fig. 6, in the insertion step, the control unit 82B acquires the recognition result 98 from the recognition unit 82A, and judges whether or not the lesion 42 is captured in the frame 40 input to the recognition model 92 by referring to the acquired recognition result 98 (e.g., lesion presence / absence information 97).

[0103] Here, if a lesion 42 is present in the frame 40 input to the recognition model 92 to obtain the recognition result 98 obtained by the control unit 82B from the recognition unit 82A, the control unit 82B generates a frame 103 with an insertion tag.

[0104] The frame 103 with the insertion tag is a frame in which the insertion tag 102 is added to the insertion step frame 40A. In other words, the frame 103 with the insertion tag is a frame in which the insertion step frame 40A and the insertion tag 102 are associated with each other. The insertion step frame 40A is a frame 40 input to the recognition model 92 to obtain the recognition result 98 in the insertion step, and is a frame 40 in which a lesion 42 is captured. The insertion tag 102 is a tag that can identify the frame 40 obtained by capturing the image of the insertion step frame 40A by the camera 52 in the insertion step. The insertion tag 102 includes time information 102A. The time information 102A is information indicating the time when the insertion step frame 40A was captured in the insertion step (i.e., the time when the image was captured to obtain the insertion step frame 40A). In the first embodiment, the insertion step frame 40A is an example of a "first medical image" of the present disclosure.

[0105] The memory 84 is provided with a storage area 84A. Every time the control unit 82B generates a frame 103 with an insertion tag, the control unit 82B stores the generated frame 103 with an insertion tag in chronological order in the storage area 84A. The storage area 84A is an example of the "first storage area" and the "second storage area" of the present disclosure.

[0106] Fig. 7 is a conceptual diagram showing an example of the processing contents of the recognition unit 82A and the control unit 82B in the removal step. As shown in Fig. 7, at the timing when the endoscopic examination enters the removal step, the reception device 64 receives a removal start signal 104. The removal start signal 104 is a signal indicating that the removal step has started in the endoscopic examination. The control unit 82B recognizes that the removal step has started in the endoscopic examination on the condition that the removal start signal 104 has been received by the reception device 64. Then, in the removal step, the control unit 82B obtains the recognition result 98 from the recognition unit 82A, and judges whether or not the lesion 42 is included in the frame 40 input to the recognition model 92 by referring to the obtained recognition result 98 (e.g., lesion presence / absence information 97).

[0107] Here, if a lesion 42 is present in the frame 40 input to the recognition model 92 to obtain the recognition result 98 obtained by the control unit 82B from the recognition unit 82A, the control unit 82B generates a frame 106 with a removal tag.

[0108] The frame 106 with the removal tag is a frame in which the removal tag 108 is added to the removal step frame 40B. In other words, the frame 106 with the removal tag is a frame in which the removal step frame 40B and the removal tag 108 are associated with each other. The removal step frame 40B is a frame 40 obtained later in time than the insertion step frame 40A. For example, the removal step frame 40B is a frame 40 input to the recognition model 92 to obtain the recognition result 98 in the removal step, and is a frame 40 in which a lesion 42 is captured. The removal tag 108 is a tag that can identify the frame 40 obtained by capturing the image of the removal step frame 40B by the camera 52 in the removal step. The removal tag 108 includes time information 108A. The time information 108A is information indicating the time when the removal step frame 40B was obtained in the removal step (i.e., the time when the image was captured to obtain the removal step frame 40B). In the first embodiment, the removal step frame 40B is an example of a "second medical image" of the present disclosure.

[0109] Fig. 8 is a conceptual diagram showing an example of a process for deriving a similarity between the lesion 42 shown in the insertion step frame 40A and the lesion 42 shown in the removal step frame 40B. As shown in Fig. 8, the control unit 82B acquires the removal step frame 40B (i.e., the removal step frame 40B used to generate the removal tagged frame 106) and the insertion step frame 40A. Here, the insertion step frame 40A obtained by the control unit 82B is an unprocessed insertion step frame 40A included in the insertion tagged frame 103 stored in the storage area 84A. Here, the unprocessed insertion step frame 40A refers to an insertion step frame 40A that has not yet been input to the similarity derivation model 94 as a frame 40 that forms a pair with the removal step frame 40B shown in Fig. 8.

[0110] The removal step frame 40B has a plurality of partitioned regions 110. The plurality of partitioned regions 110 are obtained by partitioning the removal step frame 40B according to a partitioning rule for the removal step frame. Here, the partitioning rule for the removal step frame 40B refers to a rule for partitioning the removal step frame 40B. An example of a rule for partitioning the removal step frame 40B is a rule in which a mesh formed in units of blocks (e.g., vertical x horizontal = several pixels x several pixels) is applied to the removal step frame 40B, thereby partitioning the removal step frame 40B into a mesh shape. In the first embodiment, the partitioning rule for the removal step frame is an example of the "first rule" of the present disclosure. The partitioned region 110 is an example of the "second partitioned region" of the present disclosure.

[0111] The inserting step frame 40A has a plurality of partitioned regions 112. The plurality of partitioned regions 112 are obtained by partitioning the inserting step frame 40A according to an inserting step frame partitioning rule. Here, the inserting step frame partitioning rule refers to a rule for partitioning the inserting step frame 40A. One example of a rule for partitioning the inserting step frame 40A is a rule that partitions the inserting step frame 40A into a mesh shape by applying a mesh formed in units of blocks (e.g., vertical x horizontal = several pixels x several pixels) to the inserting step frame 40A. In this first embodiment, the inserting step frame partitioning rule is an example of the "first rule" of the present disclosure. Also, the partitioned region 112 is an example of the "first partitioned region" of the present disclosure.

[0112] In the first embodiment, the removal step frame division rule and the insertion step frame division rule are the same rule. Therefore, the shape, size, and number of the multiple partitioned regions 110 are the same as the shape, size, and number of the multiple partitioned regions 112. In addition, the removal step frame division rule and the insertion step frame division rule may be different rules.

[0113] The similarity derivation model 94 is an example of the "AI" of the present disclosure. The similarity derivation model 94 is optimized by performing machine learning using the second teacher data on the neural network. The second teacher data is a data set including a plurality of data (i.e., data for a plurality of frames) in which the second example data and the second answer data are associated with each other.

[0114] The second example data is a frame set of a frame assuming the insertion step frame 40A (hereinafter referred to as the "insertion step example frame") and a frame assuming the removal step frame 40B (hereinafter referred to as the "removal step example frame"). A first example of the insertion step example frame is an image obtained by actually capturing images of the inside of the large intestine with a camera, and divided according to the insertion step frame division rules. A second example of the insertion step example frame is an image created virtually and divided according to the insertion step frame division rules. A first example of the removal step example frame is an image obtained by actually capturing images of the inside of the large intestine with a camera, and divided according to the removal step frame division rules. A second example of the removal step example frame is an image created virtually and divided according to the removal step frame division rules.

[0115] The second correct answer data is correct answer data (i.e., annotation) for the second example data. The second correct answer data is associated with each of a plurality of partitioned areas (hereinafter referred to as "insertion process example partitioned areas") obtained by partitioning the insertion process example frame according to the insertion process frame partitioning rules, and each of a plurality of partitioned areas (hereinafter referred to as "removal process example partitioned areas") obtained by partitioning the removal process example frame according to the removal process frame partitioning rules. An example of the second correct answer data is data indicating whether or not a lesion is captured in the insertion process example partitioned area and the removal process example partitioned area, and indicating whether or not the lesion captured in the insertion process example partitioned area and the lesion captured in the removal process example partitioned area are the same when a lesion is captured in the insertion process example partitioned area and the removal process example partitioned area.

[0116] The control unit 82B inputs the insertion step frame 40A and the removal step frame 40B to the similarity derivation model 94. In this way, the similarity derivation model 94 derives a similarity, which is a degree of similarity between the lesion 42 captured in the partitioned region 110 and the lesion 42 captured in the partitioned region 112, when the lesion 42 is captured in the partitioned region 110 and the partitioned region 112 that correspond to each other. The similarity is derived for each partitioned region 110 for the removal step frame 40B, and is derived for each partitioned region 112 for the insertion step frame 40A. Then, every time a pair of the removal step frame 40B and the insertion step frame 40A is input, the similarity derivation model 94 outputs a first derivation result 114 corresponding to the removal step frame 40B and a second derivation result 116 corresponding to the insertion step frame 40A.

[0117] The first derivation result 114 includes a map 114A indicating the similarity for each of the multiple partitioned regions 110 and the distribution of the similarity. The map 114A is an example of the "map" of the present disclosure. In the first embodiment, a feature map indicating the distribution of similarity exceeding a threshold value with a shade according to the similarity value is used as the map 114A. Here, the similarity value is defined as 0% to 100%, and 85% is used as the threshold value in this case. In the example shown in FIG. 8, a high similarity distribution region 114A1 is shown as a region in which similarity values ​​exceeding the threshold value are distributed in the map 114A.

[0118] The second derivation result 116 includes a map 116A indicating the similarity for each of the multiple partitioned regions 112 and the distribution of the similarity. The map 116A is an example of the "map" of the present disclosure. In the first embodiment, a feature map indicating the distribution of similarity exceeding a threshold value with a shade according to the similarity value is used as the map 116A. Here, the similarity value is defined as 0 to 100%, and 85% is used as the threshold value in this case. In the example shown in FIG. 8, a high similarity distribution region 116A1 is shown as a region in which similarity values ​​exceeding the threshold value are distributed in the map 116A.

[0119] The control unit 82B performs the process shown in FIG. 8 until the high similarity distribution regions 114A1 and 116A1 are obtained, or until the derivation of the similarity between one removal step frame 40B shown in FIG. 8 and each of all the insertion step frames 40A stored in the storage area 84A (i.e., the derivation of the similarity using the similarity derivation model 94) is completed. The process shown in FIG. 8 refers to an inference using the similarity derivation model 94 for the pair of the insertion step frame 40A and the removal step frame 40B. The inference using the similarity derivation model 94 for the pair of the insertion step frame 40A and the removal step frame 40B refers to a process in which the pair of the insertion step frame 40A and the removal step frame 40B is input to the similarity derivation model 94, whereby the similarity is derived by the similarity derivation model 94, and the first derivation result 114 and the second derivation result 116 are output.

[0120] 9 is a conceptual diagram showing an example of processing contents by the control unit 82B using the removal step frame 40B and the map 114A when the high similarity distribution regions 114A1 and 116A1 are obtained, and processing contents by the control unit 82B using the insertion step frame 40A and the map 116A. As shown in FIG. 9, when the map 114A including the high similarity distribution region 114A1 and the map 116A including the high similarity distribution region 116A1 are obtained from the similarity derivation model 94 (see FIG. 8), the control unit 82B generates a removal step combined frame 40B1 based on the removal step frame 40B and the map 114A. Then, the control unit 82B executes a removal step combined frame display process. The removal step combined frame display process is an example of the "second output process" of the present disclosure. The removal step combined frame display process is a process of displaying the removal step combined frame 40B1 in the first display area 36. As a result of the removal step combined frame display process being performed, the removal step combined frame 40B1 is displayed in the first display area 36 in a live view manner.

[0121] The removal step combined frame 40B1 is a frame obtained by combining the removal step frame 40B and the first lesion identification image 118 corresponding to the high similarity distribution region 114A1. In the example shown in Fig. 9, the removal step combined frame 40B1 is a frame obtained by superimposing the first lesion identification image 118 on the removal step frame 40B.

[0122] The first lesion identification image 118 is superimposed on a region of the entire region of the removal step frame 40B at a position corresponding to the position of the high similarity distribution region 114A1 in the map 114A. Here, the high similarity distribution region 114A1 is used as it is as an example of the first lesion identification image 118. However, this is merely an example, and the high similarity distribution region 114A1 may be processed as an image. For example, the high similarity distribution region 114A1 may be an image (e.g., a mark) obtained by adjusting the color, darkness, and / or brightness of the high similarity distribution region 114A1. The transparency of the first lesion identification image 118 may be adjusted by alpha blending. An example of the transparency of the first lesion identification image 118 is a transparency that allows the doctor 12 to visually perceive the first lesion identification image 118 and does not visually interfere with observing the lesion 42 through the first display region 36 or performing a medical treatment on the lesion 42.

[0123] In this way, the removal step combined frame 40B1 obtained by superimposing the first lesion identifying image 118 on the removal step frame 40B is represented in a manner in which a first image area 40B1a, which is an image area in which the lesion 42 in the removal step frame 40B is captured, and a second image area 40B1b, which is an image area in which an object other than the lesion 42 in the removal step frame 40B is captured, can be distinguished by the first lesion identifying image 118. In the example shown in FIG. 9, in the removal step combined frame 40B1, the first image area 40B1a is emphasized more than the second image area 40B1b by the first lesion identifying image 118. Here, the image area in which the lesion 42 in the removal step frame 40B is captured, i.e., the first image area 40B1a which is a part of the removal step frame 40B, is determined in units of the divided area 110. In addition, an image area in the removal step frame 40B in which an area other than the lesion 42 is shown, that is, a second image area 40B1b which is a part of the removal step frame 40B, is also determined in units of the divided areas 110.

[0124] In the first embodiment, the first image region 40B1a is an example of a "second partial region" in the present disclosure, and the second image region 40B1b is an example of a "second other region" in the present disclosure.

[0125] When the map 114A including the high similarity distribution region 114A1 and the map 116A including the high similarity distribution region 116A1 are obtained from the similarity derivation model 94 (see FIG. 8), the control unit 82B generates the inserting step composite frame 40A1 based on the inserting step frame 40A and the map 116A. Then, the control unit 82B executes the inserting step composite frame display process. The inserting step composite frame display process is an example of the "first output process" of the present disclosure. The inserting step composite frame display process is a process of displaying the inserting step composite frame 40A1 in the second display area 38 as one of the reference images 44A.

[0126] The insertion step combined frame 40A1 is a frame obtained by combining the insertion step frame 40A and the second lesion identification image 120 corresponding to the high similarity distribution region 116A1. In the example shown in Fig. 9, the insertion step combined frame 40A1 is a frame obtained by superimposing the second lesion identification image 120 on the insertion step frame 40A.

[0127] The second lesion specific image 120 is superimposed on a region of the entire region of the insertion step frame 40A at a position corresponding to the position of the high similarity distribution region 116A1 in the map 116A. Here, the high similarity distribution region 116A1 is used as it is as an example of the second lesion specific image 120. However, this is merely an example, and the high similarity distribution region 116A1 may be processed as an image. For example, the high similarity distribution region 116A1 may be an image (e.g., a mark) obtained by adjusting the color, darkness, and / or brightness of the high similarity distribution region 116A1. The transparency of the second lesion specific image 120 may be adjusted by alpha blending. An example of the transparency of the second lesion specific image 120 is a transparency that allows the doctor 12 to visually perceive the second lesion specific image 120 and does not visually interfere with observing the lesion 42 through the second display region 38 or performing a medical treatment on the lesion 42.

[0128] In this way, the insertion step combined frame 40A1 obtained by superimposing the second lesion identifying image 120 on the insertion step frame 40A is represented in a manner in which a third image region 40A1a, which is an image region in which the lesion 42 in the insertion step frame 40A is captured, and a fourth image region 40A1b, which is an image region in which an object other than the lesion 42 in the insertion step frame 40A is captured, can be distinguished by the second lesion identifying image 120. In the example shown in Fig. 9, in the insertion step combined frame 40A1, the third image region 40A1a is emphasized more than the fourth image region 40A1b by the second lesion identifying image 120. Here, the image region in which the lesion 42 in the insertion step frame 40A is captured, i.e., the third image region 40A1a which is a part of the insertion step frame 40A, is determined in units of the divided regions 112. In addition, an image region in the insertion step frame 40A in which an object other than the lesion 42 is shown, that is, a fourth image region 40A1b which is a part of the insertion step frame 40A, is also defined in units of the divided regions 112.

[0129] In the first embodiment, the third image region 40A1a is an example of a "first partial region" in the present disclosure, and the fourth image region 40A1b is an example of a "first other region" in the present disclosure.

[0130] Fig. 10 is a conceptual diagram showing an example of processing performed by the control unit 82B using the frame 103 with an insertion tag and the frame 106 with a removal tag. As shown in Fig. 10, the control unit 82B generates a frame set 122 that associates the frame 103 with the insertion tag and the frame 106 with a removal tag, which includes a removal step frame 40B that shows the same lesion 42 as the lesion 42 shown in the insertion step frame 40A included in the frame 103 with an insertion tag. The control unit 82B stores the generated frame set 122 in the storage 86. Every time the control unit 82B generates a frame set 122, it stores the generated frame set 122 in the storage 86 in chronological order.

[0131] The insertion-tagged frame 103 used in the generation of the frame set 122 by the control unit 82B is the insertion-tagged frame 103 including the removal step frame 40B input to the similarity derivation model 94 to obtain the first derivation result 114 including the high similarity distribution region 114A1, among all the insertion-tagged frames 103 stored in the storage area 84A. The removal-tagged frame 106 used in the generation of the frame set 122 by the control unit 82B is the removal-tagged frame 106 including the removal step frame 40B input to the similarity derivation model 94 in a pair with the insertion step frame 40A depicting the same lesion 42 to obtain the second derivation result 116 including the high similarity distribution region 116A1.

[0132] In other words, frame set 122 is a pair of a removal tagged frame 106 including a removal process frame 40B used in the removal process composite frame 40B1 shown in Figure 9, and an insertion tagged frame 103 including an insertion process frame 40A used in the insertion process composite frame 40A1 shown in Figure 9.

[0133] 10 shows an example in which the frame set 122 is stored in the storage 86, but this is merely one example, and the frame set 122 may be stored in a storage area other than the storage 86. Examples of storage areas other than the storage 86 include an electronic medical record, a server, and / or a taglet terminal.

[0134] Next, the operation of the portion of the endoscope 10 according to the present disclosure will be described with reference to Fig. 11A and Fig. 11B. The flow of the medical support process shown in Fig. 11A and Fig. 11B is an example of the "medical support method" according to the present disclosure.

[0135] In the following, it is assumed that the same type of light (e.g., light for BLI and / or light for LCI) is irradiated into the large intestine 28 during the insertion process and the removal process, and the area irradiated with light (e.g., intestinal wall 32, etc.) is imaged by camera 52.

[0136] In the medical support process shown in Fig. 11A, first, in step ST10, the recognition unit 82A determines whether or not one frame of images has been captured by the camera 52 inside the large intestine 28. If one frame of images has not been captured by the camera 52 inside the large intestine 28 in step ST10, the determination is negative, and the medical support process proceeds to step ST38 shown in Fig. 11B. If one frame of images has been captured by the camera 52 inside the large intestine 28 in step ST10, the determination is positive, and the medical support process proceeds to step ST12.

[0137] In step ST12, the recognition unit 82A and the control unit 82B acquire the frame 40 obtained by imaging the large intestine 28 by the camera 52. Then, the control unit 82B displays the frame 40 in the first display area 36. After the processing of step ST12 is executed, the medical support processing proceeds to step ST14.

[0138] In step ST14, the recognition unit 82A executes recognition processing 96 on the frame 40 acquired in step ST12. After the processing in step ST14 is executed, the medical support processing proceeds to step ST16.

[0139] In step ST16, the control unit 82B determines whether or not a lesion 42 is captured in the frame 40 acquired in step ST12 (i.e., whether or not the lesion 42 has been recognized by the recognition unit 82A) based on the recognition result 98 obtained by performing the recognition process 96 in step ST14. If the lesion 42 is not captured in the frame 40 acquired in step ST12 in step ST16, the determination is negative, and the medical support process proceeds to step ST38 shown in FIG. 11B. If the lesion 42 is captured in the frame 40 acquired in step ST12 in step ST16, the determination is positive, and the medical support process proceeds to step ST18.

[0140] In step ST18, the control unit 82B determines whether the endoscopic examination is an insertion process. If the removal start signal 104 has not been received by the reception device 64, it is determined that the endoscopic examination is an insertion process, and if the removal start signal 104 has been received by the reception device 64, it is determined that the endoscopic examination is not an insertion process (i.e., the endoscopic examination is a removal process). In step ST18, if the endoscopic examination is not an insertion process, the determination is negative, and the medical support processing proceeds to step ST22. In step ST18, if the endoscopic examination is an insertion process, the determination is positive, and the medical support processing proceeds to step ST20.

[0141] In step ST20, the control unit 82B generates a frame 103 with an insertion tag based on the frame 40 acquired in step ST12. Then, the control unit 82B stores the generated frame 103 with an insertion tag in the storage area 84A. After the process of step ST20 is executed, the medical support process proceeds to step ST38 shown in FIG. 11B.

[0142] In step ST22, the control unit 82B generates the frame 106 with the removal tag based on the frame 40 acquired in step ST12. After the process of step ST22 is executed, the medical support process proceeds to step ST24.

[0143] In step ST24, the control unit 82B judges whether or not the frame 103 with the insertion tag is stored in the storage area 84A. If the frame 103 with the insertion tag is not stored in the storage area 84A in step ST24, the judgment is negative, and the medical support processing proceeds to step ST38 shown in Fig. 11B. If the frame 103 with the insertion tag is stored in the storage area 84A in step ST24, the judgment is positive, and the medical support processing proceeds to step ST26.

[0144] In step ST26, the control unit 82B acquires the frames 103 with insertion tags including the unprocessed insertion step frame 40A from among all the frames 103 with insertion tags stored in the storage area 84A. After the processing of step ST26 is executed, the medical support processing proceeds to step ST28.

[0145] In step ST28, the control unit 82B derives the similarity between each of the partitioned areas 112 of the insertion step frame 40A included in the insertion tagged frame 103 acquired in step ST26 and each partitioned area 110 of the removal step frame 40B included in the removal tagged frame 106 generated in step ST22, using the similarity derivation model 94 (see FIG. 8). The similarity derivation model 94 outputs a first derivation result 114 including the similarity and map 114A for each partitioned area 110, and a second derivation result 116 including the similarity and map 116A for each partitioned area 112 (see FIG. 8). After the process of step ST28 is executed, the medical support process proceeds to step ST30 shown in FIG. 11B.

[0146] 11B, the control unit 82B determines whether or not the similarity derived in step ST28 exceeds a threshold value. If the similarity derived in step ST28 does not exceed the threshold value in step ST30, the determination is negative, and the medical support process proceeds to step ST38. If the similarity derived in step ST28 exceeds the threshold value in step ST30, the determination is positive, and the medical support process proceeds to step ST32.

[0147] In step ST32, the control unit 82B generates a combined removal step frame 40B1 based on the map 114A obtained by executing the process of step ST28 and the removal step frame 40B included in the removal tagged frame 106 generated in step ST22 (see FIG. 9). Then, the control unit 82B displays the generated combined removal step frame 40B1 in the first display area 36 (see FIG. 9). After the process of step ST32 is executed, the medical support process proceeds to step ST34.

[0148] In step ST34, the control unit 82B generates an insertion step combined frame 40A1 based on the map 116A obtained by executing the process of step ST28 and the insertion step frame 40A used in step ST28 (see FIG. 9). Then, the control unit 82B displays the generated insertion step combined frame 40A1 in the second display area 38 (see FIG. 9). After the process of step ST34 is executed, the medical support process proceeds to step ST36.

[0149] In step ST36, the control unit 82B generates a frame set 122 based on the removal process combined frame 40B1 generated in step ST32 and the insertion process combined frame 40A1 generated in step ST34 (see FIG. 10). Then, the control unit 82B stores the generated frame set 122 in the storage 86 (see FIG. 10). After the processing of step ST36 is executed, the medical support processing proceeds to step ST38.

[0150] In step ST38, the control unit 82B determines whether or not a condition for terminating the medical support process is satisfied. One example of the condition for terminating the medical support process is that an instruction to terminate the medical support process is given to the endoscope 10 (for example, that an instruction to terminate the medical support process is accepted by the acceptance device 64).

[0151] In step ST38, if the condition for terminating the medical support process is not satisfied, the determination is negative, and the medical support process proceeds to step ST10. In step ST38, if the condition for terminating the medical support process is satisfied, the determination is positive, and the medical support process ends.

[0152] As described above, in the endoscope 10, when the similarity between the lesion 42 depicted in the insertion process frame 40A and the lesion 42 depicted in the removal process frame 40B exceeds a threshold value, the insertion process composite frame 40A1 is displayed in the second display area 38, and the removal process composite frame 40B1 is displayed in the first display area 36.

[0153] The insertion step composite frame 40A1 displayed in the second display region 38 is expressed in a manner that allows distinction between a third image region 40A1a depicting the lesion 42 and a fourth image region 40A1b depicting no lesion 42. For example, by superimposing the second lesion specific image 120 on the third image region 40A1a depicting the lesion 42, the third image region 40A1a is emphasized more than the fourth image region 40A1b.

[0154] Meanwhile, the removal process composite frame 40B1 displayed in the first display region 36 is expressed in a manner that allows distinction between a first image region 40B1a showing the lesion 42 and a second image region 40B1b showing no lesion 42. For example, by superimposing the first lesion identifying image 118 on the first image region 40B1a showing the lesion 42, the first image region 40B1a is emphasized more than the second image region 40B1b.

[0155] Here, the lesion 42 shown in the third image region 40A1a is a lesion observed by the doctor 12 in the insertion process, and the lesion 42 shown in the first image region 40B1a is a lesion observed by the doctor 12 in the removal process. The first lesion identifying image 118 superimposed on the first image region 40B1a and the second lesion identifying image 120 superimposed on the third image region 40A1a are images that are not displayed unless the similarity exceeds a threshold value.

[0156] Therefore, by displaying the second lesion identifying image 120 superimposed on the third image region 40A1a and displaying the first lesion identifying image 120 superimposed on the first image region 40B1a, the doctor 12 can visually identify that the lesion 42 shown in the first image region 40B1a and the lesion 42 shown in the third image region 40A1a are the same lesion. As a result, while the endoscope scope 16 is being inserted into the large intestine 28, the doctor 12 can observe the lesion 42 observed in the large intestine 28 by the doctor 12 through the endoscope scope 16 again through the endoscope scope 16 without omission. For example, the doctor 12 can observe the lesion 42 observed in the large intestine 28 by the doctor 12 through the endoscope scope 16 in the insertion process again through the endoscope scope 16 without omission.

[0157] In addition, in the endoscope 10, the first image region 40B1a is emphasized more than the second image region 40B1b, and the third image region 40A1a is emphasized more than the fourth image region 40A1b, making it easier for the doctor 12 to visually identify the position of the lesion 42 from the removal process composite frame 40B1 displayed in the first display region 36, and making it easier for the doctor 12 to visually identify the position of the lesion 42 from the insertion process composite frame 40A1 displayed in the second display region 38.

[0158] Furthermore, in the endoscope 10, the third image region 40A1a on which the second lesion identifying image 120 is superimposed is defined in units of the segmented regions 112, and the first image region 40B1a on which the first lesion identifying image 118 is superimposed is defined in units of the segmented regions 110. Therefore, the image region in which the similarity exceeds the threshold in the insertion step frame 40A can be specified in units of the segmented regions 112, and the image region in which the similarity exceeds the threshold in the removal step frame 40B can be specified in units of the segmented regions 110.

[0159] In addition, in the endoscope 10, a high similarity distribution region 114A1 in which the similarity exceeds a threshold is extracted from the plurality of partitioned regions 110 according to the map 114A, and a high similarity distribution region 116A1 in which the similarity exceeds a threshold is extracted from the plurality of partitioned regions 112 according to the map 116A. As a result, one or more partitioned regions 110 in which the similarity exceeds the threshold can be accurately extracted as the high similarity distribution region 114A1, and one or more partitioned regions 112 in which the similarity exceeds the threshold can be accurately extracted as the high similarity distribution region 116A1. The maps 114A and 116A are derived by an AI method by using the similarity derivation model 94. Therefore, the maps 114A and 116A can be derived with higher accuracy than when the maps 114A and 116A are derived only by relying on the intuition and experience of the doctor 12.

[0160] Furthermore, in the endoscope 10, the same type of light (e.g., light for BLI and / or light for LCI) is irradiated into the large intestine 28 during the insertion process and the removal process, and the area irradiated with the light (e.g., the intestinal wall 32, etc.) is imaged by the camera 52. This makes it possible to prevent the occurrence of a situation in which the similarity does not exceed the threshold value because the type of light used in the imaging to obtain the insertion process frame 40A is different from the type of light used in the imaging to obtain the removal process frame 40B.

[0161] Furthermore, in the endoscope 10, a frame set 122 is stored in the storage 86. In the frame set 122, an insertion step frame 40A and a removal step frame 40B, which show the same lesion 42, are associated with each other. Therefore, by checking the contents of the frame set 122 stored in the storage 86, the doctor 12 can determine whether or not all of the lesions 42 visually recognized during the insertion step have been visually recognized during the removal step.

[0162] In the above first embodiment, an example in which the removal step combined frame 40B1 is displayed in the first display area 36 and the insertion step combined frame 40A1 is displayed in the second display area 38 has been given, but this is merely one example. For example, as shown in Fig. 12, the removal step frame 40B may be displayed in the first display area 36 and the insertion step combined frame 40A1 may be displayed in the second display area 38 as in the above first embodiment. Also, the removal step combined frame 40B1 may be displayed in the first display area 36 and the insertion step frame 40A may be displayed in the second display area 38 as in the above first embodiment.

[0163] In the first embodiment, the second lesion specific image 120 is included in the insertion step composite frame 40A1, but this is merely an example. For example, as shown in FIG. 13, a frame 121 may be used instead of the second lesion specific image 120. The frame 121 is a frame that circumscribes the third image region 40A1a. In the example shown in FIG. 13, an elliptical frame is illustrated as an example of the frame 121, but this is merely an example, and any frame that can identify the third image region 40A1a (for example, a frame along the outer edge of the third image region 40A1a), that is, a frame that can distinguish the third image region 40A1a from the fourth image region 40A1b, may be used. The frame 121 may not be a continuous line, but may be an intermittent frame in which a part of the frame 121 is cut out. Also, as an example, as shown in FIG. 13, a numerical value 123 indicating a representative similarity may be displayed on the screen 35. A representative example of similarity is the maximum value of similarity of all the divided regions 112 included in the high similarity distribution region 116A1. The maximum value is merely an example, and statistical values ​​such as the average value, the minimum value, the median value, and / or the mode value may be displayed together with or instead of the maximum value. The display position of the numerical value 123 may be a position adjacent to the frame 121 in the second display region 38, or may be a position outside the second display region 38, and the numerical value 123 may be displayed in a manner associated with the frame 121 (for example, a manner using a speech bubble attached to the frame 121). Also, an identifier indicating the degree of similarity may be used instead of the numerical value 123.

[0164] In the example shown in FIG. 13, a frame 121 is used instead of the second lesion identification image 120. Similarly, a frame may be used instead of the first lesion identification image 118. In this case, the frame may be used so long as the first image region 40B1a and the second image region 40B1b can be distinguished from each other. Similarly to the numerical value 123 displayed in the second display region 38, a numerical value indicating a representative similarity may be displayed in the first display region 36. An example of the representative numerical value displayed in the first display region 36 is a statistical value (e.g., a maximum value, an average value, a minimum value, a median value, and / or a mode value) of the similarity of all the divided regions 110 included in the high similarity distribution region 114A1. The representative numerical value may be displayed outside the first display region 36. In this case, the representative numerical value may be displayed in a manner associated with the first lesion identification image 118 or the frame displayed in the second display region 38 (e.g., a manner using a speech bubble attached to the first lesion identification image 118 or the frame). Also, instead of a numerical value indicating a representative similarity, an identifier indicating the degree of similarity may be used.

[0165] In the above-mentioned first embodiment, an example of the form in which the similarity is derived on the condition that the removal tagged frame 106 is generated has been given, but the present disclosure is not limited thereto. For example, as shown in FIG. 14, the map 114A and / or 116A may be derived in response to a map derivation instruction 124 given from the outside (for example, the doctor 12). The map derivation instruction 124 is an instruction to the control unit 82B to derive the map 114A and / or 116A. In the example shown in FIG. 14, on the condition that the map derivation instruction 124 is accepted by the accepting device 64, the control unit 82B derives the maps 114A and 116A using the similarity derivation model 94. The map derivation instruction 124 is an example of a "first instruction" in the present disclosure.

[0166] In this way, by deriving maps 114A and / or 116A in response to map derivation instructions 124 given from the outside, doctor 12 can have control unit 82B derive maps 114A and / or 116A at the timing of his or her own intention.

[0167] Although the map derivation instruction 124 is exemplified here, a similarity derivation instruction may be used instead of the map derivation instruction 124. The similarity derivation instruction is an instruction to the control unit 82B to derive a similarity. For example, in this case, on condition that the similarity derivation instruction is accepted by the accepting device 64, the control unit 82B derives a similarity (for example, the first derivation result 114 and / or the second derivation result 116) using the similarity derivation model 94.

[0168] In the first embodiment, the high similarity distribution region 114A1 is defined in units of the partitioned region 110, and the high similarity distribution region 116A1 is defined in units of the partitioned region 112, but this is merely an example. For example, as shown in Figs. 15 and 16, the high similarity distribution regions 114A1 and 116A1 may be defined in units of pixels (here, as an example, in units of one pixel). In this case, the pixel is an example of the "first partitioned region" and the "second partitioned region" of the present disclosure. In this way, the high similarity distribution regions 114A1 and 116A1 are defined in units of pixels, and thus the first lesion specific image 118 and the second lesion specific image 120 are also defined in units of pixels (see Fig. 16). This allows the shape and size of the lesion 42 to be aligned with the shapes and sizes of the first lesion specific image 118 and the second lesion specific image 120.

[0169] Note that high similarity distribution region 114A1 or 116A1 may be defined in pixel units (here, as an example, in units of one pixel). By defining high similarity distribution region 114A1 or 116A1 in pixel units, first lesion identification image 118 or second lesion identification image 120 is also defined in pixel units (here, as an example, in units of one pixel).

[0170] In the first embodiment, the insertion step frame 40A is stored in the storage area 84A on the condition that the lesion 42 is recognized by the recognition unit 82A, but this is merely one example. For example, as shown in Fig. 17, the insertion step frame 40A (in the example shown in Fig. 17, the frame 103 with an insertion tag) may be stored in the storage area 84A in response to a frame storage instruction 126 given from the outside (for example, the doctor 12). The frame storage instruction 126 is an example of a "third instruction" in the present disclosure.

[0171] The frame storage instruction 126 is an instruction to store the insertion process frame 40A in the storage area 84A. In the example shown in Fig. 17, on condition that the frame storage instruction 126 is accepted by the acceptance device 64, the control unit 82B stores the insertion process frame 40A (the insertion tagged frame 103 in the example shown in Fig. 17) in the storage area 84A.

[0172] In this way, the insertion step frame 40A is stored in the storage area 84A in response to the frame storage instruction 126 given from the outside, so that the doctor 12 can store the insertion step frame 40A in the storage area 84A at the timing intended by the doctor 12. In addition, the number of the insertion step frames 40A stored in the storage area 84A can be reduced compared to the case where the insertion step frame 40A is stored in the storage area 84A every time a lesion 42 is recognized. As a result, the number of the insertion step frames 40A that are paired with the removal step frame 40B and input to the similarity derivation model 94 can be reduced, so that the overall load required for derivation of the similarity can be reduced. Furthermore, the number of the insertion step frames 40A stored in the storage area 84A can be reduced, so that the number of the frame sets 122 can also be reduced. As a result, the overall load required for generating the frame sets 122 can be reduced, and the pressure on the capacity of the storage 86 caused by the frame sets 122 being stored in the storage 86 can be suppressed.

[0173] 17, the frame 103 with the insertion tag is stored in the storage area 84A, but the frame 103 with the insertion tag does not need to be stored in the storage area 84A forever. For example, when the frame 103 with the insertion tag is used to derive a high similarity using the similarity derivation model 94 by the control unit 82B (for example, when the insertion step frame 40A included in the frame 103 with the insertion tag is input to the similarity derivation model 94), the frame 103 with the insertion tag including the insertion step frame 40A used to derive the high similarity may be deleted from the storage area 84A by the control unit 82B. The high similarity refers to a similarity exceeding a threshold value. In this case, for example, the insertion tagged frame 103 including the insertion step frame 40A used to generate the map 116A having the high similarity distribution region 116A1 (i.e., the insertion tagged frame 103 including the insertion step frame 40A showing the same lesion 42 as the lesion 42 shown in the removal step frame 40B) may be deleted from the storage area 84A by the control unit 82B. Also, for example, on the condition that the frame set 122 has been generated by the control unit 82B, the insertion tagged frame 103 used to generate the frame set 122 may be deleted from the storage area 84A by the control unit 82B.

[0174] In this way, the control unit 82B can determine whether or not the same lesion 42 as the lesion 42 found in the insertion step is visually recognized by the doctor 12 through the removal step composite frame 40B1 displayed on the screen 35 in the removal step by determining whether or not the insertion tagged frame 103 remains in the storage area 84A. For example, when the insertion tagged frame 103 remains in the storage area 84A, the control unit 82B determines that the same lesion 42 as the lesion 42 found in the insertion step is not visually recognized by the doctor 12 through the removal step composite frame 40B1 displayed on the screen 35 in the removal step. Also, for example, when all the insertion tagged frames 103 are deleted from the storage area 84A and no insertion tagged frames 103 remain in the storage area 84A, the control unit 82B determines that the same lesion 42 as the lesion 42 found in the insertion step is visually recognized by the doctor 12 through the removal step composite frame 40B1 displayed on the screen 35 in the removal step. These determination results may be displayed on screen 35, allowing physician 12 to easily determine whether or not the lesion 42 seen during the insertion step was also seen during the removal step.

[0175] In the above first embodiment, an example has been given in which the insertion tagged frame 103 is stored in the storage area 84A, but this is merely one example, and the insertion step frame 40A may also be stored in the storage area 84A.

[0176] In the first embodiment, the control unit 82B generates the frame 40B with the removal tag, but this is merely an example. For example, the control unit 82B may derive the similarity using the similarity derivation model 94 by using the removal process frame 40B in which the lesion 42 is shown, without generating the frame 40B with the removal tag.

[0177] In the above first embodiment, an example was given in which the frame set 122 is generated by associating the frame 103 with an insertion tag with the frame 106 with a removal tag, but this is merely one example, and for example, the insertion step frame 40A and the removal step frame 40B may be associated with each other. The frame set obtained by associating the insertion step frame 40A with the removal step frame 40B is output by the control unit 82B to a predetermined output destination (for example, the storage 86 and / or a server, etc.).

[0178] In the above first embodiment, an example was described in which the removal process composite frame 40B1 is displayed in the first display area 36 and the insertion process composite frame 40A1 is displayed in the second display area 38, but the removal process composite frame 40B1 and the insertion process composite frame 40A1 may also be displayed in a single display area.

[0179] In the first embodiment, the display device 18 is used as an output destination of the insertion step combined frame 40A1 and the removal step combined frame 40B1, but this is merely an example. The output destination of the insertion step combined frame 40A1 and / or the removal step combined frame 40B1 may be a storage area such as the storage 86, or a printer.

[0180] [Second embodiment] In the above first embodiment, an example in which the removal step frame 40B is divided according to the removal step frame division rule and the insertion step frame 40A is divided according to the insertion step frame division rule is described. In the present second embodiment, an example in which the area indicating the lesion 42 in the removal step frame 40B is divided and the area indicating the lesion 42 in the insertion step frame 40A is divided is described. In the present second embodiment, the same components as those in the above first embodiment are denoted by the same reference numerals and description thereof is omitted. In addition, the above description regarding Figures 12 to 17 (for example, the examples shown in Figures 12 to 17 and modified examples of the examples shown in Figures 12 to 17) can also be applied to the present second embodiment.

[0181] Fig. 18 is a conceptual diagram showing an example of the processing contents of the recognition unit 82A and the control unit 82B in the insertion step. As shown in Fig. 18, the insertion tag 102 included in the insertion tag-attached frame 103 stored in the storage area 84A by the control unit 82B includes a recognition result 98. The recognition result 98 included in the insertion tag 102 is the recognition result 98 output from the recognition model 92 when the insertion step frame 40A to which the insertion tag 102 corresponds is input to the recognition model 92.

[0182] FIG. 19 is a conceptual diagram showing an example of a process for deriving a similarity between the lesion 42 included in the insertion step frame 40A and the lesion 42 included in the removal step frame 40B. As shown in FIG. 19, the control unit 82B derives a similarity, which is a degree of similarity between the lesion 42 shown in the removal step frame 40B and the lesion 42 shown in the insertion step frame 40A, in response to a similarity derivation instruction 129 given from the outside (for example, the doctor 12). The similarity derivation instruction 129 is an instruction for causing the control unit 82B to derive a similarity. In the example shown in FIG. 19, the control unit 82B derives a similarity using a similarity derivation model 128 on the condition that the similarity derivation instruction 129 is accepted by the acceptance device 64. The similarity derivation instruction 129 is an example of a "second instruction" in the present disclosure.

[0183] Controller 82B refers to recognition result 98 corresponding to removal step frame 40B (i.e., recognition result 98 obtained by performing recognition process 96 on removal step frame 40B) and extracts removal step lesion area 42B, which is an image area indicating lesion 42 captured in removal step frame 40B, from removal step frame 40B, as a prerequisite for deriving similarity using similarity derivation model 128. Removal step lesion area 42B is a feature area recognized by performing recognition process 96 as an area having a predetermined feature.

[0184] Furthermore, the control unit 82B refers to the recognition result 98 corresponding to the unprocessed insertion step frame 40A (i.e., the recognition result 98 in the insertion tag 102 associated with the insertion step frame 40A included in the insertion tagged frame 103 stored in the storage area 84A) as a prerequisite for deriving the similarity using the similarity derivation model 128, and extracts the insertion step lesion area 42A, which is an image area showing the lesion 42 shown in the unprocessed insertion step frame 40A, from the insertion step frame 40A. The insertion step lesion area 42A is a feature area recognized by performing the recognition process 96 as an area having a predetermined feature.

[0185] The removal step lesion area 42B has a plurality of partitioned areas 130. The plurality of partitioned areas 130 are obtained by partitioning the removal step frame 40B according to a removal step lesion partitioning rule. Here, the removal step lesion partitioning rule refers to a rule for partitioning the removal step lesion area 42B. An example of a rule for partitioning the removal step lesion area 42B is a rule in which a mesh formed in units of blocks (e.g., vertical x horizontal = several pixels x several pixels) is applied to the removal step lesion area 42B, thereby partitioning the removal step lesion area 42B into a mesh shape. In the second embodiment, the removal step lesion area 42B is an example of the "second area," "characteristic area," and "area in which a lesion is photographed" of the present disclosure. The removal step lesion partitioning rule is an example of the "second rule" of the present disclosure. The partitioned area 130 is an example of the "fourth partitioning area" of the present disclosure.

[0186] The insertion step lesion area 42A has a plurality of divided areas 132. The plurality of divided areas 132 are obtained by dividing the insertion step lesion area 42A according to an insertion step lesion division rule. Here, the insertion step lesion division rule refers to a rule for dividing the insertion step lesion area 42A. An example of a rule for dividing the insertion step lesion area 42A is a rule in which a mesh formed in units of blocks (for example, vertical x horizontal = several pixels x several pixels) is applied to the insertion step lesion area 42A, thereby dividing the insertion step lesion area 42A into a mesh shape. In the second embodiment, the insertion step lesion area 42A is an example of the "first area", "characteristic area", and "area where a lesion is photographed" of the present disclosure. The insertion step lesion division rule is an example of the "second rule" of the present disclosure. The divided area 132 is an example of the "third divided area" of the present disclosure.

[0187] In the second embodiment, the lesion classification rule for the removal process and the lesion classification rule for the insertion process are the same rule. Therefore, the shape, size, and number of the multiple partitioned regions 130 are the same as the shape, size, and number of the multiple partitioned regions 132.

[0188] In the second embodiment, the control unit 82B uses a similarity derivation model 128. The similarity derivation model 128 is an example of the "AI" of the present disclosure. The similarity derivation model 128 is optimized by performing machine learning using third teacher data on a neural network. The third teacher data is a data set including a plurality of data (i.e., data for a plurality of frames) in which the third example data and the third answer data are associated with each other.

[0189] The third example data is an image set of an image assuming an insertion process lesion area 42A (hereinafter referred to as an "insertion process lesion example image") and an image assuming a removal process lesion area 42B (hereinafter referred to as an "removal process lesion example image"). A first example of an insertion process lesion example image is an image in which a lesion shown in an image obtained by actually capturing an image of the inside of the large intestine with a camera is classified according to the insertion process lesion classification rules. A second example of an insertion process lesion example image is an image that is virtually created and classified according to the insertion process lesion classification rules. A first example of a removal process lesion example image is an image in which an image obtained by actually capturing an image of the inside of the large intestine with a camera is classified according to the removal process lesion classification rules. A second example of a removal process lesion example image is an image that is virtually created and classified according to the removal process lesion classification rules.

[0190] The third correct answer data is correct answer data (i.e., annotation) for the third example data. The third correct answer data corresponds to each of a plurality of divided areas (hereinafter referred to as "insertion process lesion example divided areas") obtained by dividing the insertion process lesion example image according to the insertion process lesion dividing rule, and each of a plurality of divided areas (hereinafter referred to as "removal process lesion example divided areas") obtained by dividing the removal process lesion example image according to the removal process lesion dividing rule. An example of the third correct answer data is data indicating whether a lesion shown in the insertion process lesion example divided area is the same as a lesion shown in the removal process lesion example divided area.

[0191] The control unit 82B inputs the insertion process lesion region 42A and the removal process lesion region 42B to the similarity derivation model 128. As a result, the similarity derivation model 128 derives a similarity, which is a degree of similarity between the partitioned region 130 and the partitioned region 132. In the second embodiment, the similarity is derived for each partitioned region 130 for the removal process lesion region 42B, and is derived for each partitioned region 132 for the insertion process lesion region 42A. Then, every time a pair of the removal process lesion region 42B and the insertion process lesion region 42A is input, the similarity derivation model 128 outputs a third derivation result 134 corresponding to the removal process lesion region 42B and a fourth derivation result 136 corresponding to the insertion process lesion region 42A.

[0192] The third derivation result 134 includes a map 134A showing the similarity for each of the multiple partitioned regions 130 and the distribution of the similarity. In the second embodiment, a feature map showing the distribution of similarity exceeding a threshold value with a shade according to the similarity is used as the map 134A. Here, the similarity is defined as 0% to 100%, and 85% is used as the threshold value in this case. In the example shown in FIG. 19, a high similarity distribution region 134A1 is shown as a region in which similarity exceeding the threshold value is distributed in the map 134A.

[0193] The fourth derivation result 136 includes a map 136A showing the similarity for each of the multiple partitioned regions 132 and the distribution of the similarity. In the second embodiment, a feature map showing the distribution of similarity exceeding a threshold value with a shade according to the similarity is used as the map 136A. Here, the similarity is defined as 0% to 100%, and 85% is used as the threshold value in this case. In the example shown in FIG. 19, a high similarity distribution region 136A1 is shown as a region in which similarity exceeding the threshold value is distributed in the map 136A.

[0194] The control unit 82B performs the process shown in FIG. 19 until the high similarity distribution regions 134A1 and 136A1 are obtained, or until the derivation of the similarity between one removal step frame 40B shown in FIG. 19 and each of all insertion step frames 40A stored in the storage area 84A (i.e., the derivation of the similarity using the similarity derivation model 128) is completed. The process shown in FIG. 19 refers to an inference using the similarity derivation model 128 for the pair of the removal step lesion region 42B and the insertion step lesion region 42A. The inference using the similarity derivation model 128 for the pair of the removal step lesion region 42B and the insertion step lesion region 42A refers to a process in which the pair of the removal step lesion region 42B and the insertion step lesion region 42A is input to the similarity derivation model 128, the similarity is derived by the similarity derivation model 128, and the third derivation result 134 and the fourth derivation result 136 are output.

[0195] FIG. 20 is a conceptual diagram showing an example of processing contents by the controller 82B using the removal step lesion region 42B and the map 134A when the high similarity distribution regions 134A1 and 136A1 are obtained, and processing contents by the controller 82B using the insertion step lesion region 42A and the map 136A. As shown in FIG. 20, when the map 134A including the high similarity distribution region 134A1 and the map 136A including the high similarity distribution region 136A1 are obtained from the similarity derivation model 128 (see FIG. 19), the controller 82B generates a removal step combined frame 40B2 based on the removal step frame 40B and the map 134A. Then, the controller 82B executes a removal step combined frame display process according to the second embodiment. The removal step combined frame display process according to the second embodiment is an example of the "second output process" of the present disclosure. The removal step combined frame display process according to the second embodiment is a process of displaying the removal step combined frame 40B2 in the first display region 36. As a result of the removal step combined frame display process being performed, the removal step combined frame 40B2 is displayed in the first display area 36 in a live view manner.

[0196] The removal step combined frame 40B2 is a frame obtained by combining the removal step frame 40B and the third lesion identification image 138 corresponding to the high similarity distribution region 134A1. In the example shown in Fig. 20, the removal step combined frame 40B2 is a frame obtained by superimposing the third lesion identification image 138 on the removal step frame 40B.

[0197] The third lesion specific image 138 is superimposed on an area of ​​the entire area of ​​the removal step frame 40B at a position corresponding to the position of the high similarity distribution area 134A1 in the map 134A. Here, the high similarity distribution area 134A1 is used as it is as an example of the third lesion specific image 138. However, this is merely an example, and the high similarity distribution area 134A1 may be processed as an image. For example, the high similarity distribution area 134A1 may be an image (e.g., a mark) obtained by adjusting the color, darkness, and / or brightness of the high similarity distribution area 134A1. The transparency of the third lesion specific image 138 may be adjusted by alpha blending. An example of the transparency of the third lesion specific image 138 is a transparency that allows the doctor 12 to visually perceive the third lesion specific image 138 and does not visually interfere with observing the lesion 42 through the first display area 36 or performing a medical treatment on the lesion 42.

[0198] In this way, the removal step combined frame 40B2 obtained by superimposing the third lesion identifying image 138 on the removal step frame 40B is represented in a manner in which a fifth image region 40B2a, which is an image region in which the lesion 42 in the removal step frame 40B is captured, and a sixth image region 40B2b, which is an image region in which an object other than the lesion 42 in the removal step frame 40B is captured, can be distinguished by the third lesion identifying image 138. In the example shown in FIG. 20, in the removal step combined frame 40B2, the fifth image region 40B2a is emphasized more than the sixth image region 40B2b by the third lesion identifying image 138. Here, the image region in which the lesion 42 in the removal step frame 40B is captured, i.e., the fifth image region 40B2a which is a part of the removal step frame 40B, is determined in units of the divided regions 130. In addition, an image area in the removal step frame 40B in which an area other than the lesion 42 is shown, that is, a sixth image area 40B2b which is a part of the removal step frame 40B, is also determined in units of the divided area 130.

[0199] In the second embodiment, the fifth image region 40B2a is an example of a "second partial region" in the present disclosure, and the sixth image region 40B2b is an example of a "second other region" in the present disclosure.

[0200] When the map 134A including the high similarity distribution region 134A1 and the map 136A including the high similarity distribution region 136A1 are obtained from the similarity derivation model 128 (see FIG. 20), the control unit 82B generates the inserting step composite frame 40A2 based on the inserting step frame 40A and the map 136A. Then, the control unit 82B executes the inserting step composite frame display process according to the second embodiment. The process composite frame display process according to the second embodiment is an example of the "first output process" of the present disclosure. The inserting step composite frame display process according to the second embodiment is a process of displaying the inserting step composite frame 40A2 in the second display area 38 as one of the reference images 44A.

[0201] The insertion step combined frame 40A2 is a frame obtained by combining the insertion step frame 40A and the fourth lesion identification image 140 corresponding to the high similarity distribution region 136A1. In the example shown in Fig. 20, the insertion step combined frame 40A2 is a frame obtained by superimposing the fourth lesion identification image 140 on the insertion step frame 40A.

[0202] The fourth lesion specific image 140 is superimposed on a region of the entire region of the insertion step frame 40A at a position corresponding to the position of the high similarity distribution region 136A1 in the map 136A. Here, the high similarity distribution region 136A1 is used as it is as an example of the fourth lesion specific image 140. However, this is merely an example, and the high similarity distribution region 136A1 may be processed as an image. For example, the high similarity distribution region 136A1 may be an image (e.g., a mark) obtained by adjusting the color, darkness, and / or brightness of the high similarity distribution region 136A1. The transparency of the fourth lesion specific image 140 may be adjusted by alpha blending. An example of the transparency of the fourth lesion specific image 140 is a transparency that allows the doctor 12 to visually perceive the fourth lesion specific image 140 and does not visually interfere with observing the lesion 42 through the second display region 38 or performing a medical treatment on the lesion 42.

[0203] In this way, the insertion step combined frame 40A2 obtained by superimposing the fourth lesion identifying image 140 on the insertion step frame 40A is represented in a manner in which a seventh image region 40A2a, which is an image region in which the lesion 42 in the insertion step frame 40A is captured, and an eighth image region 40A2b, which is an image region in which an object other than the lesion 42 in the insertion step frame 40A is captured, can be distinguished by the fourth lesion identifying image 140. In the example shown in FIG. 20, in the insertion step combined frame 40A1, the seventh image region 40A2a is emphasized more than the eighth image region 40A2b by the fourth lesion identifying image 118. Here, the image region in which the lesion 42 in the insertion step frame 40A is captured, i.e., the seventh image region 40A2a which is a part of the insertion step frame 40A, is determined in units of the divided regions 132. In addition, an image region in the insertion step frame 40A in which an object other than the lesion 42 is shown, that is, an eighth image region 40A2b which is a part of the insertion step frame 40A, is also defined in units of the divided regions 132.

[0204] In the second embodiment, the seventh image region 40A2a is an example of a "first partial region" in the present disclosure, and the eighth image region 40A2b is an example of a "first other region" in the present disclosure.

[0205] As described above, in the endoscope 10 according to the second embodiment, the fifth image region 40B2a on which the third lesion identifying image 138 is superimposed is defined in units of the divided regions 130, and the seventh image region 40A2a on which the fourth lesion identifying image 140 is superimposed is defined in units of the divided regions 132. Therefore, compared to a case in which the entire removal step frame 40B and the entire insertion step frame 40A are divided, the fifth image region 40B2a and the seventh image region 40A2a whose similarity is equal to or greater than the threshold value can be quickly identified.

[0206] In the endoscope 10 according to the second embodiment, a high similarity distribution region 134A1 in which the similarity exceeds a threshold is extracted from a plurality of partitioned regions 130 according to the map 134A, and a high similarity distribution region 136A1 in which the similarity exceeds a threshold is extracted from a plurality of partitioned regions 132 according to the map 136A. This allows one or more partitioned regions 130 in which the similarity exceeds the threshold to be accurately extracted as the high similarity distribution region 134A1, and allows one or more partitioned regions 132 in which the similarity exceeds the threshold to be accurately extracted as the high similarity distribution region 136A1. In the second embodiment, the similarity is derived by an AI method by using the similarity derivation model 128. Therefore, the similarity can be derived more accurately than when the similarity is derived only by relying on the intuition and experience of the doctor 12.

[0207] Moreover, in the endoscope 10 according to the second embodiment, the similarity is derived in response to an externally applied similarity derivation instruction 129. This allows the doctor 12 to cause the control unit 82B to derive the similarity at a timing intended by the doctor 12.

[0208] In the endoscope 10 according to the second embodiment, the insertion step lesion area 42A is extracted from the insertion step frame 40A, and the removal step lesion area 42B is extracted from the removal step frame 40B. Both the insertion step lesion area 42A and the removal step lesion area 42B are characteristic areas recognized by performing a recognition process 96 as areas having predetermined characteristics. The characteristic areas are areas in which the doctor 12 is likely to be interested. Therefore, while the endoscope scope 16 is inserted into the large intestine 28, it is possible to contribute to allowing the doctor 12 to observe, without omission, the areas observed in the large intestine 28 through the endoscope scope 16 by the doctor 12 in the insertion step again through the endoscope scope 16 in the removal step, in areas in which the doctor 12 is likely to be interested (here, the image area in which the lesion 42 is shown).

[0209] In the second embodiment, the insertion step lesion area 42A and the removal step lesion area 42B are exemplified as characteristic areas, but the present disclosure is not limited thereto. For example, the characteristic area may be a marked area, an area showing an organ (e.g., duodenal papilla, etc.), and / or an area showing a treatment tool 58.

[0210] Furthermore, in the endoscope 10 according to the second embodiment, a process using the recognition model 92 is performed as the recognition process 96, whereby the insertion process lesion region 42A and the removal process lesion region 42B are recognized. Therefore, compared to a case where the insertion process lesion region 42A and the removal process lesion region 42B are identified solely by relying on the intuition and experience of the doctor 12, the insertion process lesion region 42A and the removal process lesion region 42B can be identified with higher accuracy.

[0211] In the endoscope 10 according to the second embodiment, on the condition that the insertion step lesion area 42A is recognized by performing the recognition process 96 in the insertion step, the insertion step frame 40A used in the recognition process 96 is stored in the storage area 84A. This makes it possible to easily obtain the insertion step frame 40A used together with the removal step frame 40B to derive the similarity.

[0212] [Other embodiments] In the above-mentioned embodiments, the control unit 82B derives the similarity, but the present disclosure is not limited thereto. For example, the control unit 82B may derive the dissimilarity. Here, the dissimilarity refers to the degree of difference between the lesion 42 captured in the insertion step frame 40A and the lesion 42 captured in the removal step frame 40B. In the case where the dissimilarity is derived by the control unit 82B in this way, the dissimilarity may be derived by the AI ​​method in the same manner as in the above-mentioned embodiments, and a map showing the distribution of the dissimilarity may be derived. In the above-mentioned embodiments, the various maps showing the distribution of the similarity exceeding a threshold (for example, maps 114A, 116A, 134A, and 136A) are derived. In the case where the dissimilarity is derived, for example, instead of the various maps described in the above-mentioned embodiments, various maps showing the distribution of the dissimilarity less than a threshold (for example, 15%) different from the threshold described in the above-mentioned embodiments may be derived. In this way, even if the dissimilarity is derived instead of the similarity, the same effect as in the above-mentioned embodiments can be obtained.

[0213] In the above embodiments, in order to facilitate understanding of the present disclosure, an example in which one lesion 42 is captured in the frame 40 has been described, but this is merely one example. A case in which multiple lesions 42 are captured in the frame 40 is also conceivable.

[0214] To explain this by applying it to the above-mentioned first embodiment, the removal step frame 40B includes a plurality of first image regions 40B1a, and the insertion step frame 40A includes a plurality of third image regions 40A1a, so that the first lesion identifying image 118 is superimposed on each of the plurality of first image regions 40B1a, and the second lesion identifying image 120 is superimposed on each of the plurality of third image regions 40A1a. Here, the first lesion identifying image 118 and the second lesion identifying image 120 are displayed in a display manner that allows visual identification of which first image region 40B1a included in the removal step frame 40B is the same as which third image region 40A1a included in the insertion step frame 40A.

[0215] For example, the first lesion identifying image 118 and the second lesion identifying image 120 applied to the first image region 40B1a and the third image region 40A1a in which the same lesion 42 appears are displayed in the same color. In a similar manner, when the frame 121 shown in Fig. 13 is used, the frame 121 may be displayed in a display manner that allows visual identification of which first image region 40B1a included in the removal step frame 40B and which third image region 40A1a included in the insertion step frame 40A are the same as each other.

[0216] In addition, the first image region 40B1a and the third image region 40A1a showing the same lesion 42 may be assigned the same identifier (e.g., ID), and the identifier assigned to the first image region 40B1a and the third image region 40A1a may be displayed in a speech bubble format.

[0217] Note that, here, the above-mentioned first embodiment has been taken as an example to explain the case where multiple lesions 42 are captured in the frame 40. However, even if multiple lesions 42 are captured in the frame 40 in the above-mentioned second embodiment, the display mode can be changed for each lesion 42 in a similar manner, or an identifier defined for each lesion 42 can be displayed.

[0218] In the above-mentioned embodiments, the similarity between the lesion 42 shown in the insertion step frame 40A obtained in the insertion step and the lesion 42 shown in the removal step frame 40B obtained in the removal step is derived, but the present disclosure is not limited thereto. For example, the similarity between the lesion 42 shown in each temporally adjacent frame 40 obtained in the insertion step or the removal step may be derived. The processing using the similarity thus obtained may be performed in the same manner as in the above-mentioned embodiments. As a result, for example, when the doctor 12 finds the lesion 42 (i.e., when the lesion 42 is shown again in the first display area 36) after losing sight of the lesion 42 being observed due to the shaking of the endoscope 16 and / or a positional shift caused by body movement or the like (i.e., after the lesion 42 being observed has moved out of the first display area 36), the doctor 12 can visually identify whether the lesion 42 he found is the same as the lesion 42 he was observing before losing sight of it.

[0219] In the above embodiments, the recognition process 96 using AI in a bounding box method is exemplified, but the present disclosure is not limited to this. For example, recognition process using AI in a segmentation method (e.g., semantic segmentation, instance segmentation, and / or panoptic segmentation) may be performed.

[0220] In this case, the recognition model 92 is a trained model for object recognition by an AI segmentation method. An example of a trained model for object recognition by an AI segmentation method is a model for semantic segmentation. An example of a model for semantic segmentation is a model with an encoder-decoder structure. An example of a model with an encoder-decoder structure is U-Net or HRNet.

[0221] In each of the above embodiments, an example in which the medical support process is performed by the computer 78 has been described, but the present disclosure is not limited to this, and at least a part of the processes included in the medical support process may be performed by a device provided outside the computer 78. An example of this case will be described below with reference to FIG.

[0222] 21 is a conceptual diagram showing an example of the configuration of an endoscope 142. The endoscope 142 is an example of the "endoscope" according to the technology of the present disclosure. The endoscope 142 differs from the endoscope 10 described in each of the above embodiments in that it has an external device 144.

[0223] External device 144 is communicatively coupled to computer 78 via a network 146 (eg, a WAN and / or LAN, etc.).

[0224] An example of the external device 144 is at least one server that directly or indirectly transmits and receives data to and from the computer 78 via the network 146. The external device 144 receives a processing execution instruction provided from the processor 82 of the computer 78 via the network 146. The external device 144 then executes processing according to the received processing execution instruction and transmits the processing result to the computer 78 via the network 146. In the computer 78, the processor 82 receives the processing result transmitted from the external device 144 via the network 146 and executes processing using the received processing result.

[0225] An example of the processing execution instruction is an instruction to have the external device 144 execute at least a part of the medical support processing. A first example of at least a part of the medical support processing (i.e., a processing to be executed by the external device 144) is the recognition processing 96. In this case, the external device 144 executes the recognition processing 96 in accordance with the processing execution instruction provided from the processor 82 via the network 146, and transmits the recognition result 98 to the computer 78 via the network 146. In the computer 78, the processor 82 receives the recognition result 98, and executes the same processing as in each of the above-described embodiments using the received recognition result 98.

[0226] A second example of at least a part of the medical support process (i.e., the process to be executed by the external device 144) is a process by the control unit 82B. In this case, the external device 144 executes the process by the control unit 82B according to a process execution instruction given from the processor 82 via the network 146, and transmits the process result (e.g., the insertion step combined frame 40A1, the insertion step combined frame 40A2, the removal step combined frame 40B1, and / or the removal step combined frame 40B2, etc.) to the computer 78 via the network 146. In the computer 78, the processor 82 receives the process result (e.g., the insertion step combined frame 40A1, the insertion step combined frame 40A2, the removal step combined frame 40B1, and / or the removal step combined frame 40B2, etc.), and executes the same process as in each of the above-mentioned embodiments using the received process result.

[0227] For example, the external device 144 is realized by cloud computing. Note that cloud computing is merely an example, and the external device 144 may be realized by network computing such as fog computing, edge computing, or grid computing. Instead of a server, at least one personal computer or the like may be used as the external device 144. Also, the external device 144 may be a computing device with a communication function equipped with multiple types of AI functions.

[0228] In each of the above embodiments, the medical support program 90 is stored in the storage 86, but the present disclosure is not limited to this. For example, the medical support program 90 may be stored in a portable computer-readable non-transitory storage medium such as an SSD or a USB memory. The medical support program 90 stored in the non-transitory storage medium is installed in the computer 78 of the endoscope 10. The processor 82 executes medical support processing in accordance with the medical support program 90.

[0229] In addition, the medical support program 90 may be stored in a storage device such as another computer or server connected to the endoscope 10 via a network, and the medical support program 90 may be downloaded and installed in the computer 78 upon request from the endoscope 10.

[0230] It is not necessary to store the entire medical support program 90 in a storage device such as another computer or server device connected to the endoscope 10, or to store the entire medical support program 90 in the storage 86; only a portion of the medical support program 90 may be stored.

[0231] The hardware resources for executing the medical support processing can be various processors as shown below. An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the medical support processing by executing software, i.e., a program. Another example of a processor is a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing a specific process, such as an FPGA, a PLD, or an ASIC. Each processor has a built-in or connected memory, and each processor executes the medical support processing by using the memory.

[0232] The hardware resource for executing the medical support process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource for executing the medical support process may be a single processor.

[0233] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes medical support processing. Second, there is a configuration using a processor that realizes the functions of the entire system including multiple hardware resources that execute medical support processing on a single IC chip, as typified by SoC. In this way, the medical support processing is realized using one or more of the above various processors as hardware resources.

[0234] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. Also, the above medical support process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processing may be changed, without departing from the spirit of the invention.

[0235] The above description and illustrations are detailed descriptions of the parts related to the present disclosure and are merely an example of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations within the scope of the gist of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the present disclosure.

[0236] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, and standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0237] 10,142 Endoscopes 12. Doctor 14. Nurse 16 Endoscope 18 Display device 20 Light source device 22 Control device 24 Medical Support Equipment 26 Subject 28 Large intestine 28A Ileocecal area 29A Insertion Path 29B Removal route 30 light 32 Intestinal wall 34 Wagon 35 screens 36 1st display area 38 Second display area 39 Endoscopic video images 40 frames 40A Insertion Process Frame 40A1 Insertion Process Composite Frame 40A1a Third image area 40A1b Fourth image area 40A2a 7th image area 40A2b 8th image area 40B Removal process frame 40B1 Removal process composite frame 40B1a 1st image area 40B1b 2nd image area 40B2a 5th image area 40B2b 6th image area 42 Lesions 42A Insertion Step Lesion Area 42B Extraction process lesion area 44 Medical Information 44A Reference Image 46 Control section 48 Insertion section 50 Tip 50A Tip surface 52 Camera 54 Lighting Equipment 54A, 54B Lighting window 56 Treatment opening 58 Treatment tools 60 Treatment tool insertion port 62 Universal Code 64 Reception device 66,78 Computer 68,88 Bus 70,80 External I / F 72,82 processor 74,84 Memory 76,86 Storage 82A recognition part 82B Control section 84A Storage Area 90 Medical Assistance Program 92 Recognition Model 94,128 Similarity Derivation Model 96 Recognition Processing 97 Lesion Presence Information 98 Recognition results 99 Geometric property information 100 Lesion Location Map 101 Lesion characteristic information 102 Insertion Tag 102A,108A Time information 103 Insert tagged frame 104 Removal start signal 106 Frame with removal tag 108 Removal tag 110,112,130,132 segmented area 114 First derivation result 114A, 116A, 134A, 136A Map 114A1,116A1,134A1,136A1 High similarity distribution area 116 Second derivation result 118 First Lesion Identification Image 120 Second Lesion Identification Image 121 slots 122 Frameset 123 Numbers 124 Map Derivation Instructions 126 Frame storage instruction 129 Similarity derivation instructions 134 Third Derivation Result 136 Fourth Derivation Result 138 Third Lesion Identification Image 140 Fourth Lesion Identification Image 144 External device 146 Network BB Bounding Box

Claims

1. A processor is provided. The processor, While an endoscope is inserted into a hollow organ, a first medical image is obtained by imaging the inside of the hollow organ with the endoscope, and a second medical image is obtained temporally after the first medical image by imaging the inside of the hollow organ with the endoscope, When a similarity between a first partial region that is a part of the first medical image and a second partial region that is a part of the second medical image exceeds a threshold, a first output process is executed to output the first medical image and / or a second output process is executed to output the second medical image; the first medical image output by executing the first output process is an image in which the first partial region and a first other region that is another region in the first medical image are represented in a discriminable manner, The second medical image output by executing the second output process is an image in which the second partial region and a second other region that is another region in the second medical image are expressed in a discriminable manner. Medical support equipment.

2. the first medical image output by executing the first output process is an image in which the first partial region is emphasized more than the first other region, The second medical image output by executing the second output process is an image in which the second partial region is emphasized more than the second other region. The medical support device according to claim 1 .

3. the first medical image is an image obtained during an insertion step in which the endoscope is inserted into the hollow organ, The second medical image is an image obtained during a removal process in which the endoscope is removed from the hollow organ. The medical support device according to claim 1 .

4. The first partial region is at least one of a plurality of first divided regions obtained by dividing the first medical image according to a first rule, The second partial region is at least one of a plurality of second divided regions obtained by dividing the second medical image according to the first rule. The medical support device according to claim 1 .

5. The processor, deriving a map indicating a distribution of the similarities between the first and second divided regions based on the first and second medical images; According to the derived map, the first partitioned regions and the second partitioned regions in which the similarity exceeds the threshold are extracted as the first partial regions and the second partial regions in which the similarity exceeds the threshold from the plurality of first partitioned regions and the plurality of second partitioned regions. The medical support device according to claim 4.

6. The processor derives the map using AI. The medical support device according to claim 5.

7. The map is derived in response to an externally provided first instruction. The medical support device according to claim 5.

8. The first partial region is at least one of a plurality of third divided regions obtained by dividing a first region, which is a part of the first medical image, according to a second rule; The second partial region is at least one of a plurality of fourth divided regions obtained by dividing a second region, which is a part of the second medical image, according to the second rule. The medical support device according to claim 1 .

9. The processor, deriving the similarity between each of the third segmented regions and each of the fourth segmented regions based on each of the third segmented regions and each of the fourth segmented regions; The third and fourth partitioned regions in which the similarity exceeds the threshold are extracted as the first and second partial regions in which the similarity exceeds the threshold from the third and fourth partitioned regions. The medical support device according to claim 8.

10. The processor derives the similarity using AI. The medical support device according to claim 9.

11. The similarity is derived in response to a second instruction given from the outside. The medical support device according to claim 10.

12. Each of the first region and the second region is a feature region recognized by performing an object recognition process as a region having a predetermined feature. The medical support device according to claim 8.

13. The characteristic region is a region in which a lesion is shown, a marked region, a region in which an organ is shown, and / or a region in which a treatment tool is shown. The medical support device according to claim 12.

14. The object recognition process uses AI for object recognition. The medical support device according to claim 12.

15. The first medical image is stored in a first storage area on the condition that the first area is recognized as the characteristic area by the object recognition processing. The medical support device according to claim 12.

16. The first medical image is stored in a second storage area in response to a third instruction given from outside. The medical support device according to claim 1 .

17. The first medical image and the second medical image are images obtained by imaging an area in the hollow organ irradiated with the same type of light using the endoscope. The medical support device according to claim 1 .

18. The first medical image outputted by executing the first output process is displayed on a first screen, The second medical image outputted by executing the second output process is displayed on a second screen. The medical support device according to claim 1 .

19. A processor is provided. The processor, While an endoscope is inserted into a hollow organ, a first medical image is obtained by imaging the inside of the hollow organ with the endoscope, and a second medical image is obtained temporally after the first medical image by imaging the inside of the hollow organ with the endoscope, When a degree of difference between a first partial region that is a part of the first medical image and a second partial region that is a part of the second medical image is less than a threshold, a first output process is executed to output the first medical image and / or a second output process is executed to output the second medical image; the first medical image output by executing the first output process is an image in which the first partial region and a first other region that is another region in the first medical image are represented in a discriminable manner, The second medical image output by executing the second output process is an image in which the second partial region and a second other region that is another region in the second medical image are expressed in a discriminable manner. Medical support equipment.

20. A medical support device according to any one of claims 1 to 19; The endoscope includes: Endoscope.

21. acquiring a first medical image obtained by imaging the inside of the tubular organ with the endoscope while the endoscope is inserted into the tubular organ, and a second medical image obtained later in time than the first medical image by imaging the inside of the tubular organ with the endoscope; and executing a first output process for outputting the first medical image and / or a second output process for outputting the second medical image when a similarity between a first partial region that is a part of the first medical image and a second partial region that is a part of the second medical image exceeds a threshold value; the first medical image output by executing the first output process is an image in which the first partial region and a first other region that is another region in the first medical image are represented in a discriminable manner, The second medical image output by executing the second output process is an image in which the second partial region and a second other region that is another region in the second medical image are expressed in a discriminable manner. Medical assistance methods.

22. A program for causing a computer to execute a medical support process, The medical support process includes: acquiring a first medical image obtained by imaging the inside of the tubular organ with the endoscope while the endoscope is inserted into the tubular organ, and a second medical image obtained later in time than the first medical image by imaging the inside of the tubular organ with the endoscope; and executing a first output process for outputting the first medical image and / or a second output process for outputting the second medical image when a similarity between a first partial region that is a part of the first medical image and a second partial region that is a part of the second medical image exceeds a threshold value; the first medical image output by executing the first output process is an image in which the first partial region and a first other region that is another region in the first medical image are represented in a discriminable manner, The second medical image output by executing the second output process is an image in which the second partial region and a second other region that is another region in the second medical image are expressed in a discriminable manner. program.

Images