Medical support device, endoscope device, medical support method, and program

Abstract

To provide a medical support device, endoscope device, medical support method, and program that can identify whether or not the same characteristic area is captured by a camera in the process of the camera moving deeper into the body and the process of the camera moving toward the entrance side of the body.SOLUTION: A medical support device includes a processor. The processor acquires position identification information that can identify the position of a camera inserted into the body in the body. The processor executes first identification processing capable of identifying the identity of a first feature area and a second feature area on the basis of insertion position identification information, which is position identification information corresponding to insertion process frames obtained by the camera capturing images of a first feature area within the body during the process of inserting the camera into the body, and removal position identification information, which is position identification information corresponding to removal process frames obtained by the camera capturing images of a second feature area within the body during the process of removing the camera from the body.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present disclosure relates to a medical support device, an endoscope device, a medical support method, and a program. [Background technology]

[0002] Patent Document 1 discloses an endoscopic image processing device having an image acquisition section, a lesion site detection section, a display section, and an image display control section. The image acquisition section acquires images corresponding to images of a subject captured over time by an endoscope inserted into a subject. The lesion site detection section detects a lesion site in an image each time an image is acquired. The display section displays the image. Based on the detection result of the lesion site detection section, the image display control section controls the display state of a plurality of images acquired by the image acquisition section, including at least a lesion site image in which a lesion site has been detected by the lesion site detection section.

[0003] The image display control unit also controls the display unit to display images for a predetermined period before and after the timing when the image acquisition unit acquires the lesion site image.The image display control unit also controls the display unit to display the lesion site image as a still image from a series of images acquired by the image acquisition unit in a reverse direction in time, the series being made up of a plurality of images from an image at the endoscope's completion of insertion to an image at the start of insertion, and images other than the lesion site image to be played in reverse as moving images.The image display control unit also controls the display unit to play in reverse as moving images the lesion site image and a predetermined number of images before and / or after the lesion site image from a series of images acquired by the image acquisition unit, the series being made up of a plurality of images from an image at the endoscope's completion of insertion to an image at the end of insertion.

[0004] The endoscope image processing device described in Patent Document 1 further includes an insertion state information acquisition unit. The insertion state information acquisition unit acquires insertion state data indicating the insertion state of the endoscope inserted into the subject from the endoscope insertion state detection device, and outputs information about the insertion state of the endoscope corresponding to the insertion state data at the timing when the image acquisition unit acquires the lesion site image to the display unit together with the lesion site image. Here, the information about the insertion state of the endoscope includes at least one of the insertion length of the endoscope, the elapsed time since the endoscope was inserted, and the insertion shape of the endoscope. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2008-301968 A Summary of the Invention

[0006] One embodiment of the present disclosure provides a medical support device, an endoscopic device, a medical support method, and a program that can determine whether the same characteristic area is imaged by the camera as the camera moves deeper into the body and as the camera moves toward the entrance of the body. [Means for solving the problem]

[0007] A first aspect of the present disclosure is a medical support device that includes a processor, and the processor acquires position identification information capable of identifying a position within the body of a camera to be inserted into the body, and executes a first identification process capable of identifying the identity of the first feature area and the second feature area based on insertion position identification information, which is position identification information corresponding to insertion process frames obtained by the camera capturing images of a first feature area within the body during the process of inserting the camera into the body, and removal position identification information, which is position identification information corresponding to removal process frames obtained by the camera capturing images of a second feature area within the body during the process of removing the camera from the body.

[0008] A second aspect of the present disclosure is a medical support device according to the first aspect, in which the first identification process includes a first notification process that notifies that the level of identity between the first feature region and the second feature region is equal to or higher than a first preset level when the level of identity between the first feature region and the second feature region is equal to or higher than a first preset level.

[0009] A third aspect of the present disclosure is a medical support device according to the first or second aspect, wherein the first identification process includes a first display process of displaying on a first screen a first image in which at least the first feature area can be identified and which is captured in an insertion process frame, and a second image in which at least the second feature area can be identified and which is captured in a removal process frame, in a comparative state when the level of identity between the first feature area and the second feature area is equal to or higher than a second predetermined level.

[0010] A fourth aspect of the present disclosure is a medical support device relating to any one of the first to third aspects, in which a processor acquires an insertion process frame and executes a first identification process on the condition that a plurality of insertion process frames equal to or greater than a first default number of frames have been acquired continuously.

[0011] A fifth aspect of the present disclosure is a medical support device according to any one of the first to fourth aspects, in which an insertion process frame is assigned an insertion process identifier that identifies it as a frame obtained by imaging a first feature region with a camera during the insertion process, and a removal process frame is assigned a removal process identifier that identifies it as a frame obtained by imaging a second feature region with a camera during the removal process.

[0012] A sixth aspect of the present disclosure is a medical support device according to any one of the first to fifth aspects, in which a processor executes a second identification process capable of identifying the identity of the third feature area and the fourth feature area based on pushing position identification information, which is position identification information corresponding to a pushing process frame obtained by imaging a third feature area inside the body with the camera during a pushing process in which the camera is pushed deep inside the body, and attraction position identification information, which is position identification information corresponding to a retraction process frame obtained by imaging a fourth feature area inside the body with the camera during a retraction process in which the camera is retracted from inside the body to the front.

[0013] A seventh aspect of the present disclosure is a medical support device that includes a processor, and the processor acquires position identification information capable of identifying a position within the body of a camera inserted into the body, and executes a second identification process capable of identifying the identity of the third feature region and the fourth feature region based on pushing position identification information, which is position identification information corresponding to a pushing process frame obtained by the camera imaging a third feature region within the body during a pushing process in which the camera is pushed deep inside the body, and attraction position identification information, which is position identification information corresponding to a pulling process frame obtained by the camera imaging a fourth feature region within the body during a pulling process in which the camera is pulled from the deep side to the front side.

[0014] An eighth aspect of the present disclosure is a medical support device according to the sixth or seventh aspect, in which the second identification process includes a second notification process that notifies the user that the level of identity between the third feature region and the fourth feature region is equal to or higher than a third preset level when the level of identity between the third feature region and the fourth feature region is equal to or higher than a third preset level.

[0015] A ninth aspect of the present disclosure is a medical support device according to any one of the sixth to eighth aspects, wherein the second identification process includes a second display process of displaying on the second screen a third image in which at least the third feature area in the pushing process frame can be identified and a fourth image in which at least the fourth feature area in the pulling process frame can be identified in a comparative state when the level of identity between the third feature area and the fourth feature area is equal to or higher than a fourth predetermined level.

[0016] A tenth aspect of the present disclosure is a medical support device relating to any one of the sixth to ninth aspects, in which a processor acquires a pushing process frame, and if the pushing process precedes the retraction process, executes a second specific process on the condition that a plurality of pushing process frames equal to or greater than a second predetermined number of frames have been continuously acquired.

[0017] An eleventh aspect of the present disclosure is a medical support device relating to any one of the sixth to ninth aspects, in which a processor acquires an attraction process frame, and when the attraction process precedes the pushing process, executes a second identification process on condition that a plurality of attraction process frames equal to or greater than a third predetermined number of frames have been continuously acquired.

[0018] A twelfth aspect of the present disclosure is a medical support device according to any one of the sixth to eleventh aspects, in which the pushing process frame is assigned a pushing process identifier that identifies it as a frame obtained by imaging the third feature region by the camera during the pushing process, and the retraction process frame is assigned a pulling process identifier that identifies it as a frame obtained by imaging the fourth feature region by the camera during the retraction process.

[0019] A thirteenth aspect of the present disclosure is a medical support device according to any one of the sixth to twelfth aspects, in which the third feature region and the fourth feature region are lesions.

[0020] A fourteenth aspect of the present disclosure is a medical support device according to any one of the first to thirteenth aspects, in which the first feature region and the second feature region are lesions.

[0021] A fifteenth aspect of the present disclosure is a medical support device according to any one of the first to fourteenth aspects, in which the position is estimated based on multiple frames obtained by imaging the inside of the body with a camera.

[0022] A sixteenth aspect of the present disclosure is a medical support device according to any one of the first to fifteenth aspects, in which the position identification information is defined based on a distance traveled by the camera within the body.

[0023] A seventeenth aspect of the present disclosure is an endoscope device including a medical support device according to any one of the first to sixteenth aspects and a camera.

[0024] An 18th aspect of the present disclosure is a medical support method that includes acquiring position identification information capable of identifying a position within the body of a camera to be inserted into the body, and performing a first identification process capable of identifying the identity of the first feature area and the second feature area based on insertion position identification information, which is position identification information corresponding to insertion process frames obtained by imaging a first feature area within the body with a camera during the process of inserting the camera into the body, and removal position identification information, which is position identification information corresponding to removal process frames obtained by imaging a second feature area within the body with the camera during the process of removing the camera from the body.

[0025] A 19th aspect of the present disclosure is a medical support method including acquiring position identification information capable of identifying a position within the body of a camera to be inserted into the body, and performing a second identification process capable of identifying the identity of the third feature region and the fourth feature region based on pushing position identification information, which is position identification information corresponding to a pushing process frame obtained by imaging a third feature region within the body by the camera during a pushing process in which the camera is pushed deep inside the body, and attraction position identification information, which is position identification information corresponding to a retraction process frame obtained by imaging a fourth feature region within the body by the camera during a retraction process in which the camera is retracted from the deep side to the front side.

[0026] A twentieth aspect of the present disclosure is a program for causing a computer to execute a process including acquiring position identification information capable of identifying a position within the body of a camera to be inserted into the body, and executing a first identification process capable of identifying the identity of the first feature region and the second feature region based on insertion position identification information, which is position identification information corresponding to insertion process frames obtained by the camera capturing images of a first feature region within the body during the process of inserting the camera into the body, and removal position identification information, which is position identification information corresponding to removal process frames obtained by the camera capturing images of a second feature region within the body during the process of removing the camera from the body.

[0027] A 21st aspect of the present disclosure is a program for causing a computer to execute processing including acquiring position identification information capable of identifying a position within the body of a camera to be inserted into the body, and executing a second identification process capable of identifying the identity of the third feature region and the fourth feature region based on pushing position identification information, which is position identification information corresponding to a pushing process frame obtained by imaging a third feature region within the body by the camera during a pushing process in which the camera is pushed deep inside the body, and attracting position identification information, which is position identification information corresponding to a pulling process frame obtained by imaging a fourth feature region within the body by the camera during a pulling process in which the camera is pulled from the deep side to the front side. [Brief description of the drawings]

[0028] [Figure 1] FIG. 1 is a conceptual diagram showing an example of an aspect in which an endoscope device is used by a doctor. [Diagram 2] 1 is a conceptual diagram showing an example of an overall configuration of an endoscope apparatus. [Diagram 3] 2 is a block diagram showing an example of a hardware configuration of an electrical system of the endoscope apparatus. FIG. [Figure 4] 2 is a block diagram showing an example of main functions of a processor included in the medical support device, and an example of information stored in a storage. [Diagram 5] 11A and 11B are conceptual diagrams showing an example of the contents of a recognition process and the contents displayed on a screen. [Figure 6]11 is a conceptual diagram showing an example of a manner in which an estimation unit executes a movement estimation process, and an example of a manner in which a control unit assigns an insertion process tag or a removal process tag, a movement distance, and a recognition result to a frame. [Figure 7] 4 is a conceptual diagram showing an example of a manner in which a control unit stores various types of information in first to third storage areas of a memory. FIG. [Figure 8] FIG. 11 is a conceptual diagram showing an example of processing performed by the control unit during the removal process based on a first reference movement distance stored in the first storage area, a first movement distance assigned to the insertion process frame stored in the second storage area, and a second movement distance assigned to the removal process frame stored in the third storage area. [Figure 9] A conceptual diagram showing an example of the processing contents by the control unit when a removal process frame is displayed in the first display area during the removal process, and an insertion process frame depicting the same lesion as the lesion depicted in the removal process frame is displayed in the second display area. [Figure 10A] 13 is a flowchart showing an example of the flow of a medical support process. [Figure 10B] This is a continuation of the flowchart shown in FIG. 10A. [Figure 11] 13 is a flowchart showing a modified example of the medical support process. [Figure 12] 4 is a conceptual diagram showing an example of a manner in which a control unit performs a first notification process. FIG. [Figure 13] FIG. 13 is a conceptual diagram showing an example of a transition from a push-in process to a pull-in process. [Figure 14] FIG. 13 is a conceptual diagram showing an example of a transition from a pulling process to a pushing process. [Figure 15] 11A and 11B are conceptual diagrams showing an example of an aspect in which an estimation unit executes a movement estimation process, and an example of an aspect in which a control unit assigns a pushing-in process tag or a pulling-in process tag, a movement distance, and a recognition result to a frame. [Figure 16] 13 is a conceptual diagram showing an example of a manner in which a control unit stores various types of information in fourth to sixth storage areas of a memory. FIG. [Figure 17]A conceptual diagram showing an example of the processing contents performed by the control unit based on the second reference movement distance stored in the fourth storage area, the third movement distance assigned to the pushing process frame stored in the fifth storage area, and the fourth movement distance assigned to the pulling process frame stored in the sixth storage area when there is a transition from the pushing process to the pulling process during the insertion process or removal process. [Figure 18] FIG. 11 is a conceptual diagram showing an example of processing by the control unit when an attraction process frame is displayed in the first display area during the attraction process, and a pushing process frame showing the same lesion as the lesion shown in the attraction process frame is displayed in the second display area. [Figure 19A] 13 is a flowchart showing an example of the flow of a first medical support process. [Figure 19B] This is a continuation of the flowchart shown in FIG. 19A. [Figure 20A] 13 is a flowchart showing an example of the flow of a second medical support process. [Figure 20B] This is a continuation of the flowchart shown in FIG. 20A. [Figure 21] 13 is a flowchart showing a modified example of the first medical support process. [Figure 22] 13 is a flowchart showing a modified example of the second medical support process. [Figure 23] 11 is a conceptual diagram showing an example of an aspect in which a control unit performs a second notification process. FIG. [Figure 24] 1 is a block diagram showing an example of a hardware configuration of an electrical system of an endoscope apparatus equipped with an endoscope insertion shape observation device. FIG. [Diagram 25] A conceptual diagram showing an example of a series of processes in which a processor included in a computer issues a processing execution request to an external device via a network, the external device executes processing in response to the processing execution request, and the processor included in the computer receives the processing result from the external device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] Hereinafter, examples of embodiments of a medical support device, an endoscope device, a medical support method, and a program according to the present disclosure will be described with reference to the accompanying drawings.

[0030] First, the terms used in the following description will be explained.

[0031] CPU is an abbreviation for "Central Processing Unit". GPU is an abbreviation for "Graphics Processing Unit". GPGPU is an abbreviation for "General-Purpose computing on Graphics Processing Units". APU is an abbreviation for "Accelerated Processing Unit". TPU is an abbreviation for "Tensor Processing Unit". RAM is an abbreviation for "Random Access Memory". NVM is an abbreviation for "Non-volatile memory". EEPROM is an abbreviation for "Electrically Erasable Programmable Read-Only Memory". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". SSD is an abbreviation for "Solid State Drive". USB is an abbreviation for "Universal Serial Bus". HDD is an abbreviation for "Hard Disk Drive". EL is an abbreviation for "Electro-Luminescence". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". AI is an abbreviation for "Artificial Intelligence". BLI is an abbreviation for "Blue Light Imaging". LCI is an abbreviation for "Linked Color Imaging". I / F is an abbreviation for "Interface". SSL is an abbreviation for "Sessile Serrated Lesion". LAN is an abbreviation for "Local Area Network". WAN is an abbreviation for "Wide Area Network".5G is an abbreviation for "5th Generation Mobile Communication System."

[0032] In the following description, a coded processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Also, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU, a GPU, a GPGPU, an APU, and a TPU.

[0033] In the following description, a signed memory is a memory, such as a RAM, in which information is temporarily stored and which is used as a working memory by a processor.

[0034] In the following description, a storage with a symbol is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory, magnetic disks, and magnetic tapes. Another example of storage is cloud storage.

[0035] In the following embodiments, the external I / F with a symbol controls the transmission and reception of various information between multiple devices connected to each other. An example of the external I / F is a USB interface. A communication I / F including a communication processor and an antenna may be applied to the external I / F. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G, Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0036] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0037] [First embodiment] Fig. 1 is a conceptual diagram showing an example of a mode in which an endoscope device 10 is used. As shown in Fig. 1, the endoscope device 10 is used by a doctor 12 in an endoscopic examination or the like. The endoscopic examination is assisted by staff such as a nurse 14.

[0038] The endoscope device 10 is communicatively connected to a communication device (not shown), and information obtained by the endoscope device 10 is transmitted to the communication device. Examples of the communication device include a server that manages various information such as electronic medical records, a personal computer, and / or a tablet terminal. The communication device receives the information transmitted from the endoscope device 10 and executes processing using the received information (e.g., processing to store the information in an electronic medical record, etc.).

[0039] The endoscope device 10 includes an endoscope scope 16, a display device 18, a light source device 20, a control device 22, and a medical support device 24. In the first embodiment, the endoscope device 10 is an example of an "endoscope device" according to the present disclosure.

[0040] The endoscope device 10 is a modality for performing medical treatment on a large intestine 28 contained in the body of a subject 26 (e.g., a patient) using an endoscope scope 16. In this first embodiment, the large intestine 28 is an object to be observed by a doctor 12.

[0041] The endoscope 16 is used by the doctor 12 and inserted into the body of a subject 26. In the first embodiment, the endoscope 16 is inserted into the large intestine 28, which is a hollow organ of the subject 26.

[0042] The endoscope device 10 causes the endoscope scope 16 inserted into the large intestine 28 of the subject 26 to capture images of the inside of the large intestine 28 of the subject 26, and also performs various medical procedures on the large intestine 28 as necessary.

[0043] The endoscope device 10 captures an image of the inside of the large intestine 28 of the subject 26, thereby acquiring and outputting an image showing the state inside the large intestine 28. In the present first embodiment, the endoscope device 10 is an endoscope having an optical imaging function for capturing an image of reflected light obtained by irradiating the inside of the large intestine 28 with light 30 and reflecting it off a wall 32 of the large intestine 28.

[0044] Although an endoscopic examination of the large intestine 28 is illustrated here, this is merely one example, and the present disclosure also applies to an endoscopic examination of a hollow organ such as the esophagus, stomach, duodenum, or trachea.

[0045] The light source device 20, the control device 22, and the medical support device 24 are installed on a wagon 34. The wagon 34 has a plurality of tables arranged vertically, and the medical support device 24, the control device 22, and the light source device 20 are installed from the lower table to the upper table. In addition, the display device 18 is installed on the top table of the wagon 34.

[0046] The control device 22 controls the entire endoscope device 10. The medical support device 24, under the control of the control device 22, performs various image processing on the images obtained by imaging the intestinal wall 32 by the endoscope scope 16.

[0047] The display device 18 displays various information including images. Examples of the display device 18 include a liquid crystal display and an EL display. Also, instead of the display device 18 or together with the display device 18, a tablet terminal with a display may be used.

[0048] The display device 18 displays a screen 35. The screen 35 includes a plurality of display areas. The plurality of display areas are arranged side by side within the screen 35. In the example shown in FIG. 1, a first display area 36 and a second display area 38 are shown as examples of the plurality of display areas. The size of the first display area 36 is larger than the size of the second display area 38. The first display area 36 is used as a main display area, and the second display area 38 is used as a sub display area. Note that the size relationship between the first display area 36 and the second display area 38 is not limited thereto, and may be any size relationship that fits within the screen 35. In the first embodiment, the screen 35 is an example of a "first screen" and a "second screen" according to the present disclosure.

[0049] An endoscopic moving image 39 is displayed in the first display area 36. The endoscopic moving image 39 is a moving image acquired by imaging an intestinal wall 32 in the large intestine 28 of the subject 26 with the endoscope scope 16. In the example shown in Fig. 1, a moving image showing the intestinal wall 32 is shown as an example of the endoscopic moving image 39.

[0050] The intestinal wall 32 shown in the endoscopic video 39 includes a lesion 42 (e.g., one lesion 42 in the example shown in FIG. 1) as a region of interest (i.e., region to be observed) gazed upon by the doctor 12, and the doctor 12 can visually recognize the state of the intestinal wall 32 including the lesion 42 through the endoscopic video 39. In this first embodiment, the lesion 42 is an example of the "first characteristic region," "second characteristic region," "third characteristic region," "fourth characteristic region," and "lesion" according to the present disclosure.

[0051] There are various types of lesions 42, and examples of the types of lesions 42 include neoplastic polyps and non-neoplastic polyps. Examples of the types of neoplastic polyps include adenomatous polyps (e.g., SSL). Examples of the types of non-neoplastic polyps include hamartomatous polyps, hyperplastic polyps, and inflammatory polyps. Note that the types exemplified here are types that are assumed in advance as types of lesions 42 when an endoscopic examination is performed on the large intestine 28, and the types of lesions 42 will differ if the organ in which the endoscopic examination is performed is different.

[0052] In this first embodiment, for ease of explanation, an example is given in which one lesion 42 is captured in the endoscopic video 39, but the present disclosure is not limited to this, and the present disclosure also applies when multiple lesions 42 are captured in the endoscopic video 39.

[0053] In this first embodiment, a lesion 42 is illustrated, but this is merely one example, and the area of ​​interest (i.e., the area to be observed) gazed upon by the doctor 12 may be a characteristic area having some unique feature, such as an organ (e.g., the duodenal papilla), a mark, an artificial treatment tool (e.g., an artificial clip), or a treated area (e.g., an area where traces remain after the removal of a polyp, etc.), etc.

[0054] The image displayed in the first display area 36 is one frame 40 included in a moving image that includes a plurality of frames 40 arranged in chronological order. In other words, the first display area 36 displays the plurality of frames 40 arranged in chronological order at a default frame rate (e.g., several tens of frames per second).

[0055] An example of a moving image displayed in the first display area 36 is a moving image of a live view system. The live view system is merely one example, and a moving image that is temporarily stored in a memory or the like and then displayed, such as a moving image of a post-view system, may also be used. Furthermore, each frame included in a moving image for recording stored in a memory or the like may be reproduced and displayed on the screen 35 (for example, the first display area 36) as an endoscopic moving image 39.

[0056] Within the screen 35, the second display area 38 is adjacent to the first display area 36, ​​and is displayed at the lower right when viewed from the front within the screen 35. The display position of the second display area 38 may be anywhere within the screen 35 of the display device 18, but it is preferable that it is displayed at a position where it can be contrasted with the endoscopic moving image 39.

[0057] In the second display area 38, auxiliary information 44 that assists the doctor 12 in making medical decisions during an endoscopic examination is displayed. The auxiliary information 44 is information that is referred to by the doctor 12. Examples of the auxiliary information 44 include various types of information related to the subject 26 into whose body the endoscope 16 is inserted, and / or various types of information obtained by performing medical support processing, which will be described later.

[0058] An endoscopic examination of the large intestine 28 includes an insertion process 41A and a removal process 41B. The insertion process 41A is a process in which the endoscope 16 advances to the back of the large intestine 28, i.e., a process in which the endoscope 16 is inserted into the large intestine 28 (as an example here, a process in which the endoscope 16 is inserted from the anus to the ileocecal part 28A). The removal process 41B is a process in which the endoscope 16 advances to the entrance side of the large intestine 28, i.e., a process in which the endoscope 16 is removed from the large intestine 28 (as an example here, a process in which the endoscope 16 is removed from the ileocecal part 28A to the anus).

[0059] In the insertion process 41A, the endoscope 16 is inserted into the large intestine 28 along the insertion path 29A. For example, the insertion path 29A refers to a path from the anus (not shown) to the ileocecal area 28A. Here, the anus is illustrated as the starting point of the insertion path 29A, but this is merely one example, and the starting point may be, for example, the position in the large intestine 28 where the endoscopic examination is started (for example, the position at the time when an instruction to start the endoscopic examination is given).

[0060] In the removal process 41B, the endoscope 16 is removed from the large intestine 28 along the removal path 29B. For example, the removal path 29B refers to a path from the ileocecal area 28A to the anus.

[0061] In the insertion process 41A, when the endoscope 16 reaches the ileocecal portion 28A of the large intestine 28 along the insertion path 29A, the process switches from the insertion process 41A to a removal process 41B, and the endoscope 16 is removed from the large intestine 28 along the removal path 29B. In the insertion process 41A, the endoscope device 10 captures images of the inside of the large intestine 28. In the removal process 41B, similar to the insertion process 41A, the endoscope device 10 captures images of the inside of the large intestine 28. In addition, in the removal process 41B, the endoscope device 10 performs a medical procedure (e.g., differentiation, marking, resection, and / or hemostasis, etc.).

[0062] Fig. 2 is a conceptual diagram showing an example of the overall configuration of the endoscope device 10. As shown in Fig. 2, the endoscope scope 16 includes an operation section 46 and an insertion section 48. The insertion section 48 is partially curved by operating the operation section 46. The insertion section 48 is inserted into the large intestine 28 (see Fig. 1) while curving in accordance with the shape of the large intestine 28 (see Fig. 1) in accordance with the operation of the operation section 46 by the doctor 12 (see Fig. 1).

[0063] The distal end 50 of the insertion section 48 is provided with a camera 52, an illumination device 54, and a treatment tool opening 56. The camera 52 and the illumination device 54 are provided on the distal end surface 50A of the distal end 50. Note that, although an example in which the camera 52 and the illumination device 54 are provided on the distal end surface 50A of the distal end 50 is given here, this is merely one example, and the camera 52 and the illumination device 54 may be provided on the side surface of the distal end 50, so that the endoscope 16 is configured as a side-viewing scope.

[0064] The camera 52 is inserted into the body cavity of the subject 26 to capture an image of the observation target region. In the first embodiment, the camera 52 captures an image of the inside of the subject 26 (e.g., inside the large intestine 28) to obtain the endoscopic moving image 39. An example of the camera 52 is a CMOS camera. However, this is merely an example, and other types of cameras such as a CCD camera may also be used. The camera 52 is an example of a "camera" according to the present disclosure.

[0065] The illumination device 54 has illumination windows 54A and 54B. The illumination device 54 irradiates light 30 (see FIG. 1) through the illumination windows 54A and 54B. Examples of the light 30 irradiated from the illumination device 54 include visible light (e.g., white light) and non-visible light (e.g., near-infrared light). The illumination device 54 also irradiates special light through the illumination windows 54A and 54B. Examples of the special light include light for BLI and / or light for LCI. The camera 52 captures an image of the inside of the large intestine 28 by an optical method while the light 30 is irradiated inside the large intestine 28 by the illumination device 54.

[0066] The treatment tool opening 56 is an opening for allowing a treatment tool 58 to protrude from the distal end portion 50. The treatment tool opening 56 is also used as a suction port for sucking blood, internal waste, and the like, and as a delivery port for delivering fluid.

[0067] A treatment tool insertion port 60 is formed in the operation section 46, and the treatment tool 58 is inserted into the insertion section 48 from the treatment tool insertion port 60. The treatment tool 58 passes through the insertion section 48 and protrudes to the outside from the treatment tool opening 56. In the example shown in FIG. 2, a puncture needle is shown as the treatment tool 58 protruding from the treatment tool opening 56. Here, a puncture needle is shown as the treatment tool 58, but this is merely one example, and the treatment tool 58 may be a grasping forceps, a papillotomy knife, a snare, a catheter, a guidewire, a cannula, and / or a puncture needle with a guide sheath, etc.

[0068] The endoscope 16 is connected to a light source device 20 and a control device 22 via a universal cord 62. A medical support device 24 and a reception device 64 are connected to the control device 22. In addition, a display device 18 is connected to the medical support device 24. That is, the control device 22 is connected to the display device 18 via the medical support device 24.

[0069] In this example, the medical support device 24 is shown as an external device for expanding the functions performed by the control device 22, and therefore an example is shown in which the control device 22 and the display device 18 are indirectly connected via the medical support device 24, but this is merely one example. For example, the display device 18 may be directly connected to the control device 22. In this case, for example, the function of the medical support device 24 may be installed in the control device 22, or the control device 22 may be equipped with a function to cause a server (not shown) to execute the same process as the process executed by the medical support device 24 (for example, the medical support process described later) and to receive and use the results of the process by the server.

[0070] The reception device 64 receives instructions from the doctor 12 and outputs the received instructions as an electrical signal to the control device 22. Examples of the reception device 64 include a keyboard, a mouse, a touch panel, a foot switch, a microphone, and / or a remote control device.

[0071] The control device 22 controls the light source device 20, exchanges various signals with the camera 52, and exchanges various signals with the medical support device 24.

[0072] The light source device 20 emits light under the control of the control device 22, and supplies the light to the illumination device 54. A light guide is built into the illumination device 54, and the light supplied from the light source device 20 passes through the light guide and is irradiated from illumination windows 54A and 54B. The control device 22 causes the camera 52 to capture an image, acquires an endoscopic video image 39 (see FIG. 1) from the camera 52, and outputs it to a predetermined output destination (for example, the medical support device 24).

[0073] The medical support device 24 performs various types of image processing on the endoscopic moving image 39 input from the control device 22 to provide support for medical care (here, as an example, an endoscopic examination). The medical support device 24 outputs the endoscopic moving image 39 that has been subjected to various types of image processing to a predetermined output destination (for example, the display device 18).

[0074] Although an example in which the endoscopic moving image 39 output from the control device 22 is output to the display device 18 via the medical support device 24 has been described here, this is merely one example. For example, the control device 22 and the display device 18 may be connected, and the endoscopic moving image 39 that has been subjected to image processing by the medical support device 24 may be displayed on the display device 18 via the control device 22.

[0075] Fig. 3 is a block diagram showing an example of a hardware configuration of an electrical system of the endoscope device 10. As shown in Fig. 3, the control device 22 includes a computer 66, a bus 68, and an external I / F 70. The computer 66 includes a processor 72, a memory 74, and a storage 76. The processor 72, the memory 74, the storage 76, and the external I / F 70 are connected to the bus 68. The processor 72 controls the entire control device 22. The memory 74 and the storage 76 are used by the processor 72.

[0076] The external I / F 70 is responsible for transmitting and receiving various types of information between the processor 72 and one or more devices (hereinafter, also referred to as “first external devices”) that exist outside the control device 22.

[0077] The camera 52 is connected to the external I / F 70 as one of the first external devices, and the external I / F 70 is responsible for transmitting and receiving various types of information between the camera 52 and the processor 72. The processor 72 controls the camera 52 via the external I / F 70. The processor 72 also acquires, via the external I / F 70, an endoscopic moving image 39 (see FIG. 1) obtained by the camera 52 capturing an image of the inside of the large intestine 28 (see FIG. 1).

[0078] The light source device 20 is connected to the external I / F 70 as one of the first external devices, and the external I / F 70 controls the exchange of various information between the light source device 20 and the processor 72. The light source device 20 supplies light to the illumination device 54 under the control of the processor 72. The illumination device 54 irradiates the light supplied from the light source device 20.

[0079] A reception device 64 is connected to the external I / F 70 as one of the first external devices, and the processor 72 acquires instructions accepted by the reception device 64 via the external I / F 70 and executes processing according to the acquired instructions.

[0080] The medical support device 24 includes a computer 78 and an external I / F 80. The computer 78 includes a processor 82, a memory 84, and a storage 86. The processor 82, the memory 84, the storage 86, and the external I / F 80 are connected to a bus 88. In the first embodiment, the medical support device 24 is an example of a "medical support device" according to the present disclosure, the computer 78 is an example of a "computer" according to the present disclosure, and the processor 82 is an example of a "processor" according to the present disclosure.

[0081] The hardware configuration of the computer 78 (i.e., the processor 82, the memory 84, and the storage 86) is basically the same as the hardware configuration of the computer 66, so a description of the hardware configuration of the computer 78 will be omitted here.

[0082] The external I / F 80 controls the exchange of various information between the processor 82 and one or more devices (hereinafter, also referred to as “second external devices”) that exist outside the medical support device 24.

[0083] The control device 22 is connected to the external I / F 80 as one of the second external devices. In the example shown in Fig. 3, the external I / F 70 of the control device 22 is connected to the external I / F 80. The external I / F 80 controls the exchange of various information between the processor 82 of the medical support device 24 and the processor 72 of the control device 22. For example, the processor 82 acquires endoscopic moving images 39 (see Fig. 1) from the processor 72 of the control device 22 via the external I / Fs 70 and 80, and performs various image processing on the acquired endoscopic moving images 39.

[0084] The display device 18, which serves as one of the second external devices, is connected to the external I / F 80. The processor 82 controls the display device 18 via the external I / F 80 to cause the display device 18 to display various information (e.g., an endoscopic moving image 39 that has been subjected to various image processing, etc.).

[0085] During an endoscopic examination, the doctor 12 checks the endoscopic moving image 39 via the display device 18 and determines whether or not medical treatment is required for the lesion 42 shown in the endoscopic moving image 39, and performs medical treatment on the lesion 42 if necessary.

[0086] In a typical endoscopy, when the doctor 12 finds a lesion 42 via the screen 35 (see FIG. 1) during the insertion process 41A (see FIG. 1), the doctor 12 performs a medical procedure on the lesion 42 found during the insertion process 41A during the removal process 41B. In this case, the doctor 12 needs to visually identify the lesion 42 found during the insertion process 41A again without omission during the removal process 41B.

[0087] However, in general, in the insertion process 41A, the endoscope 16 is inserted into the large intestine 28 with air being aspirated inside the large intestine 28, and in the removal process 41B, the endoscope 16 is removed from the large intestine 28 with air being pumped into the large intestine 28. Therefore, even if the lesion 42 is the same, the appearance of the lesion 42 observed in the insertion process 41A may differ from the appearance of the lesion 42 observed in the removal process 41B. In addition, even if the lesion 42 is the same, the appearance of the lesion 42 observed in the insertion process 41A may differ from the appearance of the lesion 42 observed in the removal process 41B due to body movement or a change in the orientation of the camera 52 between the insertion process 41A and the removal process 41B. In this case, the doctor 12 is more likely to miss the lesion 42 discovered in the insertion process 41A in the removal process 41B than when the appearance of the lesion 42 observed in the insertion process 41A is the same as the appearance of the lesion 42 observed in the removal process 41B.

[0088] In order to ensure that medical treatment is performed on each and every lesion 42 detected during endoscopic examination, it is extremely important that all lesions 42 detected during the insertion process 41A are visually identified during the removal process 41B.

[0089] In view of such circumstances, in the first embodiment, as an example shown in Fig. 4, medical support processing is performed by a processor 82 of the medical support device 24. Note that Fig. 4 is a block diagram showing an example of main functions of the processor 82 included in the medical support device 24 and an example of information stored in a storage 86.

[0090] A medical support program 90 is stored in the storage 86. The medical support program 90 is an example of a "program" according to the present disclosure. The processor 82 reads the medical support program 90 from the storage 86 and executes the read medical support program 90 on the memory 84 to perform medical support processing. The medical support processing is realized by the processor 82 operating as a recognition unit 82A, an estimation unit 82B, and a control unit 82C in accordance with the medical support program 90 executed on the memory 84.

[0091] The storage 86 stores a recognition model 92 and a movement estimation model 94. As will be described in detail later, the recognition model 92 is used by the recognition unit 82A, and the movement estimation model 94 is used by the estimation unit 82B.

[0092] Fig. 5 is a conceptual diagram showing an example of the processing contents of the recognition unit 82A, the estimation unit 82B, and the control unit 82C. As shown in Fig. 5, the recognition unit 82A, the estimation unit 82B, and the control unit 82C acquire, from the camera 52, each of a plurality of frames 40 in time series included in the endoscopic moving image 39 generated by capturing images by the camera 52 at an imaging frame rate (e.g., several tens of frames / second), in a frame-by-frame manner in time series. The recognition unit 82A, the estimation unit 82B, and the control unit 82C acquire the frames 40 at the same timing.

[0093] The control unit 82C outputs the endoscopic moving image 39 to the display device 18. For example, the control unit 82C displays the endoscopic moving image 39 as a live view image in the first display area 36. That is, every time the control unit 82C acquires a frame 40 from the camera 52, the control unit 82C displays the acquired frame 40 in sequence in the first display area 36 according to a display frame rate (for example, several tens of frames per second). The control unit 82C also displays auxiliary information 44 in the second display area 38. For example, the control unit 82C also updates the display content of the second display area 38 (for example, auxiliary information 44) in accordance with the display content of the first display area 36.

[0094] The recognition unit 82A recognizes a lesion 42 in the endoscopic video 39 based on the endoscopic video 39 acquired from the camera 52. That is, the recognition unit 82A sequentially performs a recognition process 96 on each of a plurality of frames 40 in time series included in the endoscopic video 39 acquired from the camera 52, thereby recognizing the lesion 42 appearing in the frame 40. For example, the recognition unit 82A recognizes the geometric characteristics (e.g., position and shape, etc.) of the lesion 42, the type of the lesion 42, and the type of the lesion 42 (e.g., pedunculated, subpedunculated, sessile, surface elevated, surface flat, surface depressed, etc.), etc.

[0095] Furthermore, recognition unit 82A recognizes a region (hereinafter simply referred to as a "region") in endoscopic video 39 based on endoscopic video 39 acquired from camera 52. Here, a region refers to a region of large intestine 28. Regions of large intestine 28 include, for example, the anal canal, lower rectum, upper rectum, sigmoid colon, descending colon, transverse colon, ascending colon, and ileocecal region.

[0096] Recognition process 96 is performed by recognition unit 82A on acquired frame 40 each time the frame 40 is acquired. Recognition process 96 is a process for recognizing lesions 42 and regions using an AI-based method. Here, recognition process 96 is performed using recognition model 92. Recognition model 92 is a trained model for object recognition using an AI-based bounding box method.

[0097] The recognition model 92 is optimized by performing machine learning on the neural network using first training data. The first training data is a data set including a plurality of data (i.e., data for a plurality of frames) in which the first example data and the first answer data are associated with each other.

[0098] The first example data is an image assuming frame 40. An example of a first image assuming frame 40 is an image obtained by actually capturing an image of the inside of the large intestine with a camera. An example of a second image assuming frame 40 is a virtually created image. The first correct answer data is correct answer data (i.e., annotations) for the first example data. Here, annotations that identify the geometric characteristics, type, model, and location of a lesion shown in the image used as the first example data are used as an example of the first correct answer data.

[0099] The recognition unit 82A acquires a frame 40 from the camera 52, and inputs the acquired frame 40 to the recognition model 92. As a result, each time a frame 40 is input, the recognition model 92 recognizes the lesion 42 and the area shown in the input frame 40, and outputs a recognition result 98.

[0100] The recognition unit 82A outputs the recognition result 98 itself to the display device 18 or the like in synchronization with the display of the frame 40 in the first display area 36, ​​or outputs information generated based on the recognition result 98 to the display device 18 or the like in synchronization with the display of the frame 40 in the first display area 36. The recognition result 98 itself or information generated based on the recognition result 98 is displayed on the screen 35 of the display device 18 (for example, the first display area 36 and / or the second display area 38, etc.). The display of the recognition result 98 itself and the display of the information generated based on the recognition result 98 are updated in synchronization with the display timing of the endoscopic moving image 39 displayed in the first display area 36. That is, the display of the recognition result 98 itself and the display of the information generated based on the recognition result 98 are updated according to the display frame rate applied to the first display area 36.

[0101] The recognition result 98 includes lesion presence / absence information 100. The lesion presence / absence information 100 is information indicating whether or not a lesion 42 is captured in the frame 40 input to the recognition model 92. The recognition result 98 also includes geometric property information 102, a lesion location map 104, lesion feature information 106, and the like.

[0102] The geometric property information 102 is information (e.g., coordinates) capable of identifying the shape and position of the lesion 42 within the frame 40. The lesion position map 104 is a map capable of identifying the position of the lesion 42 within the frame 40. The lesion feature information 106 is information capable of identifying the type of lesion 42 and the model of lesion 42 appearing in the frame 40 that has been input to the recognition model 92.

[0103] The geometric characteristics of the lesion location map 104 (e.g., the shape and size of the outer contour) correspond to the geometric characteristics of the frame 40 (e.g., the shape and size of the outer contour). The lesion location map 104 includes a bounding box BB. The bounding box BB is a rectangular frame (e.g., a rectangular frame circumscribing an image area showing the lesion 42) capable of identifying the position recognized by the recognition model 92 as the position within the frame 40 of the lesion 42 captured in the frame 40. In the example shown in FIG. 5, the bounding box BB is superimposed and displayed at the position where the lesion 42 is captured within the frame 40 displayed in the first display area 36. The display of the bounding box BB is updated in synchronization with the display timing of the endoscopic moving image 39 displayed in the first display area 36.

[0104] The lesion location map 104 may be displayed on the screen 35 (for example, in the second display area 38) by the control unit 82C as one piece of auxiliary information 44. In this case, the lesion location map 104 displayed on the screen 35 is updated according to the display frame rate applied to the first display area 36. With this configuration, the doctor 12 can grasp the general location of the lesion 42 in the endoscopic moving image 39 displayed in the first display area 36 by referring to the lesion location map 104 displayed in the second display area 38 while observing the endoscopic moving image 39 displayed in the first display area 36.

[0105] The control unit 82C may display the lesion characteristic information 106 on the screen 35. For example, the lesion characteristic information 106 may be displayed in the second display area 38 as one piece of auxiliary information 44, or may be displayed in a location on the screen 35 other than the second display area 38. The control unit 82C may also display an identifier (for example, a mark) in place of the bounding box BB on the screen 35 (in the example shown in FIG. 5, the first display area 36 on the screen 35). In this case, for example, the control unit 82C may specify the shape and position of the lesion 42 in the frame 40 from the geometric property information 102, and display the identifier in place of the bounding box BB at the specified shape and position.

[0106] The recognition result 98 also includes body part information 108. The body part information 108 is information that can identify a body part shown in the frame 40.

[0107] Fig. 6 is a conceptual diagram showing an example of the processing contents of the estimation unit 82B and the control unit 82C during the insertion process 41A or the removal process 41B. As shown in Fig. 6, the estimation unit 82B recognizes the movement mode of the camera 52 inserted into the large intestine 28 by performing a movement estimation process 110 based on the endoscopic video image 39 acquired from the camera 52. Here, the movement mode refers to, for example, a movement distance 112 and a movement direction 114.

[0108] In the first embodiment, information capable of identifying the position within the large intestine 28 of the camera 52 inserted into the large intestine 28 is defined by the travel distance 112. The travel distance 112 is an example of "position identification information" and "travel distance" according to the present disclosure. Note that information capable of identifying the position within the large intestine 28 of the camera 52 inserted into the large intestine 28 may be defined by the travel distance 112, the travel direction 114, etc.

[0109] The movement distance 112 is the movement distance of the camera 52 in the large intestine 28. The movement distance of the camera 52 in the large intestine 28 refers to, for example, the movement distance of the camera 52 at the time when the camera 52 captures an image from the start of the insertion of the camera 52 into the large intestine 28 to the completion of the removal of the camera 52 from the large intestine 28 (for example, the distance the camera 52 moves from the position where the camera 52 is inserted into the large intestine 28 (for example, the position of the anus) to the current position). For convenience of explanation, in the first embodiment, the camera 52 advances to the ileocecal portion 28A (see FIG. 1) along the insertion path 29A (see FIG. 1) without turning back in the insertion process 41A (see FIG. 1), and advances to the anus side along the removal path 29B (see FIG. 1) without turning back to the ileocecal portion 28A side in the removal process 41B (see FIG. 1).

[0110] The movement direction 114 is an insertion direction or a removal direction. The insertion direction refers to the direction in which the camera 52 is inserted into the large intestine 28 (i.e., the direction in which the camera 52 advances along the insertion path 29A toward the ileocecal portion 28A). The removal direction refers to the direction in which the camera 52 is removed from the large intestine 28 (i.e., the direction in which the camera 52 advances toward the anus along the removal path 29B).

[0111] The movement estimation process 110 is performed by the estimation unit 82B every time a frame 40 is acquired. The movement estimation process 110 is a process that estimates a movement distance 112 and a movement direction 114 by a method using AI. Here, as the movement estimation process 110, a process that uses a movement estimation model 94 is performed. The movement estimation model 94 is a trained model that estimates the movement distance 112 and the movement direction 114 based on a plurality of frames 40 along an input time series.

[0112] The movement estimation model 94 is optimized by performing machine learning using second training data on the neural network. The second training data is a data set including a plurality of data in which the second example data and the second answer data are associated with each other.

[0113] The second example data is an image assuming a plurality of frames 40 in a time series (for example, two frames 40 in a time series). A first example of an image assuming a plurality of frames 40 in a time series is a plurality of images in a time series obtained by actually capturing images of the inside of the large intestine with a camera. A second example of an image assuming frames 40 is a plurality of images virtually created assuming a plurality of images in a time series obtained by actually capturing images of the inside of the large intestine with a camera. The second correct answer data is correct answer data (i.e., annotation) for the second example data. Here, for example, an annotation that specifies the distance traveled within the large intestine of a camera inserted into the large intestine and the direction of travel of the camera within the large intestine is used as the second correct answer data.

[0114] The estimation unit 82B acquires a plurality of frames 40 in time series from the camera 52, and inputs the acquired plurality of frames 40 to the movement estimation model 94. As a result, each time a plurality of frames 40 in time series are input, the movement estimation model 94 estimates a movement distance 112 and a movement direction 114 based on the input plurality of frames 40, and outputs the estimated movement distance 112 and movement direction 114 to the control unit 82C.

[0115] Note that, although an example in which the movement distance 112 and the movement direction 114 are estimated by the movement estimation model 94 has been described here, the present disclosure is not limited thereto. For example, the movement distance 112 and / or the movement direction 114 may be estimated based on an optical flow calculated from a plurality of frames 40 along a time series. Also, the movement distance 112 and / or the movement direction 114 may be estimated based on both the movement estimation model 94 and the optical flow.

[0116] The control unit 82C acquires the recognition result 98 from the recognition unit 82A, and refers to the lesion presence / absence information 100 included in the acquired recognition result 98 to determine whether or not the lesion 42 is captured in the frame 40 input to the recognition model 92. Here, if the lesion 42 is captured in the frame 40 input to the recognition model 92 to obtain the recognition result 98 acquired by the control unit 82C from the recognition unit 82A, the control unit 82C generates an insertion process tag 116 or a removal process tag 118 based on the movement direction 114 input from the estimation unit 82B. The insertion process tag 116 refers to a tag that is an identifier that can identify the frame 40 acquired by imaging an area including the lesion 42 by the camera 52 during the insertion process 41A (i.e., an area including the lesion 42 that is a part of the entire area of ​​the intestinal wall 32 that falls within the angle of view of the camera 52). The removal process tag 118 refers to a tag that is an identifier that can identify the frame 40 as being obtained by imaging an area including the lesion 42 (i.e., a portion of the entire area of ​​the intestinal wall 32 that falls within the angle of view of the camera 52 and that includes the lesion 42) by the camera 52 during the removal process 41B.

[0117] When the control unit 82C generates the insertion process tag 116 or the removal process tag 118 in this manner, first, the control unit 82C determines whether the movement direction 114 input from the estimation unit 82B is the insertion direction or the removal direction. Then, the control unit 82C generates the insertion process tag 116 if the movement direction 114 input from the estimation unit 82B is the insertion direction, and generates the removal process tag 118 if the movement direction 114 input from the estimation unit 82B is the removal direction. Note that in this first embodiment, the insertion process tag 116 is an example of an "insertion process identifier" according to the present disclosure, and the removal process tag 118 is an example of a "removal process identifier" according to the present disclosure.

[0118] The control unit 82C assigns the insertion process tag 116 or the removal process tag 118, the movement distance 112, and the recognition result 98 to the frame 40 acquired from the camera 52 and showing the lesion 42. The frame 40 to which the insertion process tag 116 or the removal process tag 118, the movement distance 112, and the recognition result 98 are assigned is the same frame 40 as the latest frame 40 input to the movement estimation model 94 to estimate the movement direction 114 used to generate the insertion process tag 116 or the removal process tag 118. The movement distance 112 assigned to the frame 40 is the movement distance 112 estimated by the movement estimation model 94 together with the movement direction 114 used to generate the insertion process tag 116 or the removal process tag 118. The recognition result 98 assigned to the frame 40 is the recognition result 98 used by the control unit 82C to determine the presence or absence of the lesion 42.

[0119] The movement distance 112 estimated by the movement estimation model 94 together with the movement direction 114 used to generate the insertion process tag 116 is assigned to the frame 40 as a first movement distance 112A. The movement distance 112 estimated by the movement estimation model 94 together with the movement direction 114 used to generate the removal process tag 118 is assigned to the frame 40 as a second movement distance 112B.

[0120] Here, the first movement distance 112A is an example of "insertion position specifying information" according to the present disclosure, and the second movement distance 112B is an example of "removal position specifying information" according to the present disclosure.

[0121] FIG. 7 is a conceptual diagram showing an example of a manner in which various information is stored in the memory 84 by the control unit 82C. As shown in FIG. 7, the memory 84 has a first storage area 84A, a second storage area 84B, and a third storage area 84C. The control unit 82C stores the insertion process frame 40A in chronological order in the second storage area 84B. The insertion process frame 40A refers to a frame 40 (in the example shown in FIG. 7, a frame 40 to which the insertion process tag 116, the first movement distance 112A, and the recognition result 98 are assigned) obtained by imaging an area including the lesion 42 in the large intestine 28 by the camera 52 during the insertion process 41A. In addition, the control unit 82C stores the removal process frame 40B in chronological order in the third storage area 84C. Removal process frame 40B refers to frame 40 (in the example shown in FIG. 7, frame 40 to which removal process tag 118, second movement distance 112B, and recognition result 98 are assigned) obtained by imaging an area including lesion 42 in large intestine 28 by camera 52 during removal process 41B. Here, insertion process frame 40A is an example of an "insertion process frame" according to the present disclosure, and removal process frame 40B is an example of a "removal process frame" according to the present disclosure.

[0122] The control unit 82C acquires the region information 108 from the recognition unit 82A. The control unit 82C then refers to the region information 108 to determine whether the camera 52 has reached the ileocecal region 28A (see FIG. 1). Here, the embodiment in which the region information 108 is referred to to determine whether the camera 52 has reached the ileocecal region 28A is given, but this is merely an example. For example, the region information 108 and a plurality of movement distances 112 along a time series may be referred to to determine whether the camera 52 has reached the ileocecal region 28A. That is, when the ileocecal region 28A is specified by the region information 108 and a plurality of movement distances 112 along a time series (for example, several tens of frames) remain unchanged, it may be determined that the camera 52 has reached the ileocecal region 28A. In addition, when the ileocecal area 28A is identified by the body part information 108 and a plurality of movement distances 112 along the time series are within a predetermined allowable error range (e.g., a range of ± several millimeters) as the range of movement distances 112 for which it can be determined that the camera 52 is stationary, it may be determined that the camera 52 has reached the ileocecal area 28A.

[0123] When the camera 52 has arrived, the control unit 82C acquires the movement distance 112 estimated by the estimation unit 82B (for example, the movement distance 112 on the insertion path 29A (see FIG. 1) from the anus to the ileocecal portion 28A) from the estimation unit 82B as a first reference movement distance 120. The control unit 82C stores the first reference movement distance 120 in the first storage area 84A.

[0124] Figure 8 is a conceptual diagram showing an example of processing performed by the control unit 82C during the removal process 41B based on the first reference movement distance 120 stored in the first storage area 84A, the first movement distance 112A assigned to the insertion process frame 40A stored in the second storage area 84B, and the second movement distance 112B assigned to the removal process frame 40B stored in the third storage area 84C.

[0125] 8, the control unit 82C acquires a first reference movement distance 120 from the first storage area 84A. The control unit 82C also acquires a first movement distance 112A assigned to the insertion process frame 40A from the second storage area 84B. The control unit 82C further acquires a second movement distance 112B assigned to the latest removal process frame 40B from the third storage area 84C. The control unit 82C then performs a first comparison process 122 based on the first reference movement distance 120, the first movement distance 112A, and the second movement distance 112B.

[0126] In the first comparison process 122, a first difference 124 is calculated. The first difference 124 refers to the difference between the first reference movement distance 120 and the first movement distance 112A. The first difference 124 is calculated for each of the first movement distances 112A assigned to all of the insertion process frames 40A stored in the second storage area 84B. In addition, here, a value obtained by subtracting the first movement distance 112A from the first reference movement distance 120 is used as an example of the difference between the first reference movement distance 120 and the first movement distance 112A.

[0127] In the first comparison process 122, a second difference 126 is calculated. The second difference 126 refers to the difference between the second movement distance 112B assigned to the latest removal process frame 40B stored in the third storage area 84C and the first reference movement distance 120. Here, as an example of the difference between the second movement distance 112B and the first reference movement distance 120, a value obtained by subtracting the first reference movement distance 120 from the second movement distance 112B is used.

[0128] 8, a value obtained by subtracting the first reference travel distance 120 from the first travel distance 112A is used as the first difference 124, and a value obtained by subtracting the first reference travel distance 120 from the second travel distance 112B is used as the second difference 126. However, this is merely an example. For example, the first difference 124 may be the absolute value of the difference between the first reference travel distance 120 and the first travel distance 112A, and the second difference 126 may be the absolute value of the difference between the first reference travel distance 120 and the second travel distance 112B. Also, the first difference 124 may be the value obtained by subtracting the first reference travel distance 120 from the first travel distance 112A, and the second difference 126 may be the value obtained by subtracting the second travel distance 112B from the first reference travel distance 120.

[0129] In the first comparison process 122, it is determined whether or not each first difference 124 calculated for each first movement distance 112A assigned to each of all insertion process frames 40A stored in the second storage area 84B matches with the second difference 126. Here, "match" refers to a perfect match, as well as a match that includes an error that is generally acceptable in the technical field to which the present disclosure pertains and does not contradict the spirit of the present disclosure.

[0130] When the controller 82C determines that the first difference 124 and the second difference 126 match, the controller 82C executes a first identification process capable of identifying the identity between the lesion 42 shown in the insertion process frame 40A to which the first movement distance 112A used in calculating the first difference 124 is assigned, and the lesion 42 shown in the removal process frame 40B to which the second movement distance 112B used in calculating the second difference 126 matching the first difference 124 is assigned. In the example shown in FIG. 8, a first display process 128 is illustrated as an example of a process included in the first identification process. The first display process 128 is an example of the "first identification process" and the "first display process" according to the present disclosure.

[0131] The first display process 128 is a process for displaying the insertion process frame 40A and the removal process frame 40B on the screen 35 in a state in which they can be compared when the level of identity between the lesion 42 shown in the insertion process frame 40A and the lesion 42 shown in the removal process frame 40B is equal to or higher than a preset level. Here, the preset level is an example of the "first preset level" and the "second preset level" according to the present disclosure. In the first embodiment, a case in which the level of identity between the lesion 42 shown in the insertion process frame 40A and the lesion 42 shown in the removal process frame 40B is equal to or higher than the preset level refers to a case in which the first difference 124 and the second difference 126 match.

[0132] For example, as shown in FIG. 9, when a first difference 124 corresponding to the distance from the lesion 42 discovered during the insertion process 41A to the ileocecal portion 28A matches a second difference 126 corresponding to the distance from the ileocecal portion 28A to the lesion 42 discovered during the removal process 41B, the control unit 82C determines that the level of identity between the lesion 42 captured in the insertion process frame 40A and the lesion 42 captured in the removal process frame 40B is equal to or higher than a predetermined level.

[0133] 8 and 9, in the first display process 128, the removal process frame 40B to which the second movement distance 112B used in calculating the second difference 126 that coincides with the first difference 124 is assigned is displayed in the first display area 36, ​​and the insertion process frame 40A to which the first movement distance 112A used in calculating the first difference 124 is assigned is displayed in the second display area 38. As a result, the removal process frame 40B displayed in the first display area 36 and the insertion process frame 40A displayed in the second display area 38 can be visually compared. In addition, the lesion 42 can be visually identified from the removal process frame 40B displayed in the first display area 36, ​​and the lesion 42 can be visually identified from the insertion process frame 40A displayed in the second display area 38. This allows for visual comparison of the lesion 42 depicted in the removal process frame 40B displayed in the first display area 36 and the lesion 42 depicted in the insertion process frame 40A displayed in the second display area 38. Here, the removal process frame 40B displayed in the first display area 36 is an example of a "second image" according to the present disclosure, and the insertion process frame 40A displayed in the second display area 36 is an example of a "first image" according to the present disclosure.

[0134] Next, the operation of the portion of the endoscope device 10 according to the present disclosure will be described with reference to Fig. 10A and Fig. 10B. The flow of the medical support process shown in Fig. 10A and Fig. 10B is an example of the "medical support method" according to the present disclosure.

[0135] In the following, it is assumed that the same type of light (e.g., light for BLI and / or light for LCI) is irradiated into the large intestine 28 during the insertion process 41A and the removal process 41B, and the area irradiated with light (e.g., the intestinal wall 32, etc.) is imaged by the camera 52.

[0136] In the medical support process shown in Fig. 10A, first, in step ST10, the recognition unit 82A determines whether or not one frame of images has been captured by the camera 52 inside the large intestine 28. If one frame of images has not been captured by the camera 52 inside the large intestine 28 in step ST10, the determination is negative, and the medical support process proceeds to step ST52 shown in Fig. 10B. If one frame of images has been captured by the camera 52 inside the large intestine 28 in step ST10, the determination is positive, and the medical support process proceeds to step ST12.

[0137] In step ST12, the recognition unit 82A, the estimation unit 82B, and the control unit 82C acquire the frame 40 obtained by imaging the large intestine 28 by the camera 52. Then, the control unit 82C displays the frame 40 in the first display area 36. Note that, if the frame 40 is already displayed in the first display area 36, ​​the control unit 82C updates the frame 40 displayed in the first display area 36. That is, the processing of step ST12 is repeatedly executed, and the frame 40 is displayed in the first display area 36 in a live view format. After the processing of step ST12 is executed, the medical support processing proceeds to step ST14.

[0138] In step ST14, the recognition unit 82A executes recognition processing 96 on the frame 40 acquired in step ST12. After the processing in step ST14 is executed, the medical support processing proceeds to step ST16.

[0139] In step ST16, the estimation unit 82B determines whether or not multiple images have been captured by the camera 52. In step ST16, if multiple images have not been captured by the camera 52, the determination is negative, and the medical support process proceeds to step ST10. In step ST16, if multiple images have been captured by the camera 52, the determination is positive, and the medical support process proceeds to step ST18.

[0140] In step ST18, the estimation unit 82B inputs a plurality of frames 40 along a time series to the movement estimation model 94, thereby causing the movement estimation model 94 to estimate the movement distance 112 and the movement direction 114. After the processing of step ST18 is executed, the medical support processing proceeds to step ST20.

[0141] In step ST20, the recognition unit 82A refers to lesion presence / absence information 100 included in the recognition result 98 obtained by executing the recognition process 96 in step ST14, and determines whether or not the frame 40 acquired in step ST12 includes a lesion 42. In step ST20, if the frame 40 acquired in step ST12 does not include a lesion 42, the determination is negative, and the medical support process proceeds to step ST30. In step ST20, if the frame 40 acquired in step ST12 includes a lesion 42, the determination is positive, and the medical support process proceeds to step ST22.

[0142] In step ST22, the control unit 82C generates an insertion process tag 116 or a removal process tag 118 based on the moving direction 114 estimated by the estimation unit 82B in step ST18. That is, if the moving direction 114 is the insertion direction, the insertion process tag 116 is generated, and if the moving direction 114 is the removal direction, the removal process tag 118 is generated. After the processing of step ST22 is executed, the medical support processing proceeds to step ST24.

[0143] In step ST24, the control unit 82C assigns the insertion process tag 116 or the removal process tag 118 generated in step ST22, the movement distance 112 estimated in step ST18, and the recognition result 98 obtained by executing the recognition process 96 in step ST14 to the frame 40 acquired in step ST12. For example, if the movement direction 114 estimated in step ST18 is the insertion direction, the control unit 82C assigns the insertion process tag 116 to the frame 40 acquired in step ST12, i.e., the insertion process frame 40A, assigns the movement distance 112 estimated in step ST18 as the first movement distance 112A, and assigns the recognition result 98. Furthermore, for example, if the movement direction 114 estimated in step ST18 is the removal direction, the control unit 82C assigns a removal process tag 118 to the frame 40 acquired in step ST12, i.e., the removal process frame 40B, assigns the movement distance 112 estimated in step ST18 as a second movement distance 112B, and assigns the recognition result 98. After the process of step ST24 is executed, the medical support process proceeds to step ST26.

[0144] In step ST26, the control unit 82C determines whether or not the insertion process tag 116 has been assigned to the frame 40. If the insertion process tag 116 has not been assigned to the frame 40 in step ST26 (i.e., the removal process tag 118 has been assigned to the frame 40), the determination is negative, and the medical support process proceeds to step ST34 shown in Fig. 10B. If the insertion process tag 116 has been assigned to the frame 40 in step ST26, the determination is positive, and the medical support process proceeds to step ST28.

[0145] In step ST28, the control unit 82C chronologically stores the insertion process frame 40A to which the insertion process tag 116 has been assigned, the movement distance 112 has been assigned as the first movement distance 112A, and the recognition result 98 has been assigned, in the second storage area 84B. After the process of step ST28 is executed, the medical support process proceeds to step ST30.

[0146] In step ST30, the control unit 82C refers to the body part information 108 included in the recognition result 98 obtained by executing the recognition process 96 in step ST14, and determines whether or not the camera 52 has reached the ileocecal region 28A. In step ST30, if the camera 52 has not reached the ileocecal region 28A, the determination is negative, and the medical support process proceeds to step ST10. In step ST30, if the camera 52 has reached the ileocecal region 28A, the determination is positive, and the medical support process proceeds to step ST32.

[0147] In step ST32, the control unit 82C stores the travel distance 112 estimated in step ST18 in the first storage area 84A as a first reference travel distance 120. After the process of step ST32 is executed, the medical support process proceeds to step ST34 shown in FIG. 10B.

[0148] 10B, the control unit 82C chronologically stores the removal process frame 40B to which the removal process tag 118 has been assigned, the movement distance 112 has been assigned as the second movement distance 112B, and the recognition result 98 has been assigned, in the third storage area 84C. After the process of step ST34 is executed, the medical support process proceeds to step ST36.

[0149] In step ST36, the control unit 82C acquires the second movement distance 112B from the latest removal process frame 40B stored in the third storage area 84C. After the process of step ST34 is executed, the medical support process proceeds to step ST38.

[0150] In step ST38, the control unit 82C acquires the first reference movement distance 120 from the first storage area 84A. After the process of step ST38 is executed, the medical support process proceeds to step ST40.

[0151] In step ST40, the control unit 82C calculates the difference between the second movement distance 112B acquired in step ST38 and the first reference movement distance 120 acquired in step ST38, that is, the second difference 126. After the processing of step ST40 is executed, the medical support processing proceeds to step ST42.

[0152] In step ST42, the control unit 82C acquires the first moving distance 112A that is unused in this step ST42. After the process of step ST42 is executed, the medical support process proceeds to step ST44.

[0153] In step ST44, the control unit 82C calculates the difference between the first moving distance 112A acquired in step ST42 and the first reference moving distance 120 acquired in step ST38, that is, the first difference 124. After the processing of step ST44 is executed, the medical support processing proceeds to step ST46.

[0154] In step ST46, the control unit 82C determines whether the first difference 124 calculated in step ST44 and the second difference 126 calculated in step ST40 match. If the first difference 124 and the second difference 126 do not match in step ST46, the determination is negative, and the medical support process proceeds to step ST42. If the first difference 124 and the second difference 126 match in step ST46, the determination is positive, and the medical support process proceeds to step ST48.

[0155] In step ST48, the control unit 82C acquires from the second storage area 84B the insertion process frame 40A to which the first movement distance 112A used in calculating the first difference 124 that coincides with the second difference 126 has been assigned. After the processing of step ST48 is executed, the medical support processing proceeds to step ST50.

[0156] When the control unit 82C acquires the insertion process frame 40A from the second storage area 84B, the control unit 82C may erase the insertion process frame 40A acquired from the second storage area 84B from the second storage area 84B. When all of the insertion process frames 40A stored in the second storage area 84B are erased from the second storage area 84B, the control unit 82C may notify that all of the insertion process frames 40A stored in the second storage area 84B have been erased from the second storage area 84B. Examples of the notification in this case include displaying information (e.g., text and / or marks, etc.) on the screen 35 that can identify that all of the insertion process frames 40A stored in the second storage area 84B have been erased from the second storage area 84B, or outputting a sound from a speaker (not shown) that can identify that all of the insertion process frames 40A stored in the second storage area 84B have been erased from the second storage area 84B.

[0157] In step ST50, the control unit 82C displays the insertion process frame 40A acquired from the second storage area 84B in the second display area 38. If the insertion process frame 40A is already displayed in the second display area 38, the control unit 82C updates the insertion process frame 40A displayed in the second display area 38 to the latest insertion process frame 40A acquired from the second storage area 84B. After the processing of step ST50 is executed, the medical support processing proceeds to step ST52.

[0158] In step ST52, the control unit 82C determines whether or not a condition for terminating the medical support process is satisfied. One example of the condition for terminating the medical support process is that an instruction to terminate the medical support process is given to the endoscope device 10 (for example, that an instruction to terminate the medical support process is accepted by the acceptance device 64).

[0159] In step ST52, if the condition for terminating the medical support process is not satisfied, the determination is negative, and the medical support process proceeds to step ST10. In step ST52, if the condition for terminating the medical support process is satisfied, the determination is positive, and the medical support process ends.

[0160] As described above, in the endoscope device 10 according to the first embodiment, the first identification process is executed based on the first movement distance 112A obtained in the insertion process 41A and the second movement distance 112B obtained in the removal process 41B. In the first embodiment, when the first difference 124 and the second difference 126 match, the first identification process is executed. The first difference 124 is the difference between the first reference movement distance 120 (i.e., the movement distance 112 on the insertion path 29A (see FIG. 1) from the anus to the ileocecal portion 28A) and the first movement distance 112A, and the second difference 126 is the difference between the first reference movement distance 120 and the second movement distance 112B. The first identification process is a process capable of identifying the identity of the lesion 42 captured in the insertion process frame 40A and the lesion 42 captured in the removal process frame 40B. Therefore, with the endoscope device 10 according to the first embodiment, the doctor 12 can identify whether the same lesion 42 has been imaged by the camera 52 during the insertion process 41A and the removal process 41B.

[0161] In the endoscope device 10 according to the first embodiment, when the lesion 42 is shown in the removal process frame 40B during the removal process 41B, the control unit 82C determines whether the first difference 124 and the second difference 126 match. The first difference 124 corresponds to the distance from the lesion 42 found during the insertion process 41A to the ileocecal portion 28A, and the second difference 126 corresponds to the distance from the ileocecal portion 28A to the lesion 42 found during the removal process 41B. When the first difference 124 and the second difference 126 match, the control unit 82C determines that the level of identity between the lesion 42 shown in the insertion process frame 40A and the lesion 42 shown in the removal process frame 40B is equal to or higher than a preset level. When the control unit 82C determines that the level of identity between the lesion 42 shown in the insertion process frame 40A and the lesion 42 shown in the removal process frame 40B is equal to or higher than a preset level, the control unit 82C executes a first display process 128. In the first display process 128, the removal process frame 40B to which the second movement distance 112B used in calculating the second difference 126, which matches the first difference 124, has been assigned is displayed in the first display area 36, ​​and the insertion process frame 40A to which the first movement distance 112A used in calculating the first difference 124 has been assigned is displayed in the second display area 38.

[0162] Therefore, according to the endoscopic device 10 of the first embodiment, when the control unit 82C determines that the level of identity between the lesion 42 depicted in the insertion process frame 40A and the lesion 42 depicted in the removal process frame 40B is above a predetermined level, the doctor 12 can visually grasp the appearance of the lesion 42 depicted in the removal process frame 40B displayed in the first display area 36 and the appearance of the lesion 42 depicted in the insertion process frame 40A displayed in the second display area 38.

[0163] Furthermore, in the endoscope device 10 according to the first embodiment, an insertion process tag 116 is assigned to the insertion process frame 40A, and a removal process tag 118 is assigned to the removal process frame 40B. The insertion process tag 116 is a tag that can identify that the insertion process frame 40A is a frame 40 obtained by capturing an image of an area including the lesion 42 by the camera 52 during the insertion process 41A. The removal process tag 118 is a tag that can identify that the removal process frame 40B is a frame 40 obtained by capturing an image of an area including the lesion 42 by the camera 52 during the removal process 41B.

[0164] Therefore, according to the endoscope device 10 according to the first embodiment, it is possible to easily distinguish between the insertion process frame 40A, which is the frame 40 obtained by imaging the area including the lesion 42 by the camera 52 during the insertion process 41A, and the removal process frame 40B, which is the frame 40 obtained by imaging the area including the lesion 42 by the camera 52 during the removal process 41B. As a result, processing using the insertion process frame 40A and the removal process frame 40B can be easily performed.

[0165] Furthermore, in the endoscope device 10 according to the first embodiment, the position of the camera 52 within the large intestine 28 is defined based on the movement distance 112. The movement distance 112 is estimated by the movement estimation model 94 by inputting a plurality of frames 40 in a time series to the movement estimation model 94. Therefore, according to the endoscope device 10 according to the first embodiment, it is possible to estimate the position of the camera 52 within the large intestine 28 without using a sensor specialized for detecting the position of the camera 52 within the large intestine 28.

[0166] In the first embodiment, the insertion process frame 40A is displayed in the second display area 38 on the condition that the first difference 124 and the second difference 126 match, but the present disclosure is not limited thereto. For example, the insertion process frame 40A may be displayed in the second display area 38 on the condition that the insertion process frame 40A to which the first movement distance 112A used in calculating the first difference 124 that matches the second difference 126 is added is continuously obtained for a first preset frame number or more (e.g., a plurality of frames designated in advance). In this case, the insertion process frame 40A displayed in the second display area 38 may be the insertion process frame 40A obtained first in time, the insertion process frame 40A obtained last in time, or the insertion process frame 40A obtained between the insertion process frame 40A obtained first in time and the insertion process frame 40A obtained last in time. Furthermore, among the multiple insertion process frames 40A continuously obtained for more than the first preset frame number, the insertion process frame 40A in which the position in which the lesion 42 is captured is closest to the center of the angle of view of the camera 52 may be displayed in the second display area 38. Furthermore, all of the multiple insertion process frames 40A continuously obtained for more than the first preset frame number, or multiple insertion process frames 40A selected according to some rule, may be displayed in the second display area 38 in a comparable state.

[0167] An example of the flow of the medical support process for realizing this embodiment is shown in FIG. 11. The medical support process shown in FIG. 11 is different from the medical support process shown in FIG. 10B in that it has a process of step ST100 between the process of step ST46 and the process of step ST48. In step ST100 shown in FIG. 11, the control unit 82C judges whether or not the insertion process frame 40A to which the first movement distance 112A used in the calculation of the first difference 124 that matches the second difference 126 is given continues for a first preset frame number or more. In step ST100, if the insertion process frame 40A to which the first movement distance 112A used in the calculation of the first difference 124 that matches the second difference 126 is given does not continue for a first preset frame number or more, the judgment is denied and the medical support process proceeds to step ST52. In step ST100, if the insertion process frame 40A to which the first movement distance 112A used to calculate the first difference 124 that matches the second difference 126 has been assigned continues for more than the first specified number of frames, the judgment is positive and the medical support processing proceeds to step ST48.

[0168] In this way, by displaying the insertion process frame 40A in the second display area 38 on the condition that the insertion process frame 40A, which has the first movement distance 112A used to calculate the first difference 124 that matches the second difference 126, is obtained continuously for more than the first preset number of frames, the lesion 42 that is likely to have been observed by the doctor 12 during the insertion process 41A (in other words, the lesion 42 recognized by the recognition unit 82A) can be observed by the doctor 12 during the removal process 41B.

[0169] In the first embodiment, an example in which the first display process 128 is executed by the control unit 82C has been given, but this is merely one example, and for example, as shown in Fig. 12, the first notification process 129 may be executed by the control unit 82C. The first notification process 129 is a process for notifying that the level of identity between the lesion 42 captured in the insertion process frame 40A and the lesion 42 captured in the removal process frame 40B is equal to or higher than a preset level when the level of identity between the lesion 42 captured in the insertion process frame 40A and the lesion 42 captured in the removal process frame 40B is equal to or higher than a preset level.

[0170] As an example of the notification, a message 130 notifying that the level of identity between the lesion 42 captured in the insertion process frame 40A and the lesion 42 captured in the removal process frame 40B is equal to or higher than a preset level is displayed in the second display area 38 as one of the auxiliary information 44. Also, a mark or a numerical value, etc., notifying that the level of identity between the lesion 42 captured in the insertion process frame 40A and the lesion 42 captured in the removal process frame 40B is equal to or higher than a preset level may be displayed in the second display area 38 as one of the auxiliary information 44. Also, similar to the first embodiment, the insertion process frame 40A may be displayed in the second display area 38. Also, the information notifying that the level of identity between the lesion 42 captured in the insertion process frame 40A and the lesion 42 captured in the removal process frame 40B is equal to or higher than a preset level may be displayed in an area other than the second display area 38 in the screen 35, or may be displayed on the screen of a display device different from the display device 18. In addition, a sound may be output from a speaker (not shown) to notify that the level of identity between the lesion 42 shown in the insertion process frame 40A and the lesion 42 shown in the removal process frame 40B is equal to or higher than a predetermined level.

[0171] In this way, the first notification process 129 is executed by the control unit 82C, thereby allowing the doctor 12 to perceive that the level of identity between the lesion 42 shown in the insertion process frame 40A and the lesion 42 shown in the removal process frame 40B is at or above a predetermined level.

[0172] [Second embodiment] In the first embodiment, when the removal process frame 40B is obtained in the removal process 41B, a process (for example, the first display process 128) is executed to identify the lesion 42 shown in the removal process frame 40B and the lesion 42 shown in the insertion process frame 40A obtained in the insertion process 41A. In the second embodiment, a process is executed to identify the lesion 42 shown in the frame 40 obtained in the process of the camera 52 being pushed into the ileocecal portion 28A side (i.e., the back side) of the large intestine 28 and the lesion 42 shown in the frame 40 obtained in the process of the camera 2 being pulled from the inside of the large intestine 28 to the anus side (i.e., the near side). Here, the "near side" is synonymous with the entrance side of the large intestine 28.

[0173] In the second embodiment, the same components as those in the first embodiment are denoted by the same reference numerals and will not be described. The above description of Fig. 11 and Fig. 12 (for example, the embodiment shown in Fig. 12 and Fig. 13, and modified examples of the embodiment shown in Fig. 12 and Fig. 13) is also applicable to the second embodiment.

[0174] FIG. 13 is a conceptual diagram showing an example of a transition from a pushing process 41C, in which camera 52 is pushed into the back of large intestine 28 (e.g., toward the ileocecal portion 28A), to a retracting process 41D, in which camera 52 is retracted from inside large intestine 28 to the front (e.g., toward the anus).

[0175] 13, the pushing process 41C is a process in which the camera 52 moves from a pushing start position P1, which is the start position of the pushing process 41C, to a direction change position P2A, which is the start position of the pulling process 41D (in other words, the end position of the pushing process 41C). In the example shown in FIG 13, the pulling process 41D is a process in which the camera 52 moves from the direction change position P2A to a pulling end position P3, which is the end position of the pulling process 41D.

[0176] The direction change position P2A is a position where the camera 52 stops. For example, the direction change position P2A is recognized based on a plurality of movement distances 112 obtained by estimation by the control unit 82C for each frame 40. That is, when the difference between the plurality of time-series movement distances 112 estimated across the plurality of frames 40 is within a predetermined allowable error, the control unit 82C determines that the camera 52 has reached the direction change position P2A.

[0177] In the second embodiment, the direction in which camera 52 moves in pushing process 41C is the same as the direction in which camera 52 moves in insertion process 41A described in the first embodiment, and the direction in which camera 52 moves in drawing process 41D is the same as the direction in which camera 52 moves in removal process 41B described in the first embodiment. The drawing end position P3 when transitioning from pushing process 41C to drawing process 41D in insertion process 41A is the position where camera 52 stops, and is recognized in the same manner as the direction change position P2A is recognized.

[0178] In the second embodiment, the stroke of camera 52 moving in pushing process 41C is shorter than the stroke of camera 52 moving in insertion process 41A (for example, insertion path 29A shown in FIG. 1). In the second embodiment, the stroke of camera 52 moving in drawing process 41D is shorter than the stroke of camera 52 moving in removal process 41B (for example, removal path 29B shown in FIG. 1).

[0179] FIG. 14 is a conceptual diagram showing an example of a transition from the pulling process 41D to the pushing process 41C.

[0180] 14, the approaching process 41D is a process in which the camera 52 moves from an approaching start position P4, which is the start position of the approaching process 41D, to a direction changing position P2B, which is the start position of the pushing process 41C (in other words, the end position of the approaching process 41D). Also, in the example shown in FIG 14, the pushing process 41C is a process in which the camera 52 moves from the direction changing position P2B to a pushing end position P5, which is the end position of the pushing process 41C.

[0181] The direction change position P2B is a position where the camera 52 stops. For example, the direction change position P2B is recognized based on a plurality of movement distances 112 obtained by estimating each frame 40 by the control unit 82C. That is, when the difference between the plurality of time-series movement distances 112 estimated over the plurality of frames 40 is within a predetermined allowable error, the control unit 82C determines that the camera 52 has reached the direction change position P2B. The pushing end position P5 when the pulling process 41D transitions to the pushing process 41C in the removal process 41B is a position where the camera 52 stops, and is recognized in the same manner as the direction change position P2B is recognized.

[0182] Hereinafter, for convenience of explanation, when there is no need to distinguish between the direction change positions P2A and P2B, they will be referred to as "direction change position P2."

[0183] FIG. 15 is a conceptual diagram showing an example of the processing contents of the estimation unit 82B and the control unit 82C in the pushing process 41C and the pulling process 41D.

[0184] As shown in Fig. 15, the control unit 82C acquires the recognition result 98 from the recognition unit 82A, and judges whether or not the lesion 42 is captured in the frame 40 input to the recognition model 92 by referring to the lesion presence / absence information 100 included in the acquired recognition result 98. Here, when the lesion 42 is captured in the frame 40 input to the recognition model 92 to obtain the recognition result 98 acquired by the control unit 82C from the recognition unit 82A, the control unit 82C generates a pushing process tag 132 or a pulling process tag 134 based on the moving direction 114 input from the estimation unit 82B. The pushing process tag 132 refers to a tag that is an identifier that can identify the frame 40 obtained by imaging an area including the lesion 42 by the camera 52 during the pushing process 41C (i.e., an area including the lesion 42 that is a part of the entire area of ​​the intestinal wall 32 that falls within the angle of view of the camera 52). The attraction process tag 134 refers to a tag that is an identifier that can identify the frame 40 as being obtained by imaging an area including the lesion 42 by the camera 52 during the attraction process 41D (i.e., a portion of the entire area of ​​the intestinal wall 32 that falls within the angle of view of the camera 52 and that includes the lesion 42).

[0185] When the control unit 82C generates the push-in process tag 132 or the attraction process tag 134 in this manner, first, the control unit 82C determines whether the moving direction 114 input from the estimation unit 82B is the push-in direction or the attraction direction. Then, the control unit 82C generates the push-in process tag 132 if the moving direction 114 input from the estimation unit 82B is the push-in direction, and generates the attraction process tag 134 if the moving direction 114 input from the estimation unit 82B is the attraction direction. Note that in the second embodiment, the push-in process tag 132 is an example of a "pushing process identifier" according to the present disclosure, and the attraction process tag 134 is an example of a "pushing process identifier" according to the present disclosure.

[0186] The control unit 82C assigns the pushing process tag 132 or the attracting process tag 134, the movement distance 112, and the recognition result 98 to the frame 40 acquired from the camera 52 and showing the lesion 42. The frame 40 to which the pushing process tag 132 or the attracting process tag 134, the movement distance 112, and the recognition result 98 are assigned is the same frame 40 as the latest frame 40 input to the movement estimation model 94 to estimate the movement direction 114 used to generate the pushing process tag 132 or the attracting process tag 134. The movement distance 112 assigned to the frame 40 is the movement distance 112 estimated by the movement estimation model 94 together with the movement direction 114 used to generate the pushing process tag 132 or the attracting process tag 134. The recognition result 98 assigned to the frame 40 is the recognition result 98 used by the control unit 82C to determine the presence or absence of the lesion 42.

[0187] The movement distance 112 estimated by the movement estimation model 94 together with the movement direction 114 used in generating the pushing process tag 132 is assigned to the frame 40 as a third movement distance 112C. The movement distance 112 estimated by the movement estimation model 94 together with the movement direction 114 used in generating the pulling process tag 134 is assigned to the frame 40 as a fourth movement distance 112D.

[0188] Here, the third movement distance 112C is an example of the "pushed-in position specifying information" according to the present disclosure, and the fourth movement distance 112D is an example of the "pulled-over position specifying information" according to the present disclosure.

[0189] FIG. 16 is a conceptual diagram showing an example of a manner in which various information is stored in the memory 84 by the control unit 82C. As shown in FIG. 16, the memory 84 has a fourth storage area 84D, a fifth storage area 84E, and a sixth storage area 84F. The control unit 82C stores the pushing process frame 40C in the fifth storage area 84E in chronological order. The pushing process frame 40C refers to a frame 40 (in the example shown in FIG. 16, a frame 40 to which the pushing process tag 132, the third moving distance 112C, and the recognition result 98 are attached) obtained by imaging an area including the lesion 42 in the large intestine 28 by the camera 52 during the pushing process 41C. In addition, the control unit 82C stores the pulling process frame 40D in chronological order in the sixth storage area 84F. The approaching process frame 40D refers to a frame 40 (in the example shown in FIG. 16, a frame 40 to which an approaching process tag 134, a fourth moving distance 112D, and a recognition result 98 are assigned) obtained by imaging a region including the lesion 42 in the large intestine 28 by the camera 52 during the approaching process 41D. Here, the pushing process frame 40C is an example of a "pushing process frame" according to the present disclosure, and the approaching process frame 40D is an example of a "approaching process frame" according to the present disclosure.

[0190] The control unit 82C acquires a plurality of movement distances 112 from the estimation unit 82B, and determines whether or not the camera 52 has reached the direction change position P2 based on the plurality of movement distances 112 (e.g., a plurality of movement distances 112 continuously acquired in time series). For example, if the difference between the plurality of movement distances 112 acquired from the estimation unit 82B is within a predetermined allowable error, the control unit 82C determines that the camera 52 has reached the direction change position P2.

[0191] When the camera 52 reaches the direction change position P2, the control unit 82C acquires the movement distance 112 estimated by the estimation unit 82B from the estimation unit 82B as the second reference movement distance 136. The control unit 82C stores the second reference movement distance 136 in the fourth storage area 84D.

[0192] Figure 17 is a conceptual diagram showing an example of processing performed by the control unit 82C based on the second reference movement distance 136 stored in the fourth storage area 84D, the third movement distance 112C assigned to the pushing process frame 40C stored in the fifth storage area 84E, and the fourth movement distance 112D assigned to the pulling process frame 40D stored in the sixth storage area 84F when transitioning from the pushing process 41C to the pulling process 41D within the insertion process 41A or the removal process 41B.

[0193] 17, the control unit 82C acquires the second reference movement distance 136 from the fourth storage area 84D. The control unit 82C also acquires the third movement distance 112C assigned to the push-in process frame 40C from the fifth storage area 84E. Furthermore, the control unit 82C acquires the fourth movement distance 112D assigned to the latest pulling process frame 40D from the sixth storage area 84F. Then, the control unit 82C performs a second comparison process 142 based on the second reference movement distance 136, the third movement distance 112C, and the fourth movement distance 112D.

[0194] In the second comparison process 142, a third difference 144 is calculated. The third difference 144 refers to the difference between the second reference movement distance 136 and the third movement distance 112C. The third difference 144 is calculated for each of the third movement distances 112C assigned to all the pushing process frames 40C stored in the fifth storage area 84E. In addition, here, as an example of the difference between the second reference movement distance 136 and the third movement distance 112C, a value obtained by subtracting the third movement distance 112C from the second reference movement distance 136 is used.

[0195] In the second comparison process 142, a fourth difference 146 is calculated. The fourth difference 146 refers to the difference between the fourth movement distance 112D assigned to the latest approaching process frame 40D stored in the sixth storage area 84F and the second reference movement distance 136. Here, as an example of the difference between the fourth movement distance 112D and the second reference movement distance 136, a value obtained by subtracting the second reference movement distance 136 from the fourth movement distance 112D is used.

[0196] 17, a value obtained by subtracting the third movement distance 112C from the second reference movement distance 136 is used as the third difference 144, and a value obtained by subtracting the second reference movement distance 136 from the fourth movement distance 112D is used as the fourth difference 146. However, this is merely an example. For example, the third difference 144 may be the absolute value of the difference between the second reference movement distance 136 and the third movement distance 112C, and the fourth difference 146 may be the absolute value of the difference between the second reference movement distance 136 and the fourth movement distance 112D. Also, the third difference 144 may be a value obtained by subtracting the second reference movement distance 136 from the third movement distance 112C, and the fourth difference 146 may be a value obtained by subtracting the fourth movement distance 112D from the second reference movement distance 136.

[0197] In the second comparison process 142, it is determined whether or not each of the third differences 144 and the fourth difference 146 calculated for each of the third movement distances 112C assigned to all of the push-in process frames 40C stored in the fifth storage area 84E matches. Here, the match refers to a perfect match, as well as a match that includes an error that is generally acceptable in the technical field to which the present disclosure pertains and does not contradict the spirit of the present disclosure.

[0198] When the control unit 82C determines that the third difference 144 and the fourth difference 146 match, the control unit 82C executes a second identification process capable of identifying the identity between the lesion 42 shown in the pushing process frame 40C to which the third movement distance 112C used in calculating the third difference 144 is assigned, and the lesion 42 shown in the retraction process frame 40D to which the fourth movement distance 112D used in calculating the fourth difference 146 that matches the third difference 144 is assigned. In the example shown in FIG. 17, a second display process 148 is illustrated as an example of the process included in the second identification process. The second display process 148 is an example of the "second identification process" and the "second display process" according to the present disclosure.

[0199] The second display process 148 is a process for displaying the pushing process frame 40C and the approaching process frame 40D on the screen 35 in a state in which they can be compared when the level of identity between the lesion 42 captured in the pushing process frame 40C and the lesion 42 captured in the approaching process frame 40D is equal to or higher than a preset level. Here, the preset level is an example of the "third preset level" and the "fourth preset level" according to the present disclosure. In the second embodiment, a case in which the level of identity between the lesion 42 captured in the pushing process frame 40C and the lesion 42 captured in the approaching process frame 40D is equal to or higher than the preset level refers to a case in which the third difference 144 and the fourth difference 146 are the same.

[0200] For example, as shown in FIG. 18, when a third difference 144 corresponding to the distance from the lesion 42 discovered during the pushing process 41C to the direction change position P2 (in the example shown in FIG. 18, the direction change position P2A) matches a fourth difference 146 corresponding to the distance from the direction change position P2 to the lesion 42 discovered during the retraction process 41D, the control unit 82C determines that the level of identity between the lesion 42 captured in the pushing process frame 40C and the lesion 42 captured in the retraction process frame 40D is equal to or higher than a predetermined level.

[0201] 17 and 18, in the second display process 148, the approaching process frame 40D to which the fourth movement distance 112D used in calculating the fourth difference 146 that coincides with the third difference 144 is added is displayed in the first display area 36, ​​and the pushing process frame 40C to which the third movement distance 112C used in calculating the third difference 144 is added is displayed in the second display area 38. As a result, the approaching process frame 40D displayed in the first display area 36 and the pushing process frame 40C displayed in the second display area 38 can be visually compared. In addition, the lesion 42 can be visually identified from the approaching process frame 40D displayed in the first display area 36, ​​and the lesion 42 can be visually identified from the pushing process frame 40C displayed in the second display area 38. This allows visual comparison between the lesion 42 depicted in the approaching process frame 40D displayed in the first display area 36 and the lesion 42 depicted in the pushing-in process frame 40C displayed in the second display area 38. Here, the approaching process frame 40D displayed in the first display area 36 is an example of a "fourth image" according to the present disclosure, and the pushing-in process frame 40C displayed in the second display area 38 is an example of a "third image" according to the present disclosure.

[0202] 17 and 18, a case where the pushing process 41C transitions to the drawing process 41D (i.e., the pushing process 41C precedes the drawing process 41D) has been described as an example, but a similar process is performed when the drawing process 41D transitions to the pushing process 41C (i.e., the drawing process 41D precedes the pushing process 41C). In this case, for example, the pushing process frame 40C is displayed in the first display area 36, ​​and the drawing process frame 40D is displayed in the second display area 38. This allows the lesion 42 shown in the pushing process frame 40C displayed in the first display area 36 and the lesion 42 shown in the drawing process frame 40D displayed in the second display area 38 to be visually compared.

[0203] Next, the operation of the portion of the endoscope device 10 according to the second embodiment that is related to the present disclosure will be described with reference to FIGS. 19A, 19B, 20A, and 20B.

[0204] 19A, 19B, 20A, and 20B show an example of the flow of the medical support process according to the second embodiment. The medical support process according to the second embodiment is classified into a first medical support process and a second medical support process. FIG. 19A and FIG. 19B show an example of the flow of the first medical support process. FIG. 20A and FIG. 20B show an example of the flow of the second medical support process. The first medical support process and the second medical support process may be selectively executed by the processor 82, or may be executed in parallel by the processor 82. The flow of the medical support process according to the second embodiment shown in FIG. 19A, FIG. 19B, FIG. 20A, and FIG. 20B is an example of the "medical support method" according to the present disclosure.

[0205] In the second embodiment, the processor 82 may execute the medical support process according to the second embodiment in parallel with the medical support process according to the first embodiment, or may execute the medical support process according to the second embodiment without executing the medical support process according to the first embodiment. When the medical support process according to the second embodiment is executed in parallel with the medical support process according to the first embodiment, for example, a screen on which the processing result of the medical support process according to the first embodiment is displayed and a screen on which the processing result of the medical support process according to the second embodiment is displayed may be provided separately, or the processing result of the medical support process according to the first embodiment and the processing result of the medical support process according to the second embodiment may be displayed in a distinguishable manner in a plurality of display areas provided in one screen (for example, screen 35).

[0206] Here, the first medical support process will be described with reference to Fig. 19A and Fig. 19B. The first medical support process is an example of a process assuming a case where a transition is made from a pushing-in process 41C to a pulling-in process 41D (see Fig. 13) in an insertion process 41A or a removal process 41B, and is executed by a processor 82.

[0207] The flowchart shown in Figures 19A and 19B differs from the flowchart shown in Figures 10A and 10B in that it has steps ST200 to ST228 instead of steps ST22 to ST50. Note that, in the flowchart shown in Figures 19A and 19B, the same steps as those included in the flowchart shown in Figures 10A and 10B are denoted by the same step numbers as those shown in Figures 10A and 10B below, and descriptions thereof will be omitted.

[0208] In the following, it is assumed that the same type of light (e.g., light for BLI and / or light for LCI) is irradiated into the large intestine 28 during the pushing process 41C and the pulling process 41D, and the area irradiated with the light (e.g., the intestinal wall 32, etc.) is imaged by the camera 52.

[0209] 19A, in step ST200, the control unit 82C generates the push-in process tag 132 or the pull-in process tag 134 based on the moving direction 114 estimated by the estimation unit 82B in step ST18. That is, if the moving direction 114 is the push-in direction, the push-in process tag 132 is generated, and if the moving direction 114 is the pull-in direction, the pull-in process tag 134 is generated. After the process of step ST200 is executed, the first medical support process proceeds to step ST204.

[0210] In step ST202, the control unit 82C assigns the push-in process tag 132 or the pull-in process tag 134 generated in step ST200, the moving distance 112 estimated in step ST18, and the recognition result 98 obtained by executing the recognition process 96 in step ST14 to the frame 40 acquired in step ST12. For example, if the moving direction 114 estimated in step ST18 is the pushing direction, the control unit 82C assigns the push-in process tag 132 to the frame 40 acquired in step ST12, i.e., the push-in process frame 40C, assigns the moving distance 112 estimated in step ST18 as the third moving distance 112C, and assigns the recognition result 98. Furthermore, for example, if the moving direction 114 estimated in step ST18 is the approaching direction, the control unit 82C assigns an approaching process tag 134 to the frame 40 acquired in step ST12, i.e., the approaching process frame 40D, assigns the moving distance 112 estimated in step ST18 as a fourth moving distance 112D, and assigns the recognition result 98. After the process of step ST202 is executed, the first medical support process proceeds to step ST204.

[0211] In step ST204, the control unit 82C judges whether or not the pushing process tag 132 has been assigned to the frame 40. In step ST204, if the pushing process tag 132 has not been assigned to the frame 40 (i.e., if the pulling process tag 134 has been assigned to the frame 40), the judgment is negative, and the first medical support process proceeds to step ST212 shown in Fig. 19B. In step ST204, if the pushing process tag 132 has been assigned to the frame 40, the judgment is positive, and the first medical support process proceeds to step ST206.

[0212] In step ST206, the control unit 82C chronologically stores the pressing process frame 40C to which the pressing process tag 132 has been assigned, the movement distance 112 has been assigned as the third movement distance 112C, and the recognition result 98 has been assigned, in the fifth storage area 84E. After the process of step ST206 is executed, the first medical support process proceeds to step ST208.

[0213] In step ST208, the control unit 82C refers to the movement distance 112 for multiple frames estimated in step ST18 (i.e., the movement distance 112 for multiple frames along the time series) to determine whether or not the camera 52 has reached the direction change position P2A (see FIG. 13). In step ST208, if the camera 52 has not reached the direction change position P2A, the determination is negative, and the first medical support process proceeds to step ST10. In step ST208, if the camera 52 has reached the direction change position P2A, the determination is positive, and the first medical support process proceeds to step ST210.

[0214] In step ST210, the control unit 82C stores the travel distance 112 estimated in step ST18 in the fourth storage area 84D as the second reference travel distance 136. After the process of step ST210 is executed, the first medical support process proceeds to step ST212 shown in FIG. 19B.

[0215] 19B, the control unit 82C chronologically stores the approaching process frame 40D to which the approaching process tag 134 is assigned, the moving distance 112 is assigned as the fourth moving distance 112D, and the recognition result 98 is assigned, in the sixth storage area 84F. After the process of step ST212 is executed, the first medical support process proceeds to step ST214.

[0216] In step ST214, the control unit 82C acquires the fourth movement distance 112D from the latest retraction process frame 40D stored in the sixth storage area 84F. After the process of step ST214 is executed, the first medical support process proceeds to step ST216.

[0217] In step ST216, the control unit 82C acquires the second reference movement distance 136 from the fourth storage area 84D. After the process of step ST216 is executed, the first medical support process proceeds to step ST218.

[0218] In step ST218, the control unit 82C calculates the difference between the fourth movement distance 112D acquired in step 214 and the second reference movement distance 136 acquired in step ST216, that is, the fourth difference 146. After the process of step ST218 is executed, the first medical support process proceeds to step ST220.

[0219] In step ST220, the control unit 82C acquires the third moving distance 112C that is unused in this step ST220. After the process of step ST220 is executed, the first medical support process proceeds to step ST222.

[0220] In step ST222, the control unit 82C calculates the difference between the third movement distance 112C acquired in step ST220 and the second reference movement distance 136 acquired in step ST216, that is, the third difference 144. After the process of step ST222 is executed, the first medical support process proceeds to step ST224.

[0221] In step ST224, the control unit 82C determines whether or not the third difference 144 calculated in step ST222 and the fourth difference 146 calculated in step ST218 match. If the third difference 144 and the fourth difference 146 do not match in step ST224, the determination is negative, and the first medical support process proceeds to step ST220. If the third difference 144 and the fourth difference 146 match in step ST224, the determination is positive, and the first medical support process proceeds to step ST226.

[0222] In step ST226, the control unit 82C acquires from the fifth storage area 84E the push-in process frame 40C to which the third movement distance 112C used in calculating the third difference 144, which is equal to the fourth difference 146, has been assigned. After the process of step ST226 is executed, the first medical support process proceeds to step ST228.

[0223] In addition, when the control unit 82C acquires the pressing process frame 40C from the fifth storage area 84E, the control unit 82C may erase the pressing process frame 40C acquired from the fifth storage area 84E from the fifth storage area 84E. Then, when all of the pressing process frames 40C stored in the fifth storage area 84E are erased from the fifth storage area 84E, the control unit 82C may notify that all of the pressing process frames 40C stored in the fifth storage area 84E have been erased from the fifth storage area 84E. Examples of the notification in this case include displaying information (e.g., text and / or marks, etc.) on the screen 35 that can specify that all of the pressing process frames 40C stored in the fifth storage area 84E have been erased from the fifth storage area 84E, and outputting a sound from a speaker (not shown) that can specify that all of the pressing process frames 40C stored in the fifth storage area 84E have been erased from the fifth storage area 84E.

[0224] In step ST228, the control unit 82C displays the pushing process frame 40C acquired from the fifth storage area 84E in the second display area 38. If the pushing process frame 40C is already displayed in the second display area 38, the control unit 82C updates the pushing process frame 40C displayed in the second display area 38 to the latest pushing process frame 40C acquired from the fifth storage area 84E. After the processing of step ST228 is executed, the first medical support processing proceeds to step ST52.

[0225] Next, the second medical support process will be described with reference to Fig. 20A and Fig. 20B. The second medical support process is an example of a process assuming a case where a transition is made from a drawing process 41D to a pushing process 41C in an insertion process 41A or a removal process 41B (see Fig. 14), and is executed by the processor 82.

[0226] The flowcharts shown in Figures 20A and 20B differ from the flowcharts shown in Figures 19A and 19B in that they have steps ST300 to ST304 instead of steps ST204 to ST208, steps ST306 and ST308 instead of steps ST212 and ST214, and steps ST310 to ST320 instead of steps ST218 to ST228. In the following, the same steps in the flowcharts shown in Figures 20A and 20B as those in the flowcharts shown in Figures 10A, 10B, 19A, and 19B are denoted by the same step numbers as those in Figures 10A, 10B, 19A, and 19B, and descriptions thereof will be omitted.

[0227] 19A and 19B and the second medical support process shown in Fig. 20A and 20B are executed in parallel, the process of the steps overlapping between the first medical support process shown in Fig. 19A and 19B and the second medical support process shown in Fig. 20A and 20B may be executed in either the first medical support process or the second medical support process. Also, when the first medical support process shown in Fig. 19A and 19B and the second medical support process shown in Fig. 20A and 20B are executed in parallel, a screen displaying the process result of the first medical support process shown in Fig. 19A and 19B and a screen displaying the process result of the second medical support process shown in Fig. 20A and 20B may be provided separately, or the process result of the first medical support process and the process result of the second medical support process may be displayed in a distinguishable manner in a plurality of display areas provided in one screen (e.g., screen 35).

[0228] Furthermore, when the medical support processing according to the first embodiment, the first medical support processing shown in Figures 19A and 19B, and the second medical support processing shown in Figures 20A and 20B are executed in parallel, a screen on which the processing results of the medical support processing according to the first embodiment are displayed, a screen on which the processing results of the first medical support processing shown in Figures 19A and 19B are displayed, and a screen on which the processing results of the second medical support processing shown in Figures 20A and 20B are displayed may be provided separately, or the processing results of the medical support processing according to the first embodiment, the processing results of the first medical support processing shown in Figures 19A and 19B, and the processing results of the second medical support processing shown in Figures 20A and 20B may be displayed in a distinguishable manner in multiple display areas provided within a single screen (e.g., screen 35).

[0229] In the second medical support process shown in Fig. 20A, in step ST300, the control unit 82C judges whether or not the approaching process tag 134 has been assigned to the frame 40. In step ST300, if the approaching process tag 134 has not been assigned to the frame 40 (i.e., if the pushing-in process tag 132 has been assigned to the frame 40), the judgment is negative, and the second medical support process proceeds to step ST306 shown in Fig. 20B. In step ST300, if the approaching process tag 134 has been assigned to the frame 40, the judgment is positive, and the second medical support process proceeds to step ST302.

[0230] In step ST302, the control unit 82C chronologically stores the approaching process frame 40D to which the approaching process tag 134 is assigned, the moving distance 112 is assigned as the fourth moving distance 112D, and the recognition result 98 is assigned, in the sixth storage area 84F. After the process of step ST302 is executed, the second medical support process proceeds to step ST304.

[0231] In step ST304, the control unit 82C refers to the movement distance 112 for multiple frames estimated in step ST18 (i.e., the movement distance 112 for multiple frames along the time series) to determine whether or not the camera 52 has reached the direction change position P2B (see FIG. 14). In step ST304, if the camera 52 has not reached the direction change position P2B, the determination is negative, and the second medical support process proceeds to step ST10. In step ST304, if the camera 52 has reached the direction change position P2B, the determination is positive, and the second medical support process proceeds to step ST210.

[0232] 20B, the control unit 82C chronologically stores the pressing process frame 40C to which the pressing process tag 132 has been assigned, the moving distance 112 has been assigned as the third moving distance 112C, and the recognition result 98 has been assigned, in the fifth storage area 84E. After the process of step ST306 is executed, the second medical support process proceeds to step ST308.

[0233] In step ST308, the control unit 82C acquires the third movement distance 112C from the latest pushing process frame 40C stored in the fifth storage area 84E. After the process of step ST308 is executed, the second medical support process proceeds to step ST216.

[0234] In step ST310, the control unit 82C calculates the difference between the third movement distance 112C acquired in step ST308 and the second reference movement distance 136 acquired in step ST216, that is, the third difference 144. After the process of step ST310 is executed, the second medical support process proceeds to step ST312.

[0235] In step ST312, the control unit 82C acquires the fourth moving distance 112D that is unused in this step ST312. After the process of step ST312 is executed, the second medical support process proceeds to step ST314.

[0236] In step ST314, the control unit 82C calculates the difference between the fourth movement distance 112D acquired in step ST312 and the second reference movement distance 136 acquired in step ST216, that is, the fourth difference 146. After the process of step ST314 is executed, the second medical support process proceeds to step ST316.

[0237] In step ST316, the control unit 82C determines whether or not the third difference 144 calculated in step ST310 and the fourth difference 146 calculated in step ST314 match. If the third difference 144 and the fourth difference 146 do not match in step ST316, the determination is negative, and the second medical support process proceeds to step ST312. If the third difference 144 and the fourth difference 146 match in step ST316, the determination is positive, and the second medical support process proceeds to step ST318.

[0238] In step ST318, the control unit 82C acquires from the sixth storage area 84F the retraction process frame 40D to which the fourth movement distance 112D used in calculating the fourth difference 146 that coincides with the third difference 144 has been assigned. After the process of step ST318 is executed, the second medical support process proceeds to step ST320.

[0239] When the control unit 82C acquires the approaching process frame 40D from the sixth storage area 84F, the control unit 82C may erase the approaching process frame 40D acquired from the sixth storage area 84F from the sixth storage area 84F. When all of the approaching process frames 40D stored in the sixth storage area 84F are erased from the sixth storage area 84F, the control unit 82C may notify that all of the approaching process frames 40D stored in the sixth storage area 84F have been erased from the sixth storage area 84F. Examples of the notification in this case include displaying information (e.g., text and / or marks) on the screen 35 that can specify that all of the approaching process frames 40D stored in the sixth storage area 84F have been erased from the sixth storage area 84F, and outputting a sound from a speaker (not shown) that can specify that all of the approaching process frames 40D stored in the sixth storage area 84F have been erased from the sixth storage area 84F.

[0240] In step ST320, the control unit 82C displays the approaching process frame 40D acquired from the sixth storage area 84F in the second display area 38. If the approaching process frame 40D is already displayed in the second display area 38, the control unit 82C updates the approaching process frame 40D displayed in the second display area 38 to the latest approaching process frame 40D acquired from the sixth storage area 84F. After the process of step ST320 is executed, the second medical support process proceeds to step ST52.

[0241] As described above, in the endoscope device 10 according to the second embodiment, the second identification process is executed based on the third movement distance 112C obtained in the pushing process 41C and the fourth movement distance 112D obtained in the drawing process 41D. In the second embodiment, when the third difference 144 and the fourth difference 146 match, the second identification process is executed. The third difference 144 is the difference between the second reference movement distance 136 (i.e., the movement distance 112 from the anus to the direction change position P2) and the third movement distance 112C, and the fourth difference 146 is the difference between the second reference movement distance 136 and the fourth movement distance 112D. The second identification process is a process capable of identifying the identity of the lesion 42 captured in the pushing process frame 40C and the lesion 42 captured in the drawing process frame 40D. Therefore, according to the endoscope device 10 according to the second embodiment, it is possible to identify whether the same lesion 42 is captured by the camera 52 in the pushing process 41C and the drawing process 41D.

[0242] In the endoscope device 10 according to the second embodiment, when the pushing process 41C precedes the retracting process 41D (see FIG. 13) and the lesion 42 is captured in the retracting process frame 40D during the retracting process 41D, the control unit 82C determines whether the third difference 144 and the fourth difference 146 match. The third difference 144 corresponds to the distance from the lesion 42 found during the pushing process 41C to the direction change position P2A (see FIG. 13), and the fourth difference 146 corresponds to the distance from the direction change position P2A to the lesion 42 found during the retracting process 41D. When the third difference 144 and the fourth difference 146 match, the control unit 82C determines that the level of identity between the lesion 42 captured in the pushing process frame 40C and the lesion 42 captured in the retracting process frame 40D is equal to or higher than a preset level. When the control unit 82C determines that the level of identity between the lesion 42 captured in the pushing process frame 40C and the lesion 42 captured in the pulling process frame 40D is equal to or higher than a predetermined level, the control unit 82C executes a second display process 148. In the second display process 148, the pulling process frame 40D to which the fourth movement distance 112D used in calculating the fourth difference 146 that coincides with the third difference 144 has been added is displayed in the first display area 36, ​​and the pushing process frame 40C to which the third movement distance 112C used in calculating the third difference 144 has been added is displayed in the second display area 38.

[0243] Therefore, according to the endoscopic device 10 of the second embodiment, when the pushing process 41C precedes the retraction process 41D (see Figure 13) and the control unit 82C determines that the level of identity between the lesion 42 depicted in the retraction process frame 40D and the lesion 42 depicted in the pushing process frame 40C is above a predetermined level, the doctor 12 can visually grasp the appearance of the lesion 42 depicted in the retraction process frame 40D displayed in the first display area 36 and the appearance of the lesion 42 depicted in the pushing process frame 40C displayed in the second display area 38.

[0244] In the endoscope device 10 according to the second embodiment, when the approaching process 41D precedes the pushing process 41C (see FIG. 14) and the lesion 42 is captured in the pushing process frame 40C during the pushing process 41C, the control unit 82C determines whether the third difference 144 and the fourth difference 146 match. The fourth difference 146 corresponds to the distance from the lesion 42 found during the approaching process 41D to the direction change position P2B (see FIG. 14), and the third difference 144 corresponds to the distance from the direction change position P2A to the lesion 42 found during the pushing process 41C. When the third difference 144 and the fourth difference 146 match, the control unit 82C determines that the level of identity between the lesion 42 captured in the pushing process frame 40C and the lesion 42 captured in the approaching process frame 40D is equal to or higher than a preset level. When the controller 82C determines that the level of identity between the lesion 42 captured in the pushing process frame 40C and the lesion 42 captured in the retraction process frame 40D is equal to or higher than a predetermined level, the controller 82C executes a second display process 148. In the second display process 148, the pushing process frame 40C to which the third movement distance 112C used in calculating the third difference 144 that coincides with the fourth difference 146 has been added is displayed in the first display area 36, ​​and the retraction process frame 40D to which the fourth movement distance 112D used in calculating the fourth difference 144 has been added is displayed in the second display area 38.

[0245] Therefore, according to the endoscopic device 10 of the second embodiment, when the retraction process 41D precedes the pushing process 41C (see Figure 14) and the control unit 82C determines that the level of identity between the lesion 42 depicted in the pushing process frame 40C and the lesion 42 depicted in the retraction process frame 40D is above a predetermined level, the doctor 12 can visually grasp the appearance of the lesion 42 depicted in the pushing process frame 40C displayed in the first display area 36 and the appearance of the lesion 42 depicted in the retraction process frame 40D displayed in the second display area 38.

[0246] Furthermore, in the endoscope device 10 according to the second embodiment, a pushing process tag 132 is assigned to the pushing process frame 40C, and a pulling process tag 134 is assigned to the retraction process frame 40D. The pushing process tag 132 is a tag that can identify that the pushing process frame 40C is a frame 40 obtained by imaging an area including the lesion 42 by the camera 52 during the pushing process 41C. The retraction process tag 134 is a tag that can identify that the retraction process frame 40D is a frame 40 obtained by imaging an area including the lesion 42 by the camera 52 during the retraction process 41D.

[0247] Therefore, according to the endoscope device 10 according to the second embodiment, it is possible to easily distinguish between the pushing-in process frame 40C, which is the frame 40 obtained by imaging the area including the lesion 42 by the camera 52 during the pushing-in process 41C, and the retracting process frame 40D, which is the frame 40 obtained by imaging the area including the lesion 42 by the camera 52 during the retracting process 41D. As a result, it becomes easier to perform processing using the pushing-in process frame 40C and the retracting process frame 40D.

[0248] In the first medical support process described in the second embodiment, the pushing process frame 40C is displayed in the second display area 38 on condition that the third difference 144 and the fourth difference 146 match, but the present disclosure is not limited to this. For example, when the pushing process 41C precedes the pulling process 41D (see FIG. 13), the pushing process frame 40C may be displayed in the second display area 38 on condition that the pushing process frame 40C to which the third movement distance 112C used in calculating the third difference 144 that matches the fourth difference 146 is added is continuously obtained for more than a second preset number of frames (e.g., a plurality of frames designated in advance). In this case, the indentation process frame 40C displayed in the second display area 38 may be the indentation process frame 40C obtained first in time among the multiple indentation process frames 40C obtained continuously for more than the second preset frame number, or the indentation process frame 40C obtained last in time, or the indentation process frame 40C obtained between the indentation process frame 40C obtained first in time and the indentation process frame 40C obtained last in time. In addition, among the multiple indentation process frames 40C obtained continuously for more than the second preset frame number, the indentation process frame 40C in which the position in which the lesion 42 is captured is closest to the center of the angle of view of the camera 52 may be displayed in the second display area 38. In addition, all of the multiple indentation process frames 40C obtained continuously for more than the second preset frame number, or multiple indentation process frames 40C selected according to some rule may be displayed in the second display area 38 in a state in which they can be compared.

[0249] An example of the flow of the first medical support process for realizing this embodiment is shown in FIG. 21. The first medical support process shown in FIG. 21 is different from the first medical support process shown in FIG. 19B in that a process of step ST400 is provided between the process of step ST224 and the process of step ST226. In step ST400 shown in FIG. 21, the control unit 82C judges whether or not the pushing process frame 40C to which the third moving distance 112C used in the calculation of the third difference 144 that matches the fourth difference 146 is provided continues for the second preset frame number or more. In step ST400, if the pushing process frame 40C to which the third moving distance 112C used in the calculation of the third difference 144 that matches the fourth difference 146 is provided does not continue for the second preset frame number or more, the judgment is denied and the first medical support process proceeds to step ST52. In step ST400, if the pushing process frame 40C to which the third movement distance 112C used to calculate the third difference 144 that matches the fourth difference 146 has been assigned continues for more than the second specified number of frames, the judgment is positive and the first medical support processing proceeds to step ST226.

[0250] In this way, when the pushing process 41C precedes the retraction process 41D (see Figure 13), the pushing process frame 40C is displayed in the second display area 38 on the condition that the pushing process frame 40C having the third movement distance 112C used to calculate the third difference 144 that matches the fourth difference 146 has been obtained continuously for more than the second specified number of frames, thereby allowing the doctor 12 to observe the same lesion 42 (in other words, the lesion 42 recognized by the recognition unit 82A) that is likely to have been observed by the doctor 12 during the pushing process 41C during the retraction process 41D.

[0251] In addition, when the approaching process 41D precedes the pushing process 41C (see FIG. 14), the approaching process frame 40D may be displayed in the second display area 38 on the condition that the approaching process frame 40D to which the fourth movement distance 112D used in calculating the fourth difference 146 that coincides with the third difference 144 is given is continuously obtained for a third preset frame number or more (e.g., a plurality of frames designated in advance). In this case, the approaching process frame 40D displayed in the second display area 38 may be the approaching process frame 40D obtained first in time, the approaching process frame 40D obtained last in time, or the approaching process frame 40D obtained between the approaching process frame 40D obtained first in time and the approaching process frame 40D obtained last in time. Moreover, among the multiple attraction process frames 40D continuously obtained for the third preset frame number or more, the attraction process frame 40D in which the position in which the lesion 42 is captured is closest to the center of the angle of view of the camera 52 may be displayed in the second display area 38. Moreover, all of the multiple attraction process frames 40D continuously obtained for the third preset frame number or more, or multiple attraction process frames 40D selected according to some rule, may be displayed in the second display area 38 in a comparable state.

[0252] An example of the flow of the second medical support process for realizing this embodiment is shown in FIG. 22. The second medical support process shown in FIG. 22 is different from the second medical support process shown in FIG. 20B in that a process of step ST500 is provided between the process of step ST316 and the process of step ST318. In step ST500 shown in FIG. 22, the control unit 82C judges whether or not the approaching process frame 40D to which the fourth movement distance 112D used in the calculation of the fourth difference 146 that matches the third difference 144 is given continues for the third preset frame number or more. In step ST500, if the approaching process frame 40D to which the fourth movement distance 112D used in the calculation of the fourth difference 146 that matches the third difference 144 is given does not continue for the third preset frame number or more, the judgment is denied and the second medical support process proceeds to step ST52. In step ST500, if the retraction process frame 40D to which the fourth movement distance 112D used to calculate the fourth difference 146 that matches the third difference 144 has been assigned continues for more than the third specified number of frames, the judgment is positive and the second medical support processing proceeds to step ST318.

[0253] In this way, when the approach process 41D precedes the pushing process 41C (see Figure 14), the approach process frame 40D is displayed in the second display area 38 on the condition that the approach process frame 40D, which has the fourth movement distance 112D used to calculate the fourth difference 146 that matches the third difference 144, has been obtained continuously for more than the third specified number of frames, thereby allowing the doctor 12 to observe the same lesion 42 in the pushing process 41C as the lesion 42 that is likely to have been observed by the doctor 12 in the approach process 41D (in other words, the lesion 42 recognized by the recognition unit 82A).

[0254] In the second embodiment, an example in which the second display process 148 is executed by the control unit 82C has been given, but this is merely one example, and for example, the second notification process 147 may be executed by the control unit 82C as shown in Fig. 23. The second notification process 147 is a process for notifying that the level of identity between the lesion 42 captured in the pushing-in process frame 40C and the lesion 42 captured in the pulling-in process frame 40D is equal to or higher than a preset level when the level of identity between the lesion 42 captured in the pushing-in process frame 40C and the lesion 42 captured in the pulling-in process frame 40D is equal to or higher than a preset level.

[0255] An example of the notification is a case where a message 149 notifying that the level of identity between the lesion 42 captured in the indentation process frame 40C and the lesion 42 captured in the retraction process frame 40D is equal to or higher than a preset level is displayed in the second display area 38 as one of the auxiliary information 44. Also, a mark or a numerical value, etc., notifying that the level of identity between the lesion 42 captured in the indentation process frame 40C and the lesion 42 captured in the retraction process frame 40D is equal to or higher than a preset level may be displayed in the second display area 38 as one of the auxiliary information 44. Also, similar to the second embodiment, the indentation process frame 40C or the retraction process frame 40D may be displayed in the second display area 38. Also, the information notifying that the level of identity between the lesion 42 captured in the indentation process frame 40C and the lesion 42 captured in the retraction process frame 40D is equal to or higher than a preset level may be displayed in an area other than the second display area 38 in the screen 35, or may be displayed on the screen of a display device different from the display device 18. In addition, a sound may be output from a speaker (not shown) to notify that the level of identity between the lesion 42 shown in the pushing process frame 40C and the lesion 42 shown in the retraction process frame 40D is equal to or higher than a predetermined level.

[0256] In this way, the second notification process 147 is executed by the control unit 82C, so that the doctor 12 can perceive that the level of identity between the lesion 42 shown in the pushing process frame 40C and the lesion 42 shown in the retraction process frame 40D is equal to or higher than a predetermined level.

[0257] In the example shown in Fig. 23, as a notification example when the pushing process 41C precedes the pulling process 41D (see Fig. 13), the pulling process frame 40D is displayed in the first display area 36, ​​and a message indicating that the same lesion 42 as the lesion 42 shown in the pulling process frame 40D displayed in the first display area 36 was also recognized in the pushing process 41C (in the example shown in Fig. 23, a message "The same lesion was recognized during pushing") is displayed in the second display area 38 as the message 149, but this is merely an example. When the pulling process 41D precedes the pushing process 41C (see Fig. 14), the pushing process frame 40C may be displayed in the first display area 36, ​​and a message indicating that the same lesion 42 as the lesion 42 shown in the pushing process frame 40C displayed in the first display area 36 was also recognized in the pulling process 41D may be displayed in the second display area 38.

[0258] [Other embodiments] In each of the above embodiments, the estimation unit 82B executes the movement estimation process 110 to estimate the movement distance 112, but if the estimation unit 82B is unable to estimate the movement distance 112 by executing the movement estimation process 110, the estimation unit 82B may estimate the movement distance 112 as an exceptional process based on the previously estimated movement distance 112 and the elapsed time since the first tag was switched to the second tag. In this case, for example, the estimation unit 82B assumes that the movement speed of the camera 52 is constant and estimates the movement distance 112 using linear interpolation, a linear Kalman filter, or the like.

[0259] Here, a first example of the first tag is the insertion process tag 116. A first example of the second tag when the first tag is the insertion process tag 116 is the removal process tag 118. A second example of the first tag is the pushing process tag 132. A second example of the second tag when the first tag is the pushing process tag 132 is the pulling process tag 134. A third example of the first tag is the pulling process tag 134. A third example of the second tag when the first tag is the pulling process tag 134 is the pushing process tag 132.

[0260] In each of the above embodiments, an example in which the frame 40 is displayed in the first display region 36 and the second display region 38 has been given, but the lesion location map 104 may be displayed in at least one of the first display region 36 and the second display region 38. For example, together with the frame 40 displayed in the first display region 36 or instead of the frame 40 displayed in the first display region 36, the lesion location map 104 corresponding to the frame displayed in the first display region 36 may be displayed in the first display region 36, or together with the frame 40 displayed in the second display region 38 or instead of the frame 40 displayed in the second display region 38, the lesion location map 104 corresponding to the frame displayed in the second display region 36 may be displayed.

[0261] In the above-described embodiments, an example in which one frame 40 is displayed in the second display area 38 has been given, but this is merely an example, and a plurality of frames 40 may be displayed in the second display area 38. The plurality of frames 40 may be displayed in a comparatively arranged manner, or may be displayed in order of increasing similarity to the lesion 42 captured in the frame 40 displayed in the first display area 36. An example of a plurality of frames 40 includes a plurality of frames 40 obtained successively. In addition, one or more frames 40 capturing a lesion 42 having a certain degree of similarity or higher to the lesion 42 captured in the frame 40 displayed in the first display area 36 may be displayed in the second display area 38.

[0262] In each of the above embodiments, an example was given in which the frame 40 is displayed in the second display area 38, but rather than the entire frame 40 being displayed in the second display area 38, only a portion of the image area contained in the frame 40 (for example, a lesion area indicating the lesion 42) may be displayed in the second display area 38.

[0263] In the above-mentioned embodiments, the first identification process is performed as a process capable of identifying that the lesion 42 captured in the insertion process frame 40A and the lesion 42 captured in the removal process frame 40B are the same, but the present disclosure is not limited thereto. For example, when the lesion 42 captured in the insertion process frame 40A and the lesion 42 captured in the removal process frame 40B are not the same, a process capable of identifying that the lesion 42 captured in the insertion process frame 40A and the lesion 42 captured in the removal process frame 40B are not the same (for example, a display process and / or a notification process performed in the same manner as the first display process and the first notification process described above) may be performed. In this way, the process capable of identifying that the lesion 42 captured in the insertion process frame 40A and the lesion 42 captured in the removal process frame 40B are not the same is performed, so that the doctor 12 can understand that the lesion 42 discovered in the insertion process 41A has not been imaged in the removal process 41B.

[0264] In a similar manner, if the lesion 42 captured in the pushing process frame 40C and the lesion 42 captured in the retraction process frame 40D are not identical, a process capable of identifying that the lesion 42 captured in the pushing process frame 40C and the lesion 42 captured in the retraction process frame 40D are not identical (for example, a display process and / or a notification process performed in a manner similar to the second display process and second notification process described above) may be executed.

[0265] In each of the above-described embodiments, the position of the camera 52 inserted into the large intestine 28 in the large intestine 28 is identified based on the moving distance 112, but the present disclosure is not limited to this. For example, the position of the camera 52 in the large intestine 28 may be identified based on the detection result by an endoscope insertion shape observation device (commonly known as Coronabi).

[0266] In this case, as shown in Fig. 24 as an example, the camera 52 is provided with a magnetic field generating coil 150, and a receiver 152 is connected to the external I / F 80. The magnetic field generated from the magnetic field generating coil 150 is received by the receiver 152. The processor 82 detects the behavior (e.g., position and orientation) of the camera 52 within the large intestine 28 based on the reception result by the receiver 152, and identifies the position of the camera 52 within the large intestine 28 based on the detection result. In this way, the behavior of the camera 52 within the large intestine 28 is detected with high accuracy, and therefore it is possible to identify the position of the camera 52 within the large intestine 28 with high accuracy.

[0267] The endoscope insertion shape observation device described here is merely an example, and a sensor other than the endoscope insertion shape observation device may be used. For example, the position of the camera 52 in the large intestine 28 may be identified based on the detection result by a sensor capable of detecting the behavior of the camera 52 in the large intestine 28, such as an acceleration sensor, a gyro sensor, and / or a magnetic sensor.

[0268] In each of the above embodiments, lesion 42 has been exemplified as an example of the "first feature region," "second feature region," "third feature region," and "fourth feature region" according to the present disclosure, but the present disclosure is not limited thereto, and the above-mentioned medical support processing can be performed even if a resection region, a bleeding region, a marking region, an organ, or a treatment tool (e.g., a hemostatic clip placed inside the body) is applied instead of lesion 42.

[0269] In each of the above embodiments, the frame 40 is input to the recognition unit 82A, and the processing result of the medical support processing is output from the control unit 82C. However, the present disclosure is not limited to this. For example, instruction data (so-called prompt) including the frame 40 may be input to a so-called generation AI, and the processing result of the medical support processing may be output from the generation AI. One example of the generation AI is GPT-4 (Internet search<https: / / openai.com / gpt-4> ) is used in ChatGPT.

[0270] Also, instruction data including at least a part of the processing result of the medical support process (e.g., the frame 40 displayed on the screen 35 by executing the first display process, the frame 40 displayed on the screen 35 by executing the second display process, information notified by executing the first notification process, and / or information notified by executing the second notification process, etc.) may be used as input information for the generation AI. Examples of information output from the generation AI include information regarding the operation content of the camera 52, information indicating the content of medical procedures recommended to be performed during the endoscopic examination, and / or information indicating the content of medical procedures recommended to be performed after the endoscopic examination, etc. The information output from the generation AI may be stored in various storage areas (e.g., storage 76 and / or 86), displayed on the display device 18 as auxiliary information 44, printed on a medium by a printer, or output as audio from a speaker.

[0271] In each of the above embodiments, the recognition process 96 using AI in the bounding box method has been exemplified, but this is merely one example, and for example, recognition process using AI in the segmentation method may be performed instead of the recognition process 96 using AI in the bounding box method. Also, instead of the recognition process using AI, a recognition process using a non-AI method (for example, a template matching method) may be performed, or a recognition process using a combination of the non-AI method and the AI ​​method may be performed.

[0272] In each of the above embodiments, an example in which the medical support processing is performed by the computer 78 has been described, but the present disclosure is not limited to this, and at least a part of the processing included in the medical support processing may be performed by a device provided outside the computer 78. An example of this case will be described below with reference to FIG.

[0273] 25 is a conceptual diagram showing an example of the configuration of an endoscope device 154. The endoscope device 154 is an example of the "endoscope device" according to the present disclosure. The endoscope device 154 differs from the endoscope device 10 described in each of the above embodiments in that it includes an external device 156.

[0274] External device 156 is communicatively coupled to computer 78 via a network 158 (eg, a WAN and / or LAN, etc.).

[0275] An example of the external device 156 is at least one server that directly or indirectly transmits and receives data to and from the computer 78 via the network 158. The external device 156 receives a processing execution instruction provided from the processor 82 of the computer 78 via the network 158. The external device 156 then executes processing according to the received processing execution instruction and transmits the processing result to the computer 78 via the network 158. In the computer 78, the processor 82 receives the processing result transmitted from the external device 156 via the network 158 and executes processing using the received processing result.

[0276] An example of the processing execution instruction is an instruction to have the external device 156 execute at least a part of the medical support processing. A first example of at least a part of the medical support processing (i.e., a processing to be executed by the external device 156) is the recognition processing 96. In this case, the external device 156 executes the recognition processing 96 in accordance with the processing execution instruction provided from the processor 82 via the network 158, and transmits the recognition result 98 to the computer 78 via the network 158. In the computer 78, the processor 82 receives the recognition result 98, and executes the same processing as in each of the above-described embodiments using the received recognition result 98.

[0277] A second example of at least a part of the medical support process (i.e., a process to be executed by the external device 156) is a process by the control unit 82C. In this case, the external device 156 executes the process by the control unit 82C according to a process execution instruction given from the processor 82 via the network 158, and transmits the process result (e.g., the frame 40 displayed on the screen 35 by executing the first display process, the frame 40 displayed on the screen 35 by executing the second display process, information notified by executing the first notification process, and / or information notified by executing the second notification process, etc.) to the computer 78 via the network 158. In the computer 78, the processor 82 receives the process result and executes the same process as in each of the above-mentioned embodiments (e.g., display using the display device 18, etc.) using the received process result.

[0278] For example, the external device 156 is realized by cloud computing. Note that cloud computing is merely an example, and the external device 156 may be realized by network computing such as fog computing, edge computing, or grid computing. Instead of a server, at least one personal computer or the like may be used as the external device 156. Also, the external device 156 may be a computing device with a communication function equipped with multiple types of AI functions.

[0279] In each of the above embodiments, the medical support program 90 is stored in the storage 86, but the present disclosure is not limited to this. For example, the medical support program 90 may be stored in a portable computer-readable non-transitory storage medium such as an SSD or a USB memory. The medical support program 90 stored in the non-transitory storage medium is installed in the computer 78 of the endoscope device 10. The processor 82 executes medical support processing in accordance with the medical support program 90.

[0280] In addition, the medical support program 90 may be stored in a storage device such as another computer or server connected to the endoscopic device 10 via a network, and the medical support program 90 may be downloaded and installed on the computer 78 upon request from the endoscopic device 10.

[0281] It is not necessary to store all of the medical support program 90 in a storage device such as another computer or server device connected to the endoscopic device 10, or to store all of the medical support program 90 in the storage 86; only a portion of the medical support program 90 may be stored.

[0282] The hardware resources for executing the medical support processing can be various processors as shown below. An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the medical support processing by executing software, i.e., a program. Another example of a processor is a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing a specific process, such as an FPGA, a PLD, or an ASIC. Each processor has a built-in or connected memory, and each processor executes the medical support processing by using the memory.

[0283] The hardware resource for executing the medical support process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource for executing the medical support process may be a single processor.

[0284] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes medical support processing. Second, there is a configuration using a processor that realizes the functions of the entire system including multiple hardware resources that execute medical support processing on a single IC chip, as typified by SoC. In this way, the medical support processing is realized using one or more of the above various processors as hardware resources.

[0285] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. Also, the above medical support process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processing may be changed, without departing from the spirit of the invention.

[0286] The above description and illustrations are detailed descriptions of the parts related to the present disclosure and are merely an example of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations within the scope of the gist of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the present disclosure.

[0287] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0288] 10,154 Endoscopic equipment 12. Doctor 14. Nurse 16 Endoscope 18 Display device 20 Light source device 22 Control device 24 Medical Support Equipment 26 Subject 28 Large intestine 28A Ileocecal area 29A Insertion Path 29B Removal route 30 light 32 Intestinal wall 34 Wagon 35 screens 36 1st display area 38 Second display area 39 Endoscopic video images 40 frames 40A Insertion Process Frame 40B Removal process frame 40C Pressing process frame 40D Attraction Process Frame 41A Insertion process 41B Removal process 41C Pushing process 41D Attraction process 42 Lesions 44 Supporting Information 46 Control section 48 Insertion section 50 Tip 50A Tip surface 52 Camera 54 Lighting Equipment 54A, 54B Lighting window 56 Treatment opening 58 Treatment tools 60 Treatment tool insertion port 62 Universal Code 64 Reception device 66,78 Computer 68,88 Bus 70,80 External I / F 72,82 processor 74,84 Memory 76,86 Storage 82A recognition part 82B Estimation part 82C Control section 84A First Storage Area 84B Second storage area 84C Third storage area 84D 4th storage area 84E 5th storage area 84F 6th storage area 90 Medical Assistance Program 92 Recognition Model 94 Movement Estimation Model 96 Recognition Processing 98 Recognition results 100 Lesion presence / absence information 102 Geometric property information 104 Lesion Location Map 106 Lesion characteristic information 108 Part information 110 Movement Estimation Processing 112 Distance traveled 112A 1st movement distance 112B 2nd movement distance 112C 3rd movement distance 112D 4th movement distance 114 Direction of movement 116 Insertion Process Tags 118 Removal Process Tag 120 1st standard movement distance 122 First comparison process 124 1st difference 126 2nd difference 128 First display process 129 First Notification Processing 130,149 Messages 132 Indentation Process Tag 134 Attraction Process Tags 136 2nd standard movement distance 142 Second comparison process 144 3rd difference 146 4th difference 147 Second notification process 148 Second display processing 150 Magnetic field generating coil 152 Receiver 156 External device 158 Network BB Bounding Box P1 Push start position P2, P2A, P2B Turning position P3 End position of pull P4 Pull start position P5 Push end position

Claims

1. A processor is provided. The processor, acquiring position identification information capable of identifying a position within the body of a camera to be inserted into the body; A first identification process is performed that can identify the identity of the first feature region and the second feature region based on insertion position identification information, which is the position identification information corresponding to an insertion process frame obtained by capturing an image of a first feature region inside the body by the camera during the insertion of the camera into the body, and removal position identification information, which is the position identification information corresponding to a removal process frame obtained by capturing an image of a second feature region inside the body by the camera during the removal of the camera from the body. Medical support equipment.

2. The first identification process includes a first notification process of, when a level of identity between the first feature region and the second feature region is equal to or higher than a first preset level, notifying that a level of identity between the first feature region and the second feature region is equal to or higher than a first preset level. The medical support device according to claim 1 .

3. The first identification process includes a first display process of displaying, on a first screen, a first image in which at least the first feature region can be identified and which is captured in the insertion process frame, and a second image in which at least the second feature region can be identified and which is captured in the removal process frame, in a comparative state when a level of identity between the first feature region and the second feature region is equal to or higher than a second preset level. The medical support device according to claim 1 .

4. The processor, Obtain the insertion process frame; The first identification process is executed on the condition that a plurality of the insertion process frames equal to or greater than a first predetermined number of frames are continuously acquired. The medical support device according to claim 1 .

5. an insertion process identifier is assigned to the insertion process frame, which is capable of identifying the frame as having been obtained by capturing an image of the first feature region by the camera during the insertion process; A removal process identifier that can identify the frame as being obtained by capturing an image of the second feature area by the camera during the removal process is assigned to the removal process frame. The medical support device according to claim 1 .

6. The processor executes a second identification process capable of identifying the identity of the third feature region and the fourth feature region based on pushing position identification information, which is the position identification information corresponding to a pushing process frame obtained by imaging a third feature region inside the body by the camera during a pushing process in which the camera is pushed into the back side of the body, and drawing position identification information, which is the position identification information corresponding to a drawing process frame obtained by imaging a fourth feature region inside the body by the camera during a drawing process in which the camera is drawn from inside the body to the front side. The medical support device according to claim 1 .

7. A processor is provided. The processor, acquiring position identification information capable of identifying a position within the body of a camera to be inserted into the body; A second identification process is performed that can identify the identity of the third feature region and the fourth feature region based on pushing position identification information, which is the position identification information corresponding to a pushing process frame obtained by imaging a third feature region inside the body by the camera during a pushing process in which the camera is pushed into the inner side of the body, and drawing position identification information, which is the position identification information corresponding to a drawing process frame obtained by imaging a fourth feature region inside the body by the camera during a drawing process in which the camera is drawn from the inner side to the front side. Medical support equipment.

8. The second identification process includes a second notification process of, when a level of identity between the third feature region and the fourth feature region is equal to or higher than a third preset level, notifying that a level of identity between the third feature region and the fourth feature region is equal to or higher than a third preset level. The medical support device according to claim 6.

9. The second identification process includes a second display process of displaying, on a second screen, a third image in which at least the third feature region can be identified and which is captured in the pushing process frame, and a fourth image in which at least the fourth feature region can be identified and which is captured in the pulling process frame, in a comparative state, when a level of identity between the third feature region and the fourth feature region is equal to or higher than a fourth preset level. The medical support device according to claim 6.

10. The processor, Acquire the indentation process frame; When the pushing process precedes the pulling process, the second identification process is executed on the condition that a plurality of frames of the pushing process equal to or greater than a second predetermined number of frames are continuously acquired. The medical support device according to claim 6.

11. The processor, Acquire the attraction process frame; When the approaching process precedes the pushing process, the second identification process is executed on the condition that a plurality of the approaching process frames equal to or greater than a third predetermined number of frames are continuously acquired. The medical support device according to claim 6.

12. an indentation process identifier that can identify the indentation process frame as a frame obtained by imaging the third feature region with the camera during the indentation process is assigned to the indentation process frame; The approaching process frame is assigned an approaching process identifier that can identify the frame as being obtained by imaging the fourth feature region by the camera during the approaching process. The medical support device according to claim 6.

13. The third feature region and the fourth feature region are lesions. The medical support device according to claim 6.

14. The first feature region and the second feature region are lesions. The medical support device according to claim 1 .

15. The position is estimated based on a number of frames obtained by imaging the inside of the body with the camera. The medical support device according to claim 1 .

16. The location information is defined based on a distance traveled by the camera within the body. The medical support device according to claim 1 .

17. A medical support device according to any one of claims 1 to 16; The camera. Endoscopic device.

18. Obtaining position identification information capable of identifying a position within a body of a camera inserted into the body; and and executing a first identification process capable of identifying an identity between the first feature region and the second feature region based on insertion position identification information, which is the position identification information corresponding to an insertion process frame obtained by capturing an image of a first feature region inside the body by the camera during a process of inserting the camera into the body, and removal position identification information, which is the position identification information corresponding to a removal process frame obtained by capturing an image of a second feature region inside the body by the camera during a process of removing the camera from the body. Medical assistance methods.

19. Obtaining position identification information capable of identifying a position within a body of a camera inserted into the body; and and executing a second identification process capable of identifying an identity between the third feature region and the fourth feature region based on pushing position identification information, which is the position identification information corresponding to a pushing process frame obtained by imaging a third feature region inside the body by the camera during a pushing process in which the camera is pushed into the back side of the body, and drawing position identification information, which is the position identification information corresponding to a drawing process frame obtained by imaging a fourth feature region inside the body by the camera during a drawing process in which the camera is drawn from the back side to a front side. Medical assistance methods.

20. Obtaining position identification information capable of identifying a position within a body of a camera inserted into the body; and A program for causing a computer to execute a process including executing a first identification process capable of identifying identity between the first feature area and the second feature area based on insertion position identification information, which is the position identification information corresponding to insertion process frames obtained by the camera capturing images of a first feature area inside the body during the process of inserting the camera into the body, and removal position identification information, which is the position identification information corresponding to removal process frames obtained by the camera capturing images of a second feature area inside the body during the process of removing the camera from the body.

21. Obtaining position identification information capable of identifying a position within a body of a camera inserted into the body; and A program for causing a computer to execute a process including executing a second identification process capable of identifying identity between the third feature region and the fourth feature region based on pushing position identification information, which is the position identification information corresponding to a pushing process frame obtained by the camera capturing an image of a third feature region inside the body during a pushing process in which the camera is pushed into the back side of the body, and attraction position identification information, which is the position identification information corresponding to a pulling process frame obtained by the camera capturing an image of a fourth feature region inside the body during a pulling process in which the camera is pulled from the back side to a front side.

Images