Medical support device, endoscope apparatus, medical support method, and program

Abstract

To provide a medical support device capable of suppressing a user's recognition of an inappropriate processing result by object recognition processing of a medical image, an endoscope apparatus, a medical support method, and a program.SOLUTION: A medical support device includes a processor. The processor displays to a display unit processing result information to be information about a processing result by object recognition processing that has been performed to a first medical image generated by being photographed by a camera. The processor makes a display mode of the processing result information differ in accordance with similarity or difference between the first medical image and a second medical image stored in a storage.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present disclosure relates to a medical support device, an endoscope device, a medical support method, and a program. [Background technology]

[0002] An image processing device is disclosed in Patent Document 1. The image processing device described in Patent Document 1 includes a medical image acquisition unit that acquires a medical image, a scene of interest recognition unit that recognizes a scene of interest from a medical image acquired using the medical image acquisition unit, a similarity calculation unit that calculates a similarity between a medical image acquired using the medical image acquisition unit and a reference image defined for the scene of interest, for the scene of interest recognized using the scene of interest recognition unit, and a storage processing unit that performs processing to store the medical image in a storage device based on the similarity calculated using the similarity calculation unit.

[0003] The image processing device described in Patent Document 1 includes a medical image feature extraction unit that extracts features from a medical image, and a similarity calculation unit that calculates the similarity between the medical image and a reference image based on the features of the medical image.

[0004] The image processing device described in Patent Document 1 includes a reference image acquisition unit that acquires a reference image, and a reference image feature amount extraction unit that extracts features from the reference image acquired by using the reference image acquisition unit.

[0005] The image processing device described in Patent Document 1 is equipped with a reference image feature acquisition unit that acquires features of a reference image, and a similarity calculation unit calculates the similarity between the medical image and the reference image based on the features of the medical image and the features of the reference image.

[0006] In the image processing device described in Patent Document 1, the scene of interest recognition unit recognizes a scene including a lesion as a scene of interest. The scene of interest recognition unit also acquires a plurality of medical images from a medical image storage device in which the plurality of medical images are stored in advance, identifies scenes of interest for the plurality of medical images, and selects a reference image that is difficult for the scene of interest recognition unit to recognize from among medical images not recognized as scenes of interest. A storage processing unit stores the medical image in the storage device when the similarity is equal to or greater than a specified threshold value.

[0007] Patent Document 2 discloses a computer-assisted surgery system including an image capturing device, a display, a user interface, and a circuit. The computer-assisted surgery system described in Patent Document 2 is configured such that the circuit receives information indicating a surgery scenario and a surgery process associated with the surgery scenario, obtains an artificial image of the surgery scenario, outputs the artificial image for display to the display, and receives permission information indicating whether or not there is permission to execute the surgery process when it is determined that the surgery scenario will occur, via the user interface.

[0008] The computer-assisted surgery system described in Patent Document 2 is configured such that the circuit receives an actual image captured by an image capturing device, determines whether the actual image indicates the occurrence of a surgical scenario, and if the actual image indicates the occurrence of a surgical scenario, determines whether there is permission to perform a surgical process, and if there is permission to perform the surgical process, controls to execute a predetermined process.

[0009] In the computer-aided surgery system described in Patent Document 2, an artificial image is obtained using visualization of features of an artificial neural network configured to output information indicating a surgical scenario when an actual image of the surgical scenario captured by an image capturing device is input to the artificial neural network, and if the artificial neural network outputs information indicating the surgical scenario when the actual image is input to the artificial neural network, it is determined that the actual image indicates the occurrence of a surgical scenario.

[0010] The computer-assisted surgery system described in Patent Document 2 is configured such that the circuit compares a real image with an artificial image and executes a surgical process if the similarity between the real image and the artificial image exceeds a predetermined threshold.

[0011] In the computer-assisted surgery system described in Patent Document 2, the surgical process includes adjusting the field of view of the image capture device. Also, the surgical scenario is a surgical scenario in which an object obstructs the field of view of the image capture device, and the surgical process includes adjusting the field of view of the image capture device to avoid the obstructing object. Also, the surgical scenario is a surgical scenario in which the image capture device may collide with another object, and the surgical process includes adjusting the position of the image capture device to reduce the risk of the collision. [Prior art documents] [Patent documents]

[0012] [Patent Document 1] International Publication No. 2019 / 130924 [Patent Document 2] Special table number 2023-506355 Summary of the Invention

[0013] One embodiment of the present disclosure provides a medical support device, an endoscopic device, a medical support method, and a program that can prevent a user from recognizing inappropriate processing results due to object recognition processing on medical images. [Means for solving the problem]

[0014] A first aspect of the present disclosure is a medical support device that includes a processor, and displays, on a display device, processing result information that is information regarding the processing results of object recognition processing performed on a first medical image generated by capturing an image by a camera, and differs in the display manner of the processing result information depending on the similarity or difference between the first medical image and a second medical image stored in a storage.

[0015] A second aspect of the present disclosure is a medical support device according to the first aspect, in which, when the similarity exceeds a first threshold or the dissimilarity is equal to or less than a second threshold, the processing result information is displayed in a first display mode in which it is possible to identify that the similarity has exceeded the first threshold or that the dissimilarity is equal to or less than the second threshold, and, when the similarity is equal to or less than the first threshold or the dissimilarity exceeds the second threshold, the processing result information is displayed in a second display mode in which it is possible to identify that the similarity is equal to or less than the first threshold or that the dissimilarity has exceeded the second threshold.

[0016] A third aspect of the present disclosure is the medical support device according to the second aspect, in which the second display aspect has higher visibility than the first display aspect.

[0017] A fourth aspect of the present disclosure is the medical support device according to the third aspect, in which the processing result information is not displayed in the first display aspect, and the processing result information is displayed in the second display aspect.

[0018] A fifth aspect of the present disclosure is a medical support device according to any one of the first to fourth aspects, wherein the second medical image is an exceptional image defined as a medical image that is unsuitable for use in object recognition processing.

[0019] A sixth aspect of the present disclosure is the medical support device according to the fifth aspect, in which the exceptional image is an image showing an imaging subject that is inappropriate for use in object recognition processing.

[0020] A seventh aspect of the present disclosure is the medical support device according to the sixth aspect, in which the imaging target includes air bubbles and / or residue.

[0021] An eighth aspect of the present disclosure is the medical support device according to any one of the fifth to seventh aspects, in which the exceptional image is an image with a composition that is inappropriate for use in object recognition processing.

[0022] A ninth aspect of the present disclosure is a medical support device according to the eighth aspect, in which the composition is a first composition and / or a second composition, the first composition being a composition in which the size of a target included in the imaging range is larger than a first size, or the size of a target within the imaging range is smaller than a second size, and the second composition being a composition in which the target is outside the central area of ​​the imaging range.

[0023] A tenth aspect of the present disclosure is the medical support device according to any one of the fifth to ninth aspects, in which the exception image is an image of inappropriate image quality for use in object recognition processing.

[0024] An eleventh aspect of the present disclosure is the medical support device according to the tenth aspect, wherein the image quality includes out-of-focus and / or motion blur.

[0025] A twelfth aspect of the present disclosure is a medical support device relating to any one of the first to tenth aspects, wherein a plurality of second medical images having different contents are stored in the storage, and the similarity or difference is a representative value of a plurality of comparison results obtained by comparing the first medical image with each of the plurality of second medical images, or a comparison result obtained by comparing the first medical image with a second medical image located at the center of gravity of a cluster obtained by clustering the plurality of second medical images.

[0026] A thirteenth aspect of the present disclosure is a medical support device relating to any one of the first to twelfth aspects, in which the processing result information includes feature area information capable of identifying a feature area recognized by the object recognition processing, characteristic information indicating characteristics of the feature area, and / or part identification information capable of identifying the part in which the feature area is located.

[0027] A fourteenth aspect of the present disclosure is a medical support device relating to any one of the first to thirteenth aspects, in which similarity-related information, which is information related to similarity, or dissimilarity-related information, which is information related to dissimilarity, is displayed on the display device.

[0028] A fifteenth aspect of the present disclosure is a medical support device according to the fourteenth aspect, in which the similarity-related information includes similarity, information based on the similarity, a second medical image used to obtain the similarity, and / or an image based on the second medical image used to obtain the similarity, and the dissimilarity-related information includes dissimilarity, information based on the dissimilarity, the second medical image used to obtain the dissimilarity, and / or an image based on the second medical image used to obtain the dissimilarity.

[0029] A sixteenth aspect of the present disclosure is a medical support device according to any one of the first to fifteenth aspects, in which a camera is mounted on an endoscope and inserted into the body to capture images of the inside of the body.

[0030] A seventeenth aspect of the present disclosure is the medical support device according to any one of the first to sixteenth aspects, in which the object recognition process is a process of recognizing a lesion.

[0031] An 18th aspect of the present disclosure is a medical support device of any one of the first to seventeenth aspects, wherein the processing result is generated by a trained model by inputting a first medical image into the trained model.

[0032] A 19th aspect of the present disclosure is a medical support device including a processor, which does not perform object recognition processing on a first medical image generated by capturing an image with a camera and a second medical image stored in a storage when the similarity between the first medical image generated by capturing an image with a camera exceeds a first threshold value or the dissimilarity between the first medical image and the second medical image stored in a storage exceeds a second threshold value, and performs object recognition processing when the similarity between the first medical image and the second medical image is not greater than a first threshold value or the dissimilarity between the first medical image and the second medical image is not greater than a second threshold value.

[0033] A twentieth aspect of the present disclosure is an endoscope device including a medical support device according to any one of the first to nineteenth aspects and a camera.

[0034] A 21st aspect of the present disclosure is a medical support method that includes displaying, on a display device, processing result information, which is information regarding the processing results of object recognition processing performed on a first medical image generated by capturing an image by a camera, and varying the display manner of the processing result information depending on the similarity or difference between the first medical image and a second medical image stored in a storage.

[0035] A 22nd aspect of the present disclosure is a program for causing a computer to execute processing including displaying, on a display device, processing result information which is information regarding the processing results of object recognition processing performed on a first medical image generated by capturing an image by a camera, and varying the display manner of the processing result information depending on the similarity or dissimilarity between the first medical image and a second medical image stored in a storage. [Brief description of the drawings]

[0036] [Figure 1] FIG. 1 is a conceptual diagram showing an example of an aspect in which an endoscope device is used by a doctor. [Diagram 2] 1 is a conceptual diagram showing an example of an overall configuration of an endoscope apparatus. [Diagram 3] 2 is a block diagram showing an example of a hardware configuration of an electrical system of the endoscope apparatus. FIG. [Figure 4] 2 is a block diagram showing an example of main functions of a processor included in the medical support device, and an example of information stored in a storage. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the contents of an exception image set. [Figure 6] 3A and 3B are conceptual diagrams showing an example of processing contents of a recognition unit and an example of a configuration of the periphery of the recognition unit. [Figure 7] 4A and 4B are conceptual diagrams showing an example of processing contents of a determination unit and an example of a configuration of the periphery of the determination unit. [Figure 8] 11A and 11B are conceptual diagrams showing an example of a screen when processing result information is displayed on the screen and an example of a screen when the processing result information is not displayed on the screen. [Figure 9]13 is a flowchart showing an example of the flow of a medical support process. [Figure 10] 13 is a conceptual diagram showing an example of a screen in which processing result information is displayed on the screen regardless of whether the similarity is equal to or less than a first threshold value. FIG. [Figure 11] 13A and 13B are conceptual diagrams showing modified examples of the processing content of the determination unit and examples of aspects of the periphery of the determination unit. [Figure 12] 13 is a conceptual diagram showing an example of dissimilarity-related information, etc., displayed in the second display area when the dissimilarity is equal to or smaller than a second threshold value. FIG. [Figure 13] 11 is a conceptual diagram showing an example of a process performed by a determination unit when the determination unit compares a representative similarity with a first threshold value and an example of a configuration of the periphery of the determination unit. FIG. [Figure 14] 13 is a conceptual diagram showing an example of the processing content of the determination unit when the determination unit compares an exceptional image acquired from the center of gravity of a cluster with a frame. FIG. [Figure 15] 11 is a conceptual diagram showing an example of a process content in a case where the recognition process is controlled according to a determination result by a determination unit. FIG. [Figure 16] A conceptual diagram showing an example of a series of processes in which a processor included in a computer issues a processing execution request to an external device via a network, the external device executes processing in response to the processing execution request, and the processor included in the computer receives the processing result from the external device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0037] Hereinafter, examples of embodiments of a medical support device, an endoscope device, a medical support method, and a program according to the present disclosure will be described with reference to the accompanying drawings.

[0038] First, the terms used in the following description will be explained.

[0039] CPU is an abbreviation for "Central Processing Unit". GPU is an abbreviation for "Graphics Processing Unit". GPGPU is an abbreviation for "General-Purpose computing on Graphics Processing Units". APU is an abbreviation for "Accelerated Processing Unit". TPU is an abbreviation for "Tensor Processing Unit". RAM is an abbreviation for "Random Access Memory". NVM is an abbreviation for "Non-volatile memory". EEPROM is an abbreviation for "Electrically Erasable Programmable Read-Only Memory". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". SSD is an abbreviation for "Solid State Drive". USB is an abbreviation for "Universal Serial Bus". HDD is an abbreviation for "Hard Disk Drive". EL is an abbreviation for "Electro-Luminescence". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". AI is an abbreviation for "Artificial Intelligence". BLI is an abbreviation for "Blue Light Imaging". LCI is an abbreviation for "Linked Color Imaging". I / F is an abbreviation for "Interface". SSL is an abbreviation for "Sessile Serrated Lesion". LAN is an abbreviation for "Local Area Network". WAN is an abbreviation for "Wide Area Network".5G is an abbreviation for "5th Generation Mobile Communication System."

[0040] In the following description, a coded processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Also, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU, a GPU, a GPGPU, an APU, and a TPU.

[0041] In the following description, a signed memory is a memory, such as a RAM, in which information is temporarily stored and which is used as a working memory by a processor.

[0042] In the following description, a storage with a symbol is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory, magnetic disks, and magnetic tapes. Another example of storage is cloud storage.

[0043] In the following embodiments, the external I / F with a symbol controls the transmission and reception of various information between multiple devices connected to each other. An example of the external I / F is a USB interface. A communication I / F including a communication processor and an antenna may be applied to the external I / F. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G, Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0044] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0045] Fig. 1 is a conceptual diagram showing an example of a mode in which an endoscope device 10 is used. As shown in Fig. 1, the endoscope device 10 is used by a doctor 12 in an endoscopic examination or the like. The endoscopic examination is assisted by staff such as a nurse 14.

[0046] The endoscope device 10 is communicatively connected to a communication device (not shown), and information obtained by the endoscope device 10 is transmitted to the communication device. Examples of the communication device include a server that manages various information such as electronic medical records, a personal computer, and / or a tablet terminal. The communication device receives the information transmitted from the endoscope device 10 and executes processing using the received information (e.g., processing to store the information in an electronic medical record, etc.).

[0047] The endoscopic device 10 includes an endoscope 16, a display device 18, a light source device 20, a control device 22, and a medical support device 24. In this embodiment, the endoscopic device 10 is an example of an "endoscopic device" according to the present disclosure, and the display device 18 is an example of a "display device" according to the present disclosure.

[0048] The endoscope device 10 is a modality for performing medical treatment on a large intestine 28 contained in the body of a subject 26 (e.g., a patient) using an endoscope scope 16. In this embodiment, the large intestine 28 is an object to be observed by a doctor 12.

[0049] The endoscope 16 is used by the doctor 12 and inserted into the body of a subject 26. In this embodiment, the endoscope 16 is inserted into a large intestine 28, which is a hollow organ of the subject 26.

[0050] The endoscope device 10 causes the endoscope scope 16 inserted into the large intestine 28 of the subject 26 to capture images of the inside of the large intestine 28 of the subject 26, and also performs various medical procedures on the large intestine 28 as necessary.

[0051] The endoscope device 10 captures an image of the inside of the large intestine 28 of the subject 26, thereby acquiring and outputting an image showing the state inside the large intestine 28. In this embodiment, the endoscope device 10 is an endoscope having an optical imaging function for capturing an image of reflected light obtained by irradiating the inside of the large intestine 28 with light 30 and reflecting it off a wall 32 of the large intestine 28.

[0052] Although an endoscopic examination of the large intestine 28 is illustrated here, this is merely one example, and the present disclosure also applies to an endoscopic examination of a hollow organ such as the esophagus, stomach, duodenum, or trachea.

[0053] The light source device 20, the control device 22, and the medical support device 24 are installed on a wagon 34. The wagon 34 has a plurality of platforms arranged vertically, and the medical support device 24, the control device 22, and the light source device 20 are installed from the lower platform to the upper platform. In addition, the display device 18 is installed on the top platform of the wagon 34.

[0054] The control device 22 controls the entire endoscope device 10. The medical support device 24, under the control of the control device 22, performs various image processing on the images obtained by imaging the intestinal wall 32 by the endoscope scope 16.

[0055] The display device 18 displays various information including images. Examples of the display device 18 include a liquid crystal display and an EL display. Also, instead of the display device 18 or together with the display device 18, a tablet terminal with a display may be used.

[0056] The display device 18 displays a screen 35. The screen 35 includes a plurality of display areas. The plurality of display areas are arranged side by side within the screen 35. In the example shown in FIG. 1, a first display area 35A, a second display area 35B, and a third display area 35C are shown as examples of the plurality of display areas. The size of the first display area 35A is larger than the size of the second display area 35B. The first display area 35A is used as a main display area, and the second display area 35B is used as a sub-display area. The size relationship between the first display area 35A and the second display area 35B is not limited thereto, and may be any size relationship that fits within the screen 35. The third display area 35C is a display area other than the first display area 35A and the second display area 35B within the screen 35, and is used as a sub-display area.

[0057] An endoscopic moving image 39 is displayed in the first display region 35A. The endoscopic moving image 39 is a moving image acquired by imaging an intestinal wall 32 in the large intestine 28 of the subject 26 with the endoscope scope 16. In the example shown in Fig. 1, a moving image showing the intestinal wall 32 is shown as an example of the endoscopic moving image 39.

[0058] The intestinal wall 32 shown in the endoscopic video 39 includes a lesion 42 (e.g., one lesion 42 in the example shown in FIG. 1) as an area of ​​interest (i.e., an area to be observed) gazed upon by the doctor 12, and the doctor 12 can visually recognize the appearance of the intestinal wall 32 including the lesion 42 through the endoscopic video 39. In this embodiment, the lesion 42 is an example of a "characteristic area" and a "lesion" according to the present disclosure.

[0059] There are various types of lesions 42, and examples of the types of lesions 42 include neoplastic polyps and non-neoplastic polyps. Examples of the types of neoplastic polyps include adenomatous polyps (e.g., SSL). Examples of the types of non-neoplastic polyps include hamartomatous polyps, hyperplastic polyps, and inflammatory polyps. Note that the types exemplified here are types that are assumed in advance as types of lesions 42 when an endoscopic examination is performed on the large intestine 28, and the types of lesions 42 will differ if the organ in which the endoscopic examination is performed is different.

[0060] In this embodiment, for ease of explanation, an example is given in which one lesion 42 is captured in the endoscopic video 39, but the present disclosure is not limited to this, and the present disclosure also applies when multiple lesions 42 are captured in the endoscopic video 39.

[0061] In this embodiment, a lesion 42 is illustrated, however, this is merely one example, and the area of ​​interest (i.e., the area to be observed) gazed upon by the doctor 12 may be a characteristic area having some unique feature, such as an organ (e.g., the duodenal papilla), a mark, an artificial treatment device (e.g., an artificial clip), or a treated area (e.g., an area where traces remain after the removal of a polyp, etc.), etc.

[0062] The image displayed in first display area 35A is one frame 40 included in a moving image that includes a plurality of frames 40 arranged in chronological order. In other words, in first display area 35A, a plurality of frames 40 arranged in chronological order are displayed at a default frame rate (e.g., several tens of frames per second).

[0063] An example of a moving image displayed in the first display area 35A is a moving image in a live view format. The live view format is merely one example, and a moving image that is temporarily stored in a memory or the like and then displayed, such as a moving image in a post-view format, may also be used. Furthermore, each frame included in a moving image for recording stored in a memory or the like may be reproduced and displayed on the screen 35 (for example, the first display area 35A) as an endoscopic moving image 39.

[0064] Within the screen 35, the second display area 35B is adjacent to the first display area 35A, and is displayed in the lower right corner as viewed from the front within the screen 35. The display position of the second display area 35B may be anywhere within the screen 35 of the display device 18, but it is preferable that it is displayed in a position where it can be contrasted with the endoscopic moving image 39.

[0065] In the second display area 35B, auxiliary information 44 that assists the doctor 12 in making medical decisions during an endoscopic examination is displayed. The auxiliary information 44 is information that is referred to by the doctor 12. Examples of the auxiliary information 44 include various types of information related to the subject 26 into whose body the endoscope 16 is inserted, and / or various types of information obtained by performing medical support processing, which will be described later.

[0066] The third display area 35C displays visual information obtained by performing AI-based processing on the frame 40. This will be described in detail later.

[0067] Fig. 2 is a conceptual diagram showing an example of the overall configuration of the endoscope device 10. As shown in Fig. 2, the endoscope scope 16 includes an operation section 46 and an insertion section 48. The insertion section 48 is partially curved by operating the operation section 46. The insertion section 48 is inserted into the large intestine 28 (see Fig. 1) while curving in accordance with the shape of the large intestine 28 (see Fig. 1) in accordance with the operation of the operation section 46 by the doctor 12 (see Fig. 1).

[0068] The distal end 50 of the insertion section 48 is provided with a camera 52, an illumination device 54, and a treatment tool opening 56. The camera 52 and the illumination device 54 are provided on the distal end surface 50A of the distal end 50. Note that, although an example in which the camera 52 and the illumination device 54 are provided on the distal end surface 50A of the distal end 50 is given here, this is merely one example, and the camera 52 and the illumination device 54 may be provided on the side surface of the distal end 50, so that the endoscope 16 is configured as a side-viewing scope.

[0069] The camera 52 is mounted on the endoscope 16, and is inserted into the body cavity of the subject 26 to capture an image of an observation target region, thereby generating a frame 40. In this embodiment, the camera 52 captures an image of the inside of the subject 26 (e.g., inside the large intestine 28) to generate an endoscopic video image 39 including a plurality of frames 40 in a time series. An example of the camera 52 is a CMOS camera. However, this is merely an example, and other types of cameras such as a CCD camera may also be used. In this embodiment, the camera 52 is an example of a "camera" according to the present disclosure, and the frame 40 is an example of a "first medical image" according to the present disclosure.

[0070] The illumination device 54 has illumination windows 54A and 54B. The illumination device 54 irradiates light 30 (see FIG. 1) through the illumination windows 54A and 54B. Examples of the light 30 irradiated from the illumination device 54 include visible light (e.g., white light) and non-visible light (e.g., near-infrared light). The illumination device 54 also irradiates special light through the illumination windows 54A and 54B. Examples of the special light include light for BLI and / or light for LCI. The camera 52 captures an image of the inside of the large intestine 28 by an optical method while the light 30 is irradiated inside the large intestine 28 by the illumination device 54.

[0071] The treatment tool opening 56 is an opening for allowing a treatment tool 58 to protrude from the distal end portion 50. The treatment tool opening 56 is also used as a suction port for sucking blood, internal waste, and the like, and as a delivery port for delivering fluid.

[0072] A treatment tool insertion port 60 is formed in the operation section 46, and the treatment tool 58 is inserted into the insertion section 48 from the treatment tool insertion port 60. The treatment tool 58 passes through the insertion section 48 and protrudes to the outside from the treatment tool opening 56. In the example shown in FIG. 2, a puncture needle is shown as the treatment tool 58 protruding from the treatment tool opening 56. Here, a puncture needle is shown as the treatment tool 58, but this is merely one example, and the treatment tool 58 may be a grasping forceps, a papillotomy knife, a snare, a catheter, a guidewire, a cannula, and / or a puncture needle with a guide sheath, etc.

[0073] The endoscope 16 is connected to a light source device 20 and a control device 22 via a universal cord 62. A medical support device 24 and a reception device 64 are connected to the control device 22. In addition, a display device 18 is connected to the medical support device 24. That is, the control device 22 is connected to the display device 18 via the medical support device 24.

[0074] In this example, the medical support device 24 is shown as an external device for expanding the functions performed by the control device 22, and therefore an example is shown in which the control device 22 and the display device 18 are indirectly connected via the medical support device 24, but this is merely one example. For example, the display device 18 may be directly connected to the control device 22. In this case, for example, the function of the medical support device 24 may be installed in the control device 22, or the control device 22 may be equipped with a function to cause a server (not shown) to execute the same process as the process executed by the medical support device 24 (for example, the medical support process described later) and to receive and use the results of the process by the server.

[0075] The reception device 64 receives instructions from the doctor 12 and outputs the received instructions as an electrical signal to the control device 22. Examples of the reception device 64 include a keyboard, a mouse, a touch panel, a foot switch, a microphone, and / or a remote control device.

[0076] The control device 22 controls the light source device 20, exchanges various signals with the camera 52, and exchanges various signals with the medical support device 24.

[0077] The light source device 20 emits light under the control of the control device 22, and supplies the light to the illumination device 54. A light guide is built into the illumination device 54, and the light supplied from the light source device 20 passes through the light guide and is irradiated from illumination windows 54A and 54B. The control device 22 causes the camera 52 to capture an image, acquires an endoscopic video image 39 (see FIG. 1) from the camera 52, and outputs it to a predetermined output destination (for example, the medical support device 24).

[0078] The medical support device 24 performs various types of image processing on the endoscopic moving image 39 input from the control device 22 to provide support for medical care (here, as an example, an endoscopic examination). The medical support device 24 outputs the endoscopic moving image 39 that has been subjected to various types of image processing to a predetermined output destination (for example, the display device 18).

[0079] Although an example in which the endoscopic moving image 39 output from the control device 22 is output to the display device 18 via the medical support device 24 has been described here, this is merely one example. For example, the control device 22 and the display device 18 may be connected, and the endoscopic moving image 39 that has been subjected to image processing by the medical support device 24 may be displayed on the display device 18 via the control device 22.

[0080] Fig. 3 is a block diagram showing an example of a hardware configuration of an electrical system of the endoscope device 10. As shown in Fig. 3, the control device 22 includes a computer 66, a bus 68, and an external I / F 70. The computer 66 includes a processor 72, a memory 74, and a storage 76. The processor 72, the memory 74, the storage 76, and the external I / F 70 are connected to the bus 68. The processor 72 controls the entire control device 22. The memory 74 and the storage 76 are used by the processor 72.

[0081] The external I / F 70 is responsible for transmitting and receiving various types of information between the processor 72 and one or more devices (hereinafter, also referred to as “first external devices”) that exist outside the control device 22.

[0082] The camera 52 is connected to the external I / F 70 as one of the first external devices, and the external I / F 70 is responsible for transmitting and receiving various types of information between the camera 52 and the processor 72. The processor 72 controls the camera 52 via the external I / F 70. The processor 72 also acquires, via the external I / F 70, an endoscopic moving image 39 (see FIG. 1) obtained by the camera 52 capturing an image of the inside of the large intestine 28 (see FIG. 1).

[0083] The light source device 20 is connected to the external I / F 70 as one of the first external devices, and the external I / F 70 controls the exchange of various information between the light source device 20 and the processor 72. The light source device 20 supplies light to the illumination device 54 under the control of the processor 72. The illumination device 54 irradiates the light supplied from the light source device 20.

[0084] A reception device 64 is connected to the external I / F 70 as one of the first external devices, and the processor 72 acquires instructions accepted by the reception device 64 via the external I / F 70 and executes processing according to the acquired instructions.

[0085] The medical support device 24 includes a computer 78 and an external I / F 80. The computer 78 includes a processor 82, a memory 84, and a storage 86. The processor 82, the memory 84, the storage 86, and the external I / F 80 are connected to a bus 88. In this embodiment, the medical support device 24 is an example of a "medical support device" according to the present disclosure, the computer 78 is an example of a "computer" according to the present disclosure, and the processor 82 is an example of a "processor" according to the present disclosure.

[0086] The hardware configuration of the computer 78 (i.e., the processor 82, the memory 84, and the storage 86) is basically the same as the hardware configuration of the computer 66, so a description of the hardware configuration of the computer 78 will be omitted here.

[0087] The external I / F 80 controls the exchange of various information between the processor 82 and one or more devices (hereinafter, also referred to as “second external devices”) that exist outside the medical support device 24.

[0088] The control device 22 is connected to the external I / F 80 as one of the second external devices. In the example shown in Fig. 3, the external I / F 70 of the control device 22 is connected to the external I / F 80. The external I / F 80 controls the exchange of various information between the processor 82 of the medical support device 24 and the processor 72 of the control device 22. For example, the processor 82 acquires endoscopic moving images 39 (see Fig. 1) from the processor 72 of the control device 22 via the external I / Fs 70 and 80, and performs various image processing on the acquired endoscopic moving images 39.

[0089] The display device 18, which serves as one of the second external devices, is connected to the external I / F 80. The processor 82 controls the display device 18 via the external I / F 80 to cause the display device 18 to display various information (e.g., an endoscopic moving image 39 that has been subjected to various image processing, etc.).

[0090] In recent years, there has been progress in the development of a technology in which a trained model optimized by performing machine learning on a model (e.g., a neural network) performs object recognition processing on each frame 40 included in an endoscopic video 39 to recognize a lesion 42, and the recognition result or information based on the recognition result is displayed on a screen 35.

[0091] However, at the stage of operating the trained model (i.e., when the trained model is used in endoscopic examinations), if an image that is not included in the training data used in the machine learning (i.e., an image containing an object that was not anticipated in the learning stage) is input to the trained model, the trained model may make a false recognition.

[0092] For example, the trained model may erroneously recognize air bubbles and / or residues (e.g., residues resulting from a medical procedure) in the frame 40 as the lesion 42, or may not recognize the lesion 42 even though it exists. In addition, a large difference between the composition of the frame 40 and the composition of the image used in the machine learning (e.g., the size of the lesion 42 in the frame 40 is too large or too small) may also be a cause of the erroneous recognition. In addition, a cause of the erroneous recognition may also be the image quality of the frame 40 being too far from the image quality of the image used in the machine learning. In an endoscopic examination, if a result erroneously recognized by the trained model is displayed on the screen 35, it becomes difficult for the doctor 12 to accurately perform various medical procedures.

[0093] As a method for eliminating such misrecognition, a method of collecting a plurality of images that are likely to cause misrecognition and retraining a trained model using the collected plurality of images is conceivable, but retraining takes time and costs. Also, a method of switching from an existing trained model to a trained model trained with images that are likely to cause misrecognition is conceivable, but in this case too, training takes time and costs, and switching the trained model also takes time and costs.

[0094] In view of such circumstances, in this embodiment, as an example shown in Fig. 4, medical support processing is performed by a processor 82 of the medical support device 24. Note that Fig. 4 is a block diagram showing an example of the main functions of the processor 82 included in the medical support device 24 and an example of information stored in a storage 86.

[0095] A medical support program 90 is stored in the storage 86. The medical support program 90 is an example of a "program" according to the present disclosure. The processor 82 reads the medical support program 90 from the storage 86 and executes the read medical support program 90 on the memory 84 to perform medical support processing. The medical support processing is realized by the processor 82 operating as a recognition unit 82A, a determination unit 82B, and a control unit 82C in accordance with the medical support program 90 executed on the memory 84.

[0096] The storage 86 stores a recognition model 92 and an exception image set 94. As will be described in detail later, the recognition model 92 is used by the recognition unit 82A, and the exception image set 94 is used by the determination unit 82B. A frame 40 is input to the recognition model 92, which recognizes a lesion 42 shown in the input frame 40 and outputs the recognition result. In this embodiment, the recognition model 92 is an example of a "trained model" according to the present disclosure.

[0097] FIG. 5 is a conceptual diagram showing an example of the contents of the exception image set 94. As shown in FIG. 5, the exception image set 94 includes a plurality of exception images 96 with different contents. The exception image 96 is an image determined as a medical image (here, as an example, an endoscopic image generated by imaging the inside of the large intestine 28 by the camera 52) that is inappropriate for use in object recognition processing (for example, processing in which the recognition model 92 recognizes the lesion 42). The medical image that is inappropriate for use in object recognition processing refers to, for example, a medical image in which a target object (for example, the lesion 42) is not recognized by the object recognition processing, or an object other than the target object and / or noise is erroneously recognized as the target object. The exception image 96 may be an image generated by imaging by a camera, or may be an image generated by a generation AI. For example, the size and shape of the outer shape of the exception image 96 are the same as the size and shape of the frame 40. However, this is merely an example, and the technology of the present disclosure is valid even if the size and shape of the outer shape of the exception image 96 are different from the size and shape of the frame 40. In this embodiment, the exception image 96 is an example of a "second medical image" and an "exception image" according to the present disclosure.

[0098] The exceptional image set 94 includes an inappropriate capture target image set 94A, an inappropriate composition image set 94B, and an inappropriate image quality image set 94C.

[0099] The inappropriate imaging target image set 94A includes a plurality of inappropriate imaging target images 96A having different contents. The concept of the exception image 96 includes the inappropriate imaging target image 96A. The inappropriate imaging target image 96A refers to an image in which an imaging target 98 that is inappropriate for use in an object recognition process (for example, a process in which the recognition model 92 recognizes the lesion 42) is shown (i.e., an image in which an object that induces erroneous recognition in the object recognition process is shown). An example of the inappropriate imaging target 98 includes air bubbles and / or residue. An example of the residue includes residue generated by performing a medical treatment. The residue generated by performing a medical treatment includes, for example, blood, medicine, treatment scar, marking area, treatment tool, and / or smoke. Here, the imaging target 98 is an example of the "imaging target", "air bubbles", and "residue" according to the present disclosure.

[0100] The inappropriate composition image set 94B includes an inappropriate size image set 94B1 and an inappropriate position image set 94B2. The inappropriate size image set 94B1 includes a plurality of inappropriate size images 96B and a plurality of inappropriate position images 96C. Each of the plurality of inappropriate size images 96B and the plurality of inappropriate position images 96C is an image with an inappropriate composition within the imaging range of a target (lesion 100 in the example shown in FIG. 5) included in the imaging range of a camera used for imaging to obtain the exception image 96 for use in an object recognition process (e.g., a process in which the recognition model 92 recognizes the lesion 42) (i.e., an image with a composition in which the target (e.g., the lesion 42) is not recognized by the object recognition process, or an object other than the target and / or noise is erroneously recognized as the target).

[0101] The contents of the multiple inappropriately sized images 96B are different from one another. The concept of the exception image 96 includes the inappropriately sized image 96B. The inappropriately sized image 96B refers to an image having a composition in which the size of a target included in the imaging range of a camera used for imaging to obtain the exception image 96 is too large or too small.

[0102] An example of an image with a composition in which the size of a target included in the imaging range of a camera used for imaging to obtain inappropriate size image 96B is too large is an image with a composition in which the size of a target included in the imaging range is larger than a first size (for example, an image with a composition in which the size of a target within the imaging range is so large that a target (e.g., lesion 42) is not recognized by object recognition processing or objects other than the target and / or noise are erroneously recognized as a target). An example of the first size is an upper limit value of a size range derived in advance by testing and / or computer simulation as a size range in which erroneous recognition will not occur when used in object recognition processing.

[0103] An example of an image with a composition in which the size of a target included in the imaging range of a camera used for imaging to obtain inappropriate size image 96B is too small is an image with a composition in which the size of a target included in the imaging range is smaller than a second size (for example, an image with a composition in which the size of a target within the imaging range is so small that a target (e.g., lesion 42) is not recognized by object recognition processing or objects other than the target and / or noise are erroneously recognized as a target). An example of the second size is the lower limit of a size range derived in advance by testing and / or computer simulation as a size range in which erroneous recognition will not occur when used in object recognition processing.

[0104] Note that here, a composition in which the size of a target included in the imaging range is larger than a first size, and a composition in which the size of a target included in the imaging range is smaller than a second size, are examples of a “first composition” according to the present disclosure.

[0105] The contents of the multiple inappropriate position images 96C are different from each other. The concept of the exception image 96 includes the inappropriate position image 96C. The inappropriate position image 96C refers to an image with a composition in which a target (lesion 100 in the example shown in FIG. 5) included in the imaging range of the camera used for imaging to obtain the exception image 96 is out of the central region 99 of the imaging range. If a target included in the imaging range of the camera used for imaging to obtain the exception image 96 is out of the central region 99 of the imaging range, the target is affected by the aberration of the lens, so that there is a risk that the target (e.g., lesion 42) will not be recognized by the object recognition process, or an object other than the target and / or noise will be erroneously recognized as the target. For this reason, here, multiple inappropriate position images 96C with different contents are prepared as an inappropriate position image set 94B2. Here, a composition in which a target included in the imaging range of the camera used for imaging to obtain the inappropriate position image 96C is out of the central region 99 of the imaging range is an example of a "second composition" according to the present disclosure.

[0106] Here, as examples of inappropriate compositions, a composition in which the size of the target within the imaging range is inappropriate and a composition in which the position of the target within the imaging range is inappropriate have been given; however, this is merely an example, and a composition in which the imaging direction of the target (i.e. the direction in which the target is captured in the image) is inappropriate may also be used. In addition, an image having a combination of two or more of the following compositions may be used as the exception image 96: an inappropriate size of the target within the imaging range (i.e., a size of the target within the imaging range where the target (e.g., lesion 42) is not recognized by the object recognition process or an object other than the target and / or noise is erroneously recognized as the target), an inappropriate position of the target within the imaging range (i.e., a position of the target within the imaging range where the target (e.g., lesion 42) is not recognized by the object recognition process or an object other than the target and / or noise is erroneously recognized as the target), and an inappropriate imaging direction relative to the target (i.e., an imaging direction relative to the target within the imaging range where the target (e.g., lesion 42) is not recognized by the object recognition process or an object other than the target and / or noise is erroneously recognized as the target).

[0107] The inappropriate image quality image set 94C includes a plurality of inappropriate image quality images 96D that are different from each other in the contents that are shown. The concept of the exception image 96 includes the inappropriate image quality image 96D. The inappropriate image quality image 96D refers to an image of inappropriate image quality (for example, an image of image quality in which a target (for example, a lesion 42) is not recognized by the object recognition process or an object other than the target and / or noise is erroneously recognized as the target) for use in the object recognition process (for example, a process in which the recognition model 92 recognizes the lesion 42). Examples of inappropriate image quality include out-of-focus and / or motion blur. The image quality of the inappropriate image quality image 96D includes image quality derived in advance by tests and / or computer simulations as image quality that causes erroneous recognition when used in the object recognition process. Note that, although out-of-focus and / or motion blur are given here as examples of inappropriate image quality, this is merely an example, and may be artifacts, ghosts, flares, whiteout, and / or blackout.

[0108] Here, for ease of explanation, the inappropriately captured image 96A, the inappropriately sized image 96B, the inappropriately positioned image 96C, and the inappropriate image quality 96D are shown separately as examples, but this is merely one example, and the exception image 96 may be an image whose content combines at least two or more of the content appearing in the inappropriately captured image 96A, the content appearing in the inappropriately sized image 96B, the content appearing in the inappropriately positioned image 96C, and the content appearing in the inappropriate image quality 96D.

[0109] Fig. 6 is a conceptual diagram showing an example of the processing contents of the recognition unit 82A and an example of the surroundings of the recognition unit 82A. As shown in Fig. 6, the recognition unit 82A, the determination unit 82B, and the control unit 82C acquire, in units of frames, each of a plurality of frames 40 in time series included in the endoscopic moving image 39 generated by the camera 52 capturing images at an imaging frame rate (e.g., several tens of frames / second) from the camera 52. The timing at which the recognition unit 82A, the determination unit 82B, and the control unit 82C acquire the frames 40 is synchronized.

[0110] The control unit 82C outputs the endoscopic moving image 39 to the display device 18. For example, the control unit 82C displays the endoscopic moving image 39 as a live view image in the first display area 35A. That is, every time the control unit 82C acquires a frame 40 from the camera 52, the control unit 82C displays the acquired frame 40 in sequence in the first display area 35A according to a display frame rate (e.g., several tens of frames / second). The control unit 82C also displays auxiliary information 44 in the second display area 35B. The control unit 82C also updates the display contents of the second display area 35B (e.g., auxiliary information 44) in accordance with the display contents of the first display area 35A.

[0111] The recognition unit 82A recognizes a lesion 42 in the endoscopic video 39 based on the endoscopic video 39 acquired from the camera 52. That is, the recognition unit 82A recognizes the lesion 42 appearing in the frame 40 by sequentially performing a recognition process 95, which is an example of the object recognition process described above, on each of a plurality of frames 40 in chronological order included in the endoscopic video 39 acquired from the camera 52.

[0112] For example, the recognition unit 82A recognizes the geometric characteristics of the lesion 42 (e.g., position, size, shape, etc.), the type of the lesion 42, and the type of the lesion 42 (e.g., pedunculated, subpedunculated, sessile, surface elevated, surface flat, surface depressed, etc.). Furthermore, the recognition unit 82A recognizes a part (hereinafter simply referred to as a "part") in the endoscopic video image 39 based on the endoscopic video image 39 acquired from the camera 52. Here, a part refers to a part of the large intestine 28. Examples of parts of the large intestine 28 include the anal canal, the lower rectum, the upper rectum, the sigmoid colon, the descending colon, the transverse colon, the ascending colon, and the ileocecal area.

[0113] The recognition process 95 is performed by the recognition unit 82A on the acquired frame 40 each time the frame 40 is acquired. The recognition process 95 is a process for recognizing the lesion 42 and the body part using an AI method. Here, the recognition process 95 is a process using a recognition model 92. The recognition process 95 is an example of the "object recognition process" according to the present disclosure.

[0114] The recognition model 92 is a trained model for object recognition by an AI bounding box method. The recognition model 92 is optimized by performing machine learning using training data on a neural network. The training data is a data set including a plurality of data (i.e., data for a plurality of frames) in which example data and correct answer data are associated with each other.

[0115] The example data is an image that assumes frame 40. A first example of an image that assumes frame 40 is an image obtained by actually capturing an image of the inside of the large intestine with a camera. A second example of an image that assumes frame 40 is a virtually created image (e.g., an image generated by a generation AI). The correct answer data is correct answer data (i.e., annotations) for the example data. Here, annotations that identify the geometric characteristics, type, model, and location of the lesion depicted in the image used as the example data are used as an example of correct answer data.

[0116] The recognition unit 82A acquires a frame 40 from the camera 52, and inputs the acquired frame 40 to the recognition model 92. As a result, each time a frame 40 is input, the recognition model 92 recognizes the lesion 42 and the area shown in the input frame 40, and generates and outputs a recognition result 102. In this embodiment, the recognition result 102 is an example of a "processing result" according to the present disclosure.

[0117] The recognition unit 82A outputs the recognition result 102 itself to the display device 18 or the like in synchronization with the display of the frame 40 in the first display region 35A, or outputs information generated based on the recognition result 102 to the display device 18 or the like in synchronization with the display of the frame 40 in the first display region 35A. The recognition result 102 itself or information generated based on the recognition result 102 is displayed on the screen 35 of the display device 18 (for example, the first display region 35A, the second display region 35B, and / or the third display region 35C). The display of the recognition result 102 itself and the display of the information generated based on the recognition result 102 are updated in synchronization with the display timing of the endoscopic moving image 39 displayed in the first display region 35A. That is, the display of the recognition result 102 itself and the display of the information generated based on the recognition result 102 are updated according to the display frame rate applied to the first display region 35A.

[0118] The recognition result 102 includes lesion presence / absence information 104. The lesion presence / absence information 104 is information indicating whether or not a lesion 42 is captured in the frame 40 input to the recognition model 92. The recognition result 102 also includes geometric property information 106, a lesion location map 108, lesion feature information 110, and the like.

[0119] The geometric characteristic information 106 is information (e.g., coordinates) capable of identifying the shape, size, and position of the lesion 42 within the frame 40. The geometric characteristic information 106 or information based on the geometric characteristic information 106 may be displayed in the first display region 35A, may be displayed as a piece of auxiliary information 44 in the second display region 35B, or may be displayed in the third display region 35C.

[0120] The lesion location map 108 is a map capable of identifying the location of the lesion 42 within the frame 40. The lesion characteristic information 110 is information capable of identifying the type and model of the lesion 42 appearing in the frame 40 that has been input to the recognition model 92.

[0121] The geometric characteristics of the lesion location map 108 (e.g., the shape and size of the outer contour) correspond to the geometric characteristics of the frame 40 (e.g., the shape and size of the outer contour). The lesion location map 108 is displayed on the screen 35 by the control unit 82C. In this embodiment, the lesion location map 108 is displayed in the second display area 35B as one of the auxiliary information 44 by the control unit 82C. For example, the display and non-display of the lesion location map 108 is switched according to an instruction and / or various conditions received by the reception device 64. The lesion location map 108 displayed on the screen 35 is updated according to the display frame rate applied to the first display area 35A. With this configuration, the doctor 12 can grasp the approximate position of the lesion 42 in the endoscopic moving image 39 displayed in the first display area 35A by referring to the lesion location map 108 displayed in the second display area 35B while observing the endoscopic moving image 39 displayed in the first display area 35A.

[0122] The control unit 82C displays the lesion characteristic information 110 on the screen 35. For example, the lesion characteristic information 110 may be displayed as one piece of auxiliary information 44 in the second display area 35B, or in the first display area 35A and / or the third display area 35C.

[0123] The recognition result 102 also includes body part information 112. The body part information 112 is information that can identify a body part captured in the frame 40. The body part information 112 may be displayed in the first display area 35A, may be displayed as a piece of auxiliary information 44 in the second display area 35B, or may be displayed in the third display area 35C.

[0124] The determination unit 82B determines whether or not to perform display based on the recognition result 102 based on the frame 40 acquired from the camera 52, and outputs the determination result to the control unit 82C. The control unit 82C performs or does not perform display based on the recognition result 102 on the display device 18 based on the determination result input from the determination unit 82B.

[0125] Fig. 7 is a conceptual diagram showing an example of the processing content of the determination unit 82B and an example of the surroundings of the determination unit 82B. As shown in Fig. 7, the control unit 82C acquires the recognition result 102 from the recognition unit 82A, and controls the display device 18 so that processing result information 113, which is information related to the acquired recognition result 102, is displayed on the screen 35. The processing result information 113 includes, for example, information capable of identifying the lesion 42, information indicating the characteristics of the lesion 42, and / or information capable of identifying the site where the lesion 42 is present.

[0126] Every time the determination unit 82B obtains a frame 40 from the camera 52, it obtains all of the exception images 96 from the exception image set 94 stored in the storage 86. Then, the determination unit 82B compares the frame 40 obtained from the camera 52 with each of all of the exception images 96 obtained from the exception image set 94, thereby calculating a similarity 114 between the frame 40 and the exception image 96 for each of the multiple exception images 96. In this embodiment, the similarity 114 is an example of the "similarity" and "comparison result" according to the present disclosure.

[0127] Here, for convenience of explanation, an example is given in which frame 40 is compared with each of all the exception images 96 acquired from the exception image set 94, but this is merely one example, and the objects to be compared with frame 40 do not have to be all the exception images 96 acquired from the exception image set 94. The objects to be compared with frame 40 may be a plurality of exception images 96 selected from the exception image set 94 in accordance with instructions and / or various conditions accepted by the acceptance device 64 (for example, a plurality of exception images 96 included in the inappropriate imaging target image set 94A, the inappropriate size image set 94B1, the inappropriate position image set 94B2, and / or the inappropriate image quality image set 94C).

[0128] The similarity 114 is the degree of similarity between the frame 40 and the exception image 96. For example, if the frame 40 and the exception image 96 are a perfect match, the similarity 114 indicates the highest value (e.g., 100%), and the greater the difference between the frame 40 and the exception image 96, the lower the similarity 114. The smaller the similarity 114, the higher the reliability of the recognition result 102 obtained by performing the recognition process 95 on the frame 40 used to calculate the similarity 114 (i.e., the frame 40 compared with the exception image 96), and the smaller the similarity 114, the lower the reliability of the recognition result 102.

[0129] Therefore, the control unit 82C changes the display mode of the processing result information 113 according to the similarity 114 calculated by the determination unit 82B. This will be explained in more detail below.

[0130] The determination unit 82B outputs a non-display instruction signal 116 to the control unit 82C when all similarities 114 calculated by comparing the frame 40 acquired from the camera 52 with each of all the exception images 96 acquired from the exception image set 94 exceed a first threshold TH1 (e.g., 80%). The non-display instruction signal 116 is a signal that instructs not to display the processing result information 113. The first threshold TH1 may be a fixed value or a variable value (e.g., a value that is changed according to an instruction accepted by the acceptance device 64 and / or various conditions). The first threshold TH1 is an example of a "first threshold" according to the present disclosure.

[0131] On the other hand, when all similarities 114 calculated by comparing the frame 40 acquired from the camera 52 with each of the exception images 96 acquired from the exception image set 94 are equal to or smaller than the first threshold value TH1, the determination unit 82B outputs a display instruction signal 118 to the control unit 82C. The display instruction signal 118 is a signal that instructs the display of the processing result information 113.

[0132] The control unit 82C displays the processing result information 113 in a first display mode or a second display mode according to the similarity 114. The first display mode is a display mode in which it is possible to specify that the similarity 114 has exceeded the first threshold value TH1. In the first display mode, the processing result information 113 is not displayed. The second display mode is a display mode that has higher visibility than the first display mode and in which it is possible to specify that the similarity 114 is equal to or less than the first threshold value TH1. In the second display mode, the processing result information 113 is displayed.

[0133] In other words, when a non-display instruction signal 116 is input from the judgment unit 82B, the control unit 82C does not display the processing result information 113 on the screen 35, and when a display instruction signal 118 is input from the judgment unit 82B, the control unit 82C displays the processing result information 113 on the screen 35.

[0134] Note that the first display mode described here is an example of a "first display mode" according to the present disclosure, and the second display mode described here is an example of a "second display mode" according to the present disclosure.

[0135] Fig. 8 is a conceptual diagram showing an example of a screen when processing result information 113 is displayed on screen 35 and an example of a screen when processing result information 113 is not displayed on screen 35. As shown in Fig. 8, when a display instruction signal 118 is input from determination unit 82B, control unit 82C generates site name information 120, lesion identification information 122, differentiation information 124, and size information 126 as processing result information 113, and displays them in third display area 35C (in the example shown in Fig. 8, the area below first display area 35A).

[0136] The region name information 120 is information indicating the name of the region recognized by the recognition process 95, and is generated based on the region information 112 (see FIG. 6). Here, the region information 112 is exemplified, but any information other than the region information 112 may be used as long as it is information capable of identifying the region where the lesion 42 is present.

[0137] Lesion identification information 122 is information capable of identifying lesion 42 recognized by performing recognition process 95. Differentiation information 124 is information indicating the result of differentiating lesion 42 (e.g., the degree of malignancy), and is generated based on lesion characteristic information 110 (see FIG. 6). Size information 126 is information indicating the size of lesion 42 recognized by performing recognition process 95, and is generated based on geometric property information 106 (see FIG. 6). Although lesion characteristic information 110 and size information 126 are exemplified here, any information indicating the characteristics of lesion 42 other than lesion characteristic information 110 and size information 126 may be used.

[0138] The lesion position map 108 (see FIG. 6) includes a bounding box BB, and when a display instruction signal 118 is input from the determination unit 82B, the control unit 82C displays the bounding box BB in the first display area 35A as the processing result information 113. The bounding box BB is a rectangular frame (e.g., a rectangular frame circumscribing an image area showing the lesion 42) capable of identifying a position recognized by the recognition model 92 as the position in the frame 40 of the lesion 42 captured in the frame 40. In the example shown in FIG. 8, the bounding box BB is superimposed and displayed at the position where the lesion 42 is captured in the frame 40 displayed in the first display area 35A. The display of the bounding box BB is updated in synchronization with the display timing of the endoscopic moving image 39 displayed in the first display area 35A.

[0139] Note that although a bounding box BB is given here, this is merely one example, and the control unit 82C may display an identifier in place of the bounding box BB (for example, a mark defined by the four corners of a rectangular frame) on the screen 35 (in the example shown in FIG. 8, the first display area 35A within the screen 35).

[0140] In the example shown in FIG. 8, the bounding box BB and lesion identification information 122 are an example of “characteristic region information” according to the present disclosure, the part name information 120 is an example of “part-specific information” according to the present disclosure, and the differentiation information 124 and size information 126 are examples of “characteristic information” according to the present disclosure.

[0141] 8, the control unit 82C generates past information 128 as one piece of auxiliary information 44 and displays it in the second display area 35B. The past information 128 is information generated based on the recognition result 102 obtained in the past. For example, the past information 128 includes various pieces of information previously displayed in the third display area 35C, and images or text that can identify areas that have not been observed or treated in the current endoscopic examination.

[0142] On the other hand, when non-display instruction signal 116 is input from determination unit 82B, control unit 82C does not display processing result information 113 on screen 35, but generates past information 128 and similarity related information 130 as a piece of auxiliary information 44 and displays them in second display area 35B. Similarity related information 130 is information related to similarity 114. Here, similarity related information 130 is an example of "similarity related information" according to the present disclosure.

[0143] The similarity-related information 130 is information used in the determination process (see FIG. 7) by the determination unit 82B, a determination result by the determination unit 82B, and / or information generated based on the determination result by the determination unit 82B. In the example shown in FIG. 8, the similarity-related information 130 includes the exceptional image 96 used in calculating the similarity 114 exceeding the first threshold value TH1, the similarity 114 calculated by the determination unit 82B (i.e., the similarity 114 exceeding the first threshold value TH1), similarity level information 132 indicating whether the similarity 114 exceeds the first threshold value TH1, and a notification mark 134 indicating that the processing result information 113 is not displayed.

[0144] Here, the exception image 96 included in the similarity related information 130 is an example of a “second medical image used to obtain similarity” according to the present disclosure, and the similarity level information 132 and the warning mark 134 included in the similarity related information 130 are examples of “information based on similarity” according to the present disclosure.

[0145] 8 shows an example in which the exception image 96 is included in the similarity related information 130, but this is merely one example, and the similarity related information 130 may include an image based on the exception image 96 used in calculating the similarity 114 that exceeds the first threshold value TH1. An example of an image based on the exception image 96 is an image obtained by processing the exception image 96, such as an image obtained by making the exception image 96 a thumbnail or an image obtained by making the exception image 96 achromatic.

[0146] In the example shown in FIG. 8, an example form is shown in which information equivalent to similarity-related information 130 is not displayed on screen 35 when similarity 114 is equal to or less than first threshold value TH1, but information equivalent to similarity-related information 130 may be displayed on screen 35 (e.g., second display area 35B) when similarity 114 is equal to or less than first threshold value TH1. Examples of information equivalent to similarity-related information 130 include the exception image 96 used in calculating the similarity 114 equal to or less than the first threshold value TH1, the similarity 114 calculated by judgment unit 82B (i.e., the similarity 114 equal to or less than the first threshold value TH1), and / or information including similarity level information 132.

[0147] Next, the operation of the portion of the endoscope device 10 according to the present disclosure will be described with reference to Fig. 9. The flow of medical support processing shown in Fig. 9 is an example of the "medical support method" according to the present disclosure.

[0148] 9, first, in step ST10, the recognition unit 82A determines whether or not one frame of image has been captured by the camera 52 inside the large intestine 28. If one frame of image has not been captured by the camera 52 inside the large intestine 28 in step ST10, the determination is negative, and the medical support process proceeds to step ST24. If one frame of image has been captured by the camera 52 inside the large intestine 28 in step ST10, the determination is positive, and the medical support process proceeds to step ST12.

[0149] In step ST12, the recognition unit 82A, the determination unit 82B, and the control unit 82C acquire a frame 40 obtained by imaging the inside of the large intestine 28 (e.g., the intestinal wall 32) by the camera 52. Then, the control unit 82C displays the frame 40 in the first display area 35A. Note that, if the frame 40 is already displayed in the first display area 35A, the control unit 82C updates the frame 40 displayed in the first display area 35A. That is, the processing of step ST12 is repeatedly executed, and the frame 40 is displayed in the first display area 35A in a live view manner. After the processing of step ST12 is executed, the medical support processing proceeds to step ST14.

[0150] In step ST14, the recognition unit 82A executes recognition processing 95 on the frame 40 acquired in step ST12. After the processing in step ST14 is executed, the medical support processing proceeds to step ST16.

[0151] In step ST16, the determination unit 82B acquires all the exception images 96 from the exception image set 94 stored in the storage 86. After the process of step ST16 is executed, the medical support process proceeds to step ST18.

[0152] In step ST18, the determination unit 82B compares the frame 40 acquired in step ST12 with each of all the exception images 96 acquired in step ST16, thereby calculating the similarity 114 for each of all the exception images 96. After the processing of step ST18 is executed, the medical support processing proceeds to step ST20.

[0153] In step ST20, the judgment unit 82B judges whether or not all the calculated similarities 114 are equal to or less than the first threshold value TH1. In step ST20, if all the similarities 114 are equal to or less than the first threshold value TH1, the judgment is affirmative, and the medical support process proceeds to step ST22. In step ST20, if all the similarities 114 are not equal to or less than the first threshold value TH1, the judgment is negative, and the medical support process proceeds to step ST24.

[0154] In step ST22, the control unit 82C displays the processing result information 113 on the screen 35. After the processing in step ST22 is executed, the medical support processing proceeds to step ST24.

[0155] In step ST24, the control unit 82C determines whether or not a condition for terminating the medical support process is satisfied. One example of the condition for terminating the medical support process is that an instruction to terminate the medical support process is given to the endoscope device 10 (for example, that an instruction to terminate the medical support process is accepted by the acceptance device 64).

[0156] In step ST24, if the condition for terminating the medical support process is not satisfied, the determination is negative, and the medical support process proceeds to step ST10. In step ST24, if the condition for terminating the medical support process is satisfied, the determination is positive, and the medical support process ends.

[0157] As described above, in the endoscope device 10 according to the present embodiment, the processing result information 113, which is information about the recognition result 102 by the recognition process 95 performed on the frame 40 generated by capturing an image by the camera 52 inserted into the large intestine 28, is displayed on the screen 35 of the display device 18. In the endoscope device 10, a similarity 114 between the frame 40 and the exception image 96 stored in the storage 86 is calculated. The exception image 96 is an image determined as a medical image inappropriate for use in the recognition process 95, that is, an image in which the lesion 42 is not recognized by the recognition process 95 or an object other than the lesion 42 and / or noise is erroneously recognized as the lesion 42. In the endoscope device 10, the display mode of the processing result information 113 differs depending on the similarity 114. The lower the similarity 114, the higher the reliability of the recognition result 102. This means that the lower the similarity 114, the higher the reliability of the processing result information 113 displayed on the screen 35. Therefore, the doctor 12 can judge whether the processing result information 113 displayed on the screen 35 is reliable or not, based on the display mode of the processing result information 113. As a result, it is possible to prevent the doctor 12 from recognizing an inappropriate recognition result 102 obtained by the recognition process 95 for the frame 40.

[0158] In the endoscope device 10 according to the present embodiment, when the similarity 114 exceeds the first threshold TH1, the processing result information 113 is displayed on the screen 35 in a first display mode that allows the user to specify that the similarity 114 exceeds the first threshold TH1. When the similarity 114 is equal to or less than the first threshold TH1, the processing result information 113 is displayed on the screen 35 in a second display mode that allows the user to specify that the similarity 114 is equal to or less than the first threshold TH1. This allows the doctor 12 to visually understand whether the similarity 114 exceeds the first threshold TH1. In addition, the second display mode has higher visibility than the first display mode. For example, in the second display mode, the processing result information 113 is displayed on the screen 35, and in the first display mode, the processing result information 113 is not displayed on the screen 35. Therefore, it is possible to make the doctor 12 visually recognize that the similarity 114 is equal to or less than the first threshold value TH1 more easily than when the processing result information 113 is displayed on the screen 35 in the same display mode regardless of whether the similarity 114 is equal to or less than the first threshold value TH1. As a result, the doctor 12 can visually recognize that the processing result information 113 is highly reliable.

[0159] Furthermore, in the endoscope device 10 according to the present embodiment, a medical image that is inappropriate for use in the recognition process 95 is used as the exception image 96. When the similarity 114 between the frame 40 and the exception image 96 exceeds a first threshold value TH1, the processing result information 113 is not displayed on the screen 35, and when the similarity 114 between the frame 40 and the exception image 96 is equal to or less than the first threshold value TH1, the processing result information 113 is displayed on the screen 35. This makes it possible to prevent the doctor 12 from making a medical decision based on the recognition result 102 obtained by performing the recognition process 95 on the frame 40 that is inappropriate for use in the recognition process 95.

[0160] Moreover, in the endoscope device 10 according to the present embodiment, an inappropriate imaging target image 96A is used as one of the exception images 96. Therefore, it is possible to prevent the doctor 12 from making a medical decision based on a recognition result 102 obtained by performing the recognition process 95 on a frame 40 in which an imaging target 98 (e.g., air bubbles and / or residue) inappropriate for use in the recognition process 95 is captured.

[0161] Furthermore, in the endoscope device 10 according to the present embodiment, the inappropriate size image 96B and the inappropriate position image 96C are used as one of the exception images 96. Therefore, it is possible to prevent the doctor 12 from making a medical judgment based on the recognition result 102 obtained by performing the recognition process 95 on a frame 40 having a composition inappropriate for use in the recognition process 95 (for example, a composition in which the size of the lesion 42 included in the imaging range is too large within the imaging range, a composition in which the size of the lesion 42 included in the imaging range is too small within the imaging range, and / or a composition in which the lesion 42 included in the imaging range is outside the central region of the imaging range (i.e., the region corresponding to the central region 99 shown in FIG. 5)).

[0162] Moreover, in the endoscope device 10 according to the present embodiment, the inappropriate image quality image 96D is used as one of the exception images 96. Therefore, it is possible to prevent the doctor 12 from making a medical decision based on the recognition result 102 obtained by performing the recognition process 95 on a frame 40 with image quality inappropriate for use in the recognition process 95 (for example, a frame 40 having out-of-focus and / or motion blur, etc.).

[0163] Furthermore, in the endoscope device 10 according to the present embodiment, when the similarity 114 is equal to or less than the first threshold TH1, part name information 120 is displayed in the third display region 35C as information capable of identifying the part in which the lesion 42 exists. When the similarity 114 is equal to or less than the first threshold TH1, a bounding box BB and lesion identification information 122 are displayed in the third display region 35C as information capable of identifying the lesion 42. Furthermore, when the similarity 114 is equal to or less than the first threshold TH1, discrimination information 124 and size information 126 are displayed in the third display region 35C as information capable of identifying the characteristics of the lesion 42.

[0164] Therefore, the degree to which the doctor 12 perceives the region name information 120, the bounding box BB, the lesion identification information 122, the differentiation information 124, and the size information 126 can be varied according to the similarity 114. That is, when an inappropriate recognition result 102 is obtained by the recognition process 95 for the frame 40 (for example, when the similarity 114 exceeds the first threshold TH1), the doctor 12 can be made less likely to perceive the region name information 120, the bounding box BB, the lesion identification information 122, the differentiation information 124, and the size information 126, whereas when an appropriate recognition result 102 is obtained by the recognition process 95 for the frame 40 (for example, when the similarity 114 is equal to or less than the first threshold TH1), the doctor 12 can be made more likely to perceive the region name information 120, the bounding box BB, the lesion identification information 122, the differentiation information 124, and the size information 126.

[0165] Furthermore, in this embodiment, when the similarity 114 exceeds the first threshold value TH1, the similarity related information 130 (e.g., information including the exception image 96, the similarity 114, the similarity level information 132, and the notification mark 134) is displayed in the second display area 35B. This allows the doctor 12 to visually grasp the similarity related information 130 (e.g., information including the exception image 96, the similarity 114, the similarity level information 132, and the notification mark 134). As a result, the doctor 12 can understand the reason why the processing result information 113 was not displayed on the screen 35.

[0166] In the above embodiment, an example is given in which the processing result information 113 is not displayed on the screen 35 when the similarity 114 exceeds the first threshold TH1, but this is merely one example. For example, as shown in FIG. 10, even when the similarity 114 exceeds the first threshold TH1 (i.e., when a non-display instruction signal 116 is input to the control unit 82C), the processing result information 113 may be displayed on the screen 35. In this case, the visibility of the processing result information 113 is lowered. That is, when the similarity 114 exceeds the first threshold TH1, the visibility of the processing result information 113 displayed on the screen 35 is lowered than when the similarity 114 is equal to or lower than the first threshold TH1 (i.e., when a display instruction signal 118 is input to the control unit 82C). 10, the part name information 120, the lesion identification information 122, the differentiation information 124, and the size information 126 are displayed in a grayed-out state in the third display area 35C, and the bounding BB line is displayed thinner than the bounding BB line displayed when the similarity 114 is equal to or less than the first threshold value TH1. There are various methods for reducing visibility. For example, there are a method for reducing visibility by increasing the transparency of the display, a method for reducing visibility by covering the display part with some kind of mask, and a method for reducing visibility by blinking the display.

[0167] In the above embodiment, an example in which the similarity 114 is calculated by the determination unit 82B has been given, but this is merely one example, and for example, as shown in Fig. 11, the determination unit 82B may calculate the dissimilarity 136, and the control unit 82C may change the display mode of the processing result information 113 according to the dissimilarity 136. The dissimilarity 136 refers to the degree of difference between the frame 40 and the exceptional image 96. The determination unit 82B determines whether or not the dissimilarity 136 exceeds the second threshold value TH2.

[0168] Here, when all dissimilarities 136 calculated by comparing the frame 40 acquired from the camera 52 with each of the exception images 96 acquired from the exception image set 94 are equal to or less than a second threshold value TH2 (e.g., 80%), the determination unit 82B outputs a non-display instruction signal 116 to the control unit 82C. The second threshold value TH2 may be a fixed value or a variable value (e.g., a value that changes according to an instruction accepted by the acceptance device 64 and / or various conditions).

[0169] On the other hand, when all of the dissimilarities 136 calculated by comparing the frame 40 obtained from the camera 52 with each of the exception images 96 obtained from the exception image set 94 exceed the second threshold value TH2, the judgment unit 82B outputs a display instruction signal 118 to the control unit 82C.

[0170] The control unit 82C displays the processing result information 113 in the third display mode or the fourth display mode according to the degree of difference 136. The third display mode is a display mode in which it is possible to specify that the degree of difference 136 is equal to or less than the second threshold value TH2. In the third display mode, the processing result information 113 is not displayed. The fourth display mode is a display mode which has higher visibility than the third display mode and in which it is possible to specify that the degree of difference 136 has exceeded the second threshold value TH2. In the fourth display mode, the processing result information 113 is displayed. In this way, it is possible to obtain the same effect as in the above embodiment.

[0171] 11, the dissimilarity 136 is an example of the "dissimilarity" and "comparison result" according to the present disclosure, and the second threshold TH2 is an example of the "second threshold" according to the present disclosure. The third display mode described here is an example of the "first display mode" and "third display mode" according to the present disclosure, and the fourth display mode described here is an example of the "second display mode" and "fourth display mode" according to the present disclosure.

[0172] In the above embodiment, an example has been given in which similarity-related information 130 is displayed in second display region 35B when similarity 114 exceeds first threshold TH1, but as an example, as shown in Fig. 12, when dissimilarity 136 is equal to or smaller than second threshold TH2 (i.e., when non-display instruction signal 116 is input to control unit 82C), dissimilarity-related information 138, which is information related to dissimilarity 136, may be displayed in second display region 35B as one piece of auxiliary information 44. Here, dissimilarity-related information 138 is an example of "dissimilarity-related information" according to the present disclosure.

[0173] 12, the dissimilarity-related information 138 includes the exception image 96 used in calculating the dissimilarity 136 equal to or less than the second threshold TH2, the dissimilarity 136 calculated by the determination unit 82B (i.e., the dissimilarity 136 equal to or less than the second threshold TH2), dissimilarity level information 140 indicating whether the dissimilarity 136 exceeds the second threshold TH2, and the notification mark 134. By displaying the dissimilarity-related information 138 on the screen 35 in this manner, the doctor 12 can visually grasp the dissimilarity-related information 138. As a result, the doctor 12 can understand the reason why the processing result information 113 was not displayed on the screen 35.

[0174] The exception image 96 included in the dissimilarity-related information 138 is an example of a "second medical image used to obtain the dissimilarity" according to the present disclosure, and the warning mark 134 and the dissimilarity level information 140 included in the dissimilarity-related information 138 are an example of "information based on the dissimilarity" according to the present disclosure.

[0175] 12, when the dissimilarity 136 is equal to or less than the second threshold TH2, the dissimilarity-related information 138 (e.g., information including the exception image 96, the dissimilarity 136, the dissimilarity level information 140, and the notification mark 134) is displayed in the second display area 35B. This allows the doctor 12 to visually grasp the dissimilarity-related information 138 (e.g., information including the exception image 96, the dissimilarity 136, the dissimilarity level information 140, and the notification mark 134). As a result, the doctor 12 can understand the reason why the processing result information 113 was not displayed on the screen 35.

[0176] In the example shown in Figure 12, an example is given in which the dissimilarity-related information 138 includes an exception image 96, but this is merely one example, and the dissimilarity-related information 138 may also include an image based on the exception image 96 used in calculating the dissimilarity 136 that is less than or equal to the second threshold value TH2.

[0177] In the example shown in FIG. 12, an example form is shown in which information equivalent to dissimilarity-related information 138 is not displayed on screen 35 when dissimilarity 136 exceeds second threshold TH2, but information equivalent to dissimilarity-related information 138 may be displayed on screen 35 (e.g., second display area 35B) when dissimilarity 136 exceeds second threshold TH2. Examples of information equivalent to dissimilarity-related information 138 include the exception image 96 used in calculating the dissimilarity 136 that exceeds the second threshold TH2, the dissimilarity 136 calculated by judgment unit 82B (i.e., the dissimilarity 136 that exceeds the second threshold TH2), and / or information including dissimilarity level information 140.

[0178] In the above embodiment, the determination unit 82B calculates the similarity 114 between each of all the exception images 96 acquired from the exception image set 94 and the frame 40, and compares all the similarities 114 with the first threshold value TH1. However, this is merely an example. For example, as shown in FIG. 13, the determination unit 82B may calculate the similarity 114 between each of all the exception images 96 acquired from the exception image set 94 and the frame 40, and compare a representative similarity 146, which is a similarity representative of the calculated multiple similarities 114 (here, as an example, all the similarities 114), with the first threshold value TH1. The representative similarity 146 is an example of a "representative value" according to the present disclosure. An example of the representative similarity 146 is a statistical value of the multiple similarities 114. An example of the statistical value is an average value, a maximum value, a minimum value, a median value, or a mode. In this way, by comparing the representative similarity 146 with the first threshold value TH1, the processing load can be reduced compared to the case where a plurality of similarities 114 are compared with the first threshold value TH1.

[0179] Note that while the representative similarity 146 has been exemplified here, this is merely one example, and the judgment unit 82B may calculate the dissimilarity 136 between each of all of the exception images 96 obtained from the exception image set 94 and the frame 40, and the representative dissimilarity, which is a dissimilarity representative of the multiple calculated dissimilarity levels 136 (here, as an example, all of the dissimilarity levels 136), may be compared with the second threshold value TH2, and in this case as well, a similar effect can be expected.

[0180] As an example, as shown in FIG. 14, the judgment unit 82B may calculate the similarity 114 or the dissimilarity 136 by comparing the exceptional image 96 located at the center of gravity of a cluster 148 obtained by clustering a plurality of exceptional images 96 (here, as an example, all the exceptional images 96) included in the exceptional image set 94 with the frame 40.

[0181] Each of the plurality of exception images 96 is assigned metadata 150. The metadata 150 includes multidimensional feature amounts that define the exception image 96, and the determination unit 82B clusters the plurality of exception images 96 by referring to the multidimensional feature amounts included in the metadata 150 of each exception image 96.

[0182] The multidimensional feature included in the metadata 150 is a feature related to a plurality of elements visually recognizable by humans, such as the number of pixels, color, and brightness of the exception image 96, and / or a feature related to a plurality of elements visually unrecognizable by humans. The multidimensional feature is generated and output by a trained model (not shown) by inputting the exception image 96 to the trained model, for example. An example of the trained model used here is a trained model that reduces the dimensions of the input exception image 96, generates and outputs a plurality of feature amounts. The multidimensional feature included in the metadata 150 may be a multidimensional feature extracted from a plurality of intermediate layers included in the recognition model 92 by inputting the exception image 96 to the recognition model 92.

[0183] In this way, by comparing the exceptional image 96 located at the center of the cluster 148 with the frame 40, the processing load can be reduced compared to the case where a plurality of similarities 114 are compared with the first threshold value TH1. Also, by comparing the exceptional image 96 located at the center of the cluster 148 with the frame 40, the processing load can be reduced compared to the case where a plurality of dissimilarity scores 136 are compared with the second threshold value TH2.

[0184] In the above embodiment, an example is given on the premise that the recognition process 95 is executed by the recognition unit 82A regardless of the determination result by the determination unit 82B, but the present disclosure is not limited to this. For example, as shown in Fig. 15, the determination unit 82B may output a recognition process control signal 152 for controlling the recognition process 95 according to the determination result to the recognition unit 82A, and the recognition unit 82A may control the recognition process 95 according to the recognition process control signal 152.

[0185] For example, when the similarity 114 exceeds the first threshold TH1, the determination unit 82B controls the recognition process 95 by outputting a recognition process control signal 152 to the recognition unit 82A so that the recognition process 95 is not performed on the frame 40, and when the similarity 114 is equal to or less than the first threshold TH1, the recognition process 95 is performed on the frame 40. In this way, the same effect as in the above embodiment can be obtained.

[0186] Furthermore, when the dissimilarity 136 is equal to or less than the second threshold TH2, the determination unit 82B controls the recognition process 95 by outputting a recognition process control signal 152 to the recognition unit 82A so that the recognition process 95 is not performed on the frame 40, and when the dissimilarity 136 exceeds the second threshold TH2, the recognition process 95 is performed on the frame 40. In this way, the same effect as in the above embodiment can be obtained.

[0187] In the above embodiment, lesion 42 is exemplified as an example of a "characteristic region" according to the present disclosure, but the present disclosure is not limited thereto, and the technology of the present disclosure can be applied even if a resection region, a bleeding region, a marking region, an organ, or a treatment tool (e.g., a hemostatic clip placed inside the body) is applied instead of lesion 42.

[0188] In the above embodiment, the camera 52 mounted on the endoscope 16 is exemplified, but the present disclosure is not limited to this. For example, the technology of the present disclosure is also applicable to a fundus camera, a camera mounted on a medical microscope, or the like.

[0189] In the above embodiment, an example was given in which the frame 40 is input to the recognition unit 82A and the processing result of the medical support processing is output from the control unit 82C, but the present disclosure is not limited to this. For example, instruction data (so-called prompt) including the frame 40 may be input to a so-called generation AI, and the processing result of the medical support processing may be output from the generation AI. One example of the generation AI is GPT-4 (Internet search<https: / / openai.com / gpt-4> ) is used in ChatGPT.

[0190] Furthermore, instruction data including at least a part of the processing result of the medical support processing (e.g., at least a part of the processing result information 113, etc.) may be used as input information for the generation AI. Examples of information output from the generation AI include information regarding the operation content of the camera 52, information indicating the content of medical procedures recommended to be performed during the endoscopic examination, and / or information indicating the content of medical procedures recommended to be performed after the endoscopic examination. The information output from the generation AI may be stored in various storage areas (e.g., storage 76 and / or 86), displayed on the display device 18 as auxiliary information 44, printed on a medium by a printer, or output as audio from a speaker.

[0191] In the above embodiment, recognition processing 95 using AI in a bounding box method is exemplified, but this is merely one example, and for example, recognition processing using AI in a segmentation method may be performed instead of recognition processing using AI in a bounding box method 95. Also, instead of recognition processing using an AI method, recognition processing using a non-AI method (for example, a template matching method) may be performed, or recognition processing using a combination of a non-AI method and an AI method may be performed.

[0192] In the above embodiment, an example in which the medical support process is performed by the computer 78 has been described, but the present disclosure is not limited to this, and at least a part of the processes included in the medical support process may be performed by a device provided outside the computer 78. An example of this case will be described below with reference to FIG.

[0193] 16 is a conceptual diagram showing an example of the configuration of an endoscope device 154. The endoscope device 154 is an example of the "endoscope device" according to the present disclosure. The endoscope device 154 differs from the endoscope device 10 described in the above embodiment in that it includes an external device 156.

[0194] External device 156 is communicatively coupled to computer 78 via a network 158 (eg, a WAN and / or LAN, etc.).

[0195] An example of the external device 156 is at least one server that directly or indirectly transmits and receives data to and from the computer 78 via the network 158. The external device 156 receives a processing execution instruction provided from the processor 82 of the computer 78 via the network 158. The external device 156 then executes processing according to the received processing execution instruction and transmits the processing result to the computer 78 via the network 158. In the computer 78, the processor 82 receives the processing result transmitted from the external device 156 via the network 158 and executes processing using the received processing result.

[0196] An example of the processing execution instruction is an instruction to have the external device 156 execute at least a part of the medical support processing. A first example of at least a part of the medical support processing (i.e., a processing to be executed by the external device 156) is the recognition processing 95. In this case, the external device 156 executes the recognition processing 95 in accordance with the processing execution instruction provided from the processor 82 via the network 158, and transmits the recognition result 102 to the computer 78 via the network 158. In the computer 78, the processor 82 receives the recognition result 102, and executes the same processing as in the above embodiment using the received recognition result 102.

[0197] A second example of at least a part of the medical support process (i.e., the process to be executed by the external device 156) is a process by the determination unit 82B. In this case, the external device 156 executes the process by the determination unit 82B according to a process execution instruction given from the processor 82 via the network 158, and transmits the process results (e.g., the non-display instruction signal 116, the display instruction signal 118, the similarity 114, the dissimilarity 136, the exception image 96 used in calculating the similarity 114, and / or the exception image 96 used in calculating the dissimilarity 136, etc.) to the computer 78 via the network 158. In the computer 78, the processor 82 receives the process results and executes the same process as in the above embodiment (e.g., display using the display device 18, etc.) using the received process results.

[0198] A third example of at least a part of the medical support process (i.e., a process to be executed by the external device 156) is a process by the control unit 82C. In this case, the external device 156 executes the process by the control unit 82C according to a process execution instruction given from the processor 82 via the network 158, and transmits the process result (e.g., at least a part of the process result information 113, etc.) to the computer 78 via the network 158. In the computer 78, the processor 82 receives the process result, and executes the same process as in the above embodiment (e.g., display using the display device 18, etc.) using the received process result.

[0199] For example, the external device 156 is realized by cloud computing. Note that cloud computing is merely an example, and the external device 156 may be realized by network computing such as fog computing, edge computing, or grid computing. Instead of a server, at least one personal computer or the like may be used as the external device 156. Also, the external device 156 may be a computing device with a communication function equipped with multiple types of AI functions.

[0200] In the above embodiment, an example in which the medical support program 90 is stored in the storage 86 has been described, but the present disclosure is not limited to this. For example, the medical support program 90 may be stored in a portable computer-readable non-transitory storage medium such as an SSD or a USB memory. The medical support program 90 stored in the non-transitory storage medium is installed in the computer 78 of the endoscope device 10. The processor 82 executes medical support processing in accordance with the medical support program 90.

[0201] In addition, the medical support program 90 may be stored in a storage device such as another computer or server connected to the endoscopic device 10 via a network, and the medical support program 90 may be downloaded and installed on the computer 78 upon request from the endoscopic device 10.

[0202] It is not necessary to store all of the medical support program 90 in a storage device such as another computer or server device connected to the endoscopic device 10, or to store all of the medical support program 90 in the storage 86; only a portion of the medical support program 90 may be stored.

[0203] The hardware resources for executing the medical support processing can be various processors as shown below. An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the medical support processing by executing software, i.e., a program. Another example of a processor is a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing a specific process, such as an FPGA, a PLD, or an ASIC. Each processor has a built-in or connected memory, and each processor executes the medical support processing by using the memory.

[0204] The hardware resource for executing the medical support process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource for executing the medical support process may be a single processor.

[0205] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes medical support processing. Second, there is a configuration using a processor that realizes the functions of the entire system including multiple hardware resources that execute medical support processing on a single IC chip, as typified by SoC. In this way, the medical support processing is realized using one or more of the above various processors as hardware resources.

[0206] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. Also, the above medical support process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processing may be changed, without departing from the spirit of the invention.

[0207] The above description and illustrations are detailed descriptions of the parts related to the present disclosure and are merely an example of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations within the scope of the gist of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the present disclosure.

[0208] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or standard was specifically and individually indicated to be incorporated by reference.

[0209] The following supplementary notes are further disclosed regarding the above embodiment.

[0210] (Appendix 1) Obtaining a first medical image generated by capturing an image with a camera; and When a similarity between the first medical image and a second medical image stored in a storage exceeds a first threshold TH1, not performing an object recognition process on the first medical image, and when the similarity is equal to or less than the first threshold TH1, performing the object recognition process. Medical assistance methods.

[0211] (Appendix 2) Obtaining a first medical image generated by capturing an image with a camera; and not executing an object recognition process on the first medical image when a degree of difference between the first medical image and a second medical image stored in a storage is equal to or smaller than a second threshold value TH2, and executing the object recognition process when the degree of difference exceeds the second threshold value TH2. Medical assistance methods.

[0212] (Appendix 3) A program for causing a computer to execute a medical support process, The medical support process is as follows: Obtaining a first medical image generated by capturing an image with a camera; and When a similarity between the first medical image and a second medical image stored in a storage exceeds a first threshold TH1, not performing an object recognition process on the first medical image, and when the similarity is equal to or less than the first threshold TH1, performing the object recognition process. program.

[0213] (Appendix 4) A program for causing a computer to execute a medical support process, The medical support process is as follows: Obtaining a first medical image generated by capturing an image with a camera; and not executing an object recognition process on the first medical image when a degree of difference between the first medical image and a second medical image stored in a storage is equal to or smaller than a second threshold value TH2, and executing the object recognition process when the degree of difference exceeds the second threshold value TH2. program.

[0214] (Appendix 5) A non-transitory computer-readable storage medium for storing a program for causing a computer to execute a medical support process, The medical support process is as follows: Obtaining a first medical image generated by capturing an image with a camera; and When a similarity between the first medical image and a second medical image stored in a storage exceeds a first threshold TH1, not performing an object recognition process on the first medical image, and when the similarity is equal to or less than the first threshold TH1, performing the object recognition process. Non-transitory storage media.

[0215] (Appendix 6) A non-transitory computer-readable storage medium for storing a program for causing a computer to execute a medical support process, The medical support process is as follows: Obtaining a first medical image generated by capturing an image with a camera; and not executing an object recognition process on the first medical image when a degree of difference between the first medical image and a second medical image stored in a storage is equal to or smaller than a second threshold value TH2, and executing the object recognition process when the degree of difference exceeds the second threshold value TH2. Non-transitory storage media.

[0216] (Appendix 7) A processor is provided. The processor is displaying, on a display device, processing result information which is information regarding a processing result of an object recognition process performed on the first medical image generated by capturing the first medical image with the camera; The display mode of the processing result information is changed according to the similarity or difference between the first medical image and the second medical image stored in the storage. Medical support equipment.

[0217] (Appendix 8) When the degree of difference is equal to or less than a second threshold, the processing result information is displayed in a third display manner that allows identification that the degree of difference is equal to or less than the second threshold; When the degree of difference exceeds the second threshold, the processing result information is displayed in a fourth display manner that makes it possible to specify that the degree of difference has exceeded the second threshold. 8. The medical support device according to claim 7.

[0218] (Appendix 9) The fourth display mode has higher visibility than the third display mode. 9. The medical support device according to claim 8.

[0219] (Appendix 10) In the third display mode, the processing result information is not displayed, In the fourth display mode, the processing result information is displayed. 10. The medical support device according to claim 9.

[0220] (Appendix 11) A processor is provided. The processor is When a degree of difference between a first medical image generated by capturing an image by a camera and a second medical image stored in a storage is equal to or less than a second threshold, an object recognition process is not performed on the first medical image, and when the degree of difference exceeds the second threshold, the object recognition process is performed. Medical support equipment. [Explanation of symbols]

[0221] 10,154 Endoscopic equipment 12. Doctor 14. Nurse 16 Endoscope 18 Display device 20 Light source device 22 Control device 24 Medical Support Equipment 26 Subject 28 Large intestine 30 light 32 Intestinal wall 34 Wagon 35 screens 35A 1st display area 35B 2nd display area 35C 3rd display area 39 Endoscopic video images 40 frames 42,100,101 Lesions 44 Supporting Information 46 Control section 48 Insertion section 50 Tip 50A Tip surface 52 Camera 54 Lighting Equipment 54A, 54B Lighting window 56 Treatment opening 58 Treatment tools 60 Treatment tool insertion port 62 Universal Code 64 Reception device 66,78 Computer 68,88 Bus 70,80 External I / F 72,82 processor 74,84 Memory 76,86 Storage 82A recognition part 82B Judgment section 82C Control section 90 Medical Assistance Program 92 Recognition Model 94 Exceptional Image Set 94A Inappropriate imaging target image set 94B Inappropriate composition image set 94B1 Inappropriate size image set 94B2 Inappropriate position image set 94C Inappropriate quality image set 95 Recognition Processing 96 Exceptional Images 96A Inappropriate image capture target image 96B Inappropriate size image 96C Inappropriately positioned image 96D Inappropriate image quality 98 Imaging Target 99 Central area 102 Recognition results 104 Lesion Presence Information 106 Geometric property information 108 Lesion Location Map 110 Lesion characteristic information 112 Part information 113 Processing result information 114 Similarity 116 Hidden instruction signal 118 Display instruction signal 120 Part name information 122 Lesion Identification Information 124 Identification information 126 Size Information 128 Past information 130 Similarity Related Information 132 Similarity level information 134 Notification mark 136 Difference 138 Difference-related information 140 Dissimilarity level information 146 Representative similarity 148 Clusters 150 Metadata 152 Recognition processing control signal 156 External device 158 Network BB Bounding Box TH1 First Threshold TH2 Second Threshold

Claims

1. A processor is provided. The processor, displaying, on a display device, processing result information which is information regarding a processing result of an object recognition process performed on the first medical image generated by capturing the first medical image with the camera; A display mode of the processing result information is changed according to a similarity or a difference between the first medical image and a second medical image stored in a storage device. Medical support equipment.

2. when the similarity exceeds a first threshold or when the dissimilarity is equal to or less than a second threshold, the processing result information is displayed in a first display manner that allows identification that the similarity has exceeded the first threshold or that the dissimilarity is equal to or less than the second threshold, When the similarity is equal to or less than the first threshold or when the dissimilarity exceeds the second threshold, the processing result information is displayed in a second display manner that allows the user to specify that the similarity is equal to or less than the first threshold or that the dissimilarity exceeds the second threshold. The medical support device according to claim 1 .

3. The second display mode has higher visibility than the first display mode. The medical support device according to claim 2.

4. In the first display mode, the processing result information is not displayed, In the second display mode, the processing result information is displayed. The medical support device according to claim 3.

5. The second medical image is an exceptional image that is determined as a medical image that is inappropriate for use in the object recognition process. The medical support device according to claim 1 .

6. The exception image is an image that contains an object that is inappropriate for use in the object recognition process. The medical support device according to claim 5.

7. The imaging target includes air bubbles and / or residue. The medical support device according to claim 6.

8. The exceptional image is an image with a composition that is inappropriate for use in the object recognition process. The medical support device according to claim 5.

9. the composition is a first composition and / or a second composition, the first composition is a composition in which a size of a target included in an imaging range is larger than a first size, or a composition in which the size of a target included in an imaging range is smaller than a second size, The second composition is a composition in which the target is out of the central region of the imaging range. The medical support device according to claim 8.

10. The exceptional image is an image of inappropriate quality for use in the object recognition process. The medical support device according to claim 5.

11. The image quality includes out-of-focus and / or motion blur. The medical support device according to claim 10.

12. a plurality of second medical images having different contents are stored in the storage; The similarity or the difference is a representative value of a plurality of comparison results obtained by comparing the first medical image with each of the plurality of second medical images, or a comparison result obtained by comparing the second medical image located at the center of gravity of a cluster obtained by clustering the plurality of second medical images with the first medical image. The medical support device according to claim 1 .

13. The processing result information includes feature region information capable of identifying a feature region recognized by the object recognition process, characteristic information indicating characteristics of the feature region, and / or part identification information capable of identifying a part in which the feature region exists. The medical support device according to claim 1 .

14. The display device displays similarity-related information, which is information related to the similarity, or dissimilarity-related information, which is information related to the dissimilarity. The medical support device according to claim 1 .

15. the similarity-related information includes the similarity, information based on the similarity, the second medical image used to obtain the similarity, and / or an image based on the second medical image used to obtain the similarity, The dissimilarity-related information includes the dissimilarity, information based on the dissimilarity, the second medical image used to obtain the dissimilarity, and / or an image based on the second medical image used to obtain the dissimilarity. The medical support device according to claim 14.

16. The camera is mounted on an endoscope and is inserted into the body to capture images of the inside of the body. The medical support device according to claim 1 .

17. The object recognition process is a process for recognizing a lesion. The medical support device according to claim 1 .

18. The processing result is generated by a trained model by inputting the first medical image into the trained model. The medical support device according to claim 1 .

19. A processor is provided. The processor, When a similarity between a first medical image generated by capturing an image by a camera and a second medical image stored in a storage exceeds a first threshold value or when a dissimilarity between the first medical image and the second medical image is equal to or less than a second threshold value, an object recognition process is not performed on the first medical image, and when the similarity is equal to or less than the first threshold value or the dissimilarity exceeds the second threshold value, the object recognition process is performed. Medical support equipment.

20. A medical support device according to any one of claims 1 to 19; The camera. Endoscopic device.

21. Displaying, on a display device, processing result information which is information regarding a processing result of an object recognition process performed on the first medical image generated by capturing the image with the camera; and and changing a display mode of the processing result information according to a degree of similarity or a degree of difference between the first medical image and a second medical image stored in a storage device. Medical assistance methods.

22. Displaying, on a display device, processing result information which is information regarding a processing result of an object recognition process performed on the first medical image generated by capturing the image with the camera; and A program for causing a computer to execute processing including changing the display manner of the processing result information depending on the degree of similarity or difference between the first medical image and a second medical image stored in a storage.

Images