Medical support device, endoscope system, medical support method, and program

Abstract

Provided are a medical support device, an endoscope system, a medical support method, and a program capable of suppressing provision of information reflecting a result of erroneous recognition of a feature of a feature region by image recognition processing due to low accuracy of sharpness of the feature region included in a medical image to a user or the like.SOLUTION: The medical support device includes a processor. The processor acquires a medical image obtained by imaging a site including a feature region, and performs first processing according to sharpness of an inner region that is a region inside an outer edge of the feature region included in the medical image, or performs second processing according to the sharpness, the first processing is processing of controlling image recognition processing that is executable on the medical image and recognizes a feature of the feature region, and the second processing is processing of controlling output of information based on a processing result of the image recognition processing.SELECTED DRAWING: Figure 22

Description

[Technical Field]

[0001] The present disclosure relates to a medical support device, an endoscope system, a medical support method, and a program. [Background technology]

[0002] Patent Document 1 discloses an image assessment device comprising an acquisition means, a detection means, a setting means, a blur assessment means, and an image assessment means. In the image assessment device described in Patent Document 1, the acquisition means acquires an endoscopic image. The detection means detects lesion candidates from the endoscopic image and outputs a lesion candidate image including the lesion candidate. The setting means sets a lesion area corresponding to the lesion candidate and a corresponding area corresponding to the lesion area in the lesion candidate image. The blur assessment means assesses the blur of the image of the lesion area based on the image of the lesion area and the image of the corresponding area. The blur assessment means also assesses the blur of the image of the lesion area based on the power of high-frequency components for each of the image of the lesion area and the image of the corresponding area. The image assessment means assesses the suitability of the lesion candidate image based on the blur assessment result.

[0003] Patent Document 2 discloses an image diagnosis support system including an input unit, an identification unit, and a determination unit. In the image diagnosis system described in Patent Document 2, the input unit accepts an input of an image. The identification unit identifies specular reflection areas and non-specular reflection areas in a region of interest in the image. The determination unit determines whether the region of interest is an unqualified region unsuitable for diagnosis based on image processing results for at least one of the specular reflection areas and the non-specular reflection areas. The determination unit also determines whether the region of interest is an unqualified region accompanied by blur based on image processing results for the non-specular reflection areas.

[0004] The diagnostic imaging system described in Patent Document 2 further includes a blur amount calculation unit that calculates the amount of blur in a non-specular reflection region. The blur amount calculation unit calculates the amount of blur using an image before and after application of a Gaussian filter. The determination unit determines whether the region of interest is an unqualified region accompanied by blur based on the amount of blur calculated by the blur amount calculation unit. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2023 / 144936 [Patent Document 2] International Publication No. 2019 / 142243 Summary of the Invention

[0006] One embodiment of the present disclosure provides a medical support device, an endoscopic system, a medical support method, and a program that can prevent information that reflects the results of an image recognition process erroneously recognizing the features of a feature area due to low accuracy in the sharpness of the feature area in a medical image from being provided to a user, etc. [Means for solving the problem]

[0007] A first aspect of the present disclosure is a medical support device that includes a processor, which acquires a medical image obtained by imaging a site including a characteristic region, and performs a first processing step depending on the sharpness of an inner region that is an area inside the outer edge of the characteristic region included in the medical image, or performs a second processing step depending on the sharpness, wherein the first processing step is an image recognition processing step that can be performed on the medical image and controls the image recognition processing that recognizes the characteristics of the characteristic region, and the second processing step is a processing step that controls the output of information based on the processing result of the image recognition processing.

[0008] A second aspect of the present disclosure is a medical support device according to the first aspect, wherein the first processing includes processing that performs image recognition processing when the sharpness is equal to or greater than a first threshold, and does not perform image recognition processing when the sharpness is less than a second threshold that is equal to or less than the first threshold.

[0009] A third aspect of the present disclosure is a medical support device according to the first or second aspect, wherein the second processing includes processing of outputting information when the sharpness is equal to or greater than a first threshold, and not outputting information when the sharpness is less than a second threshold that is equal to or less than the first threshold.

[0010] A fourth aspect of the present disclosure is a medical support device according to any one of the first to third aspects, in which a region recognition process is performed on a medical image to recognize a characteristic region, and the inner region is a region based on a segmentation mask obtained by performing the region recognition process.

[0011] A fifth aspect of the present disclosure is the medical support device according to the fourth aspect, wherein the resolution of the segmentation mask is lower than the resolution of the medical image.

[0012] A sixth aspect of the present disclosure is a medical support device according to any one of the first to third aspects, in which the inner region is an area within a second frame, which is a frame formed by narrowing a first frame that surrounds the characteristic region and is closer to the inside of the characteristic region than the outer edge.

[0013] A seventh aspect of the present disclosure is a medical support device according to the sixth aspect, in which an area recognition process is performed on a medical image to recognize a characteristic area, a bounding box is used in the area recognition process, and the first frame corresponds to the bounding box.

[0014] An eighth aspect of the present disclosure is a medical support device according to any one of the first to seventh aspects, in which, when a portion of a feature area included in a medical image is outside the medical image, the inner area is an area outside the edge of the medical image.

[0015] A ninth aspect of the present disclosure is a medical support device according to any one of the first to eighth aspects, wherein when a halation area is present in a feature area included in a medical image, the inner area is an area outside the edge of the halation area.

[0016] A tenth aspect of the present disclosure is a medical support device according to any one of the first to ninth aspects, wherein the first processing and / or the second processing is performed according to multiple sharpness values ​​obtained from multiple medical images in time series.

[0017] An eleventh aspect of the present disclosure is a medical support device according to any one of the first to tenth aspects, in which at least one of multiple processing results obtained by performing image recognition processing on multiple medical images is assigned a weight determined based on multiple sharpness indices.

[0018] A twelfth aspect of the present disclosure is the medical support device according to any one of the first to eleventh aspects, wherein outputting the information includes displaying the information on a screen.

[0019] A thirteenth aspect of the present disclosure is the medical support device according to any one of the first to twelfth aspects, in which the feature is a medical feature of the feature region.

[0020] A fourteenth aspect of the present disclosure is the medical support device according to any one of the first to thirteenth aspects, in which the characteristic region is a lesion.

[0021] A fifteenth aspect of the present disclosure is the medical support device according to any one of the first to fourteenth aspects, in which the medical image is an endoscopic image.

[0022] A sixteenth aspect of the present disclosure is an endoscopic system comprising a medical support device according to any one of the first to fifteenth aspects and an endoscopic scope equipped with an image sensor for capturing images of a local area.

[0023] A seventeenth aspect of the present disclosure is a medical support method that includes acquiring a medical image obtained by imaging a site including a characteristic region, and performing a first process depending on the sharpness of an inner region that is an area inside the outer edge of the characteristic region included in the medical image, or performing a second process depending on the sharpness, wherein the first process is an image recognition process that can be performed on the medical image and is a process that controls the image recognition process that recognizes the characteristics of the characteristic region, and the second process is a process that controls the output of information based on the processing result of the image recognition process.

[0024] An 18th aspect of the present disclosure is a program for causing a computer to execute medical support processing, the medical support processing including acquiring a medical image obtained by imaging a part including a characteristic region, and performing a first processing depending on the sharpness of an inner region that is an area inside the outer edge of the characteristic region included in the medical image, or performing a second processing depending on the sharpness, wherein the first processing is an image recognition processing that can be executed on the medical image, and is processing for controlling the image recognition processing that recognizes the characteristics of the characteristic region, and the second processing is processing for controlling the output of information based on the processing result of the image recognition processing. [Brief explanation of the drawings]

[0025] [Figure 1] FIG. 1 is a conceptual diagram showing an example of how the endoscope system is used by a doctor. [Figure 2] 1 is a conceptual diagram showing an example of the overall configuration of an endoscope system. [Figure 3] FIG. 2 is a block diagram showing an example of a hardware configuration of an electrical system of the endoscope system. [Figure 4] 10 is a conceptual diagram showing an example of a mode in which feature recognition processing is performed according to the sharpness of an image of interest that includes sharp and unsharp regions. FIG. [Figure 5] FIG. 10 is a conceptual diagram showing an example of a mode in which feature recognition processing is performed according to the sharpness of an image of interest that shows a lesion in which a halation portion is present. [Figure 6]FIG. 10 is a conceptual diagram showing an example of a mode in which feature recognition processing is performed according to the sharpness of an image of interest in which the edge of a frame is captured together with a lesion. [Figure 7] 2 is a block diagram showing an example of the main functions of a processor included in the medical support device and an example of information stored in a storage. FIG. [Figure 8] 10 is a conceptual diagram showing an example of an aspect in which a recognition unit executes a first region recognition process and a control unit displays various information on a screen. FIG. [Figure 9] 10 is a conceptual diagram showing an example of a mode in which a second region recognition process is executed by a recognition unit. FIG. [Figure 10] FIG. 10 is a conceptual diagram showing an example of how the recognition unit reduces the outer shape of the segmentation mask and fits the outer edge of the segmentation mask to the image to be processed. [Figure 11] 10 is a conceptual diagram showing an example of an aspect in which an inner region is extracted from a processing target image by a recognition unit, and a first control process is executed in accordance with the sharpness of the inner region. FIG. [Figure 12] FIG. 2 is a conceptual diagram illustrating an example of the processing content of feature recognition processing executed by a recognition unit. [Figure 13] FIG. 10 is a conceptual diagram showing an example of a mode in which a second region recognition process is executed on a processing target image that shows a lesion in which a halation area exists. [Figure 14] This is a conceptual diagram showing an example of how the blank area in the segmentation mask obtained by performing the second region recognition process on the processing target image, which shows a lesion with halation areas, is enlarged, and the outline of the segmentation mask, with its outer shape reduced, is fitted to the processing target image. [Figure 15] 10 is a conceptual diagram showing an example of an aspect in which an inner region is extracted by a recognition unit from a processing target image to which the contour of a segmentation mask has been fitted, and a first control process is executed according to the sharpness of the inner region. FIG. [Figure 16] 10 is a conceptual diagram showing an example of a mode in which a feature recognition process is executed by a recognition unit on an inner region extracted from a processing target image. FIG. [Figure 17] FIG. 10 is a conceptual diagram showing an example of a mode in which the second region recognition process is executed on a processing target image in which a lesion in which a halation portion exists and an edge of a frame are captured. [Figure 18] This is a conceptual diagram showing an example of how the blank area in the segmentation mask obtained by performing the second region recognition process on the processing target image, which shows the edge of the frame along with a lesion with halation areas, is enlarged, and the outline of the segmentation mask, with its outer shape reduced, is fitted to the processing target image. [Figure 19] 10 is a conceptual diagram showing an example of an aspect in which an inner region is extracted by a recognition unit from a processing target image to which the contour of a segmentation mask has been fitted, and a first control process is executed according to the sharpness of the inner region. FIG. [Figure 20] 10 is a conceptual diagram showing an example of a mode in which a feature recognition process is executed by a recognition unit on an inner region extracted from a processing target image. FIG. [Figure 21] FIG. 10 is a conceptual diagram showing an example of a manner in which a control unit displays processing result information on a screen. [Figure 22] 10 is a flowchart showing an example of the flow of medical support processing. [Figure 23] FIG. 10 is a conceptual diagram showing a modified example of how an inner region is extracted from an image to be processed. [Figure 24] FIG. 4 is a conceptual diagram showing an example of a mode in which a second control process is executed by a control unit. [Figure 25] 10 is a flowchart showing a modified example of the flow of medical support processing. [Figure 26] FIG. 10 is a conceptual diagram showing an example of how the sharpness obtained based on the latest frame is updated based on multiple sharpness indices and multiple malignancy levels obtained based on multiple frames in chronological order. [Figure 27]A conceptual diagram showing an example of a series of processes in which a processor included in a computer issues a processing execution instruction to an external device via a network, the external device executes processing in accordance with the processing execution instruction, and the processor included in the computer receives the processing result from the external device. DETAILED DESCRIPTION OF THE INVENTION

[0026] Hereinafter, exemplary embodiments of a medical support device, an endoscope system, a medical support method, and a program according to the present disclosure will be described with reference to the accompanying drawings. Note that the present disclosure can also be applied to a program and a computer program product.

[0027] First, the terms used in the following description will be explained.

[0028] CPU is an abbreviation for "Central Processing Unit". GPU is an abbreviation for "Graphics Processing Unit". GPGPU is an abbreviation for "General-Purpose computing on Graphics Processing Units". APU is an abbreviation for "Accelerated Processing Unit". TPU is an abbreviation for "Tensor Processing Unit". RAM is an abbreviation for "Random Access Memory". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". SSD is an abbreviation for "Solid State Drive". USB is an abbreviation for "Universal Serial Bus". EL is an abbreviation for "Electro-Luminescence". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". AI is an abbreviation for "Artificial Intelligence". WLI is an abbreviation for "White Light Imaging". BLI is an abbreviation for "Blue Light Imaging". LCI is an abbreviation for "Linked Color Imaging". NBI is an abbreviation for "Narrow Band Imaging". CT is an abbreviation for "Computed Tomography". MRI is an abbreviation for "Magnetic Resonance Imaging". Mask R-CNN is an abbreviation for "Mask Regional Convolutional Neural Network". I / F is an abbreviation for "Interface". SSL is an abbreviation for "Sessile Serrated Lesion".LAN is an abbreviation for "Local Area Network." WAN is an abbreviation for "Wide Area Network." 5G is an abbreviation for "5th Generation Mobile Communication System."

[0029] In the following description, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU, a GPU, a GPGPU, an APU, and a TPU.

[0030] In the following description, a signed memory is a memory such as a RAM in which information is temporarily stored, and is used as a work memory by a processor.

[0031] In the following description, the term "storage" refers to one or more nonvolatile storage devices that store various programs, various parameters, etc. Examples of nonvolatile storage devices include flash memory, magnetic disks, and magnetic tapes. Another example of storage is cloud storage.

[0032] In the following embodiments, the external I / F with a symbol controls the exchange of various information between multiple devices connected to each other. An example of the external I / F is a USB interface. A communication I / F including a communication processor, an antenna, etc. may be applied to the external I / F. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G, Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0033] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0034] Fig. 1 is a conceptual diagram showing an example of an embodiment in which an endoscope system 10 is used. As shown in Fig. 1, the endoscope system 10 is used by a doctor 12 in an endoscopic examination or the like. The endoscopic examination is assisted by a staff member 14, such as a nurse.

[0035] The endoscopic system 10 includes an endoscope 16, a display device 18, a control device 20, a light source device 22, and a medical support device 24. In this embodiment, the endoscopic system 10 is an example of an "endoscopic system" according to the present disclosure. Also, in this embodiment, the medical support device 24 is an example of a "medical support device" according to the present disclosure.

[0036] The endoscopy system 10 is a modality that allows a physician 12 to use an endoscope 16 to examine a large intestine 28 contained within the body of a subject 26 (e.g., a patient). In this embodiment, the large intestine 28 is an example of a "site" according to the present disclosure. Here, a lower endoscopy is illustrated as an example for the purpose of examining the large intestine 28, but this is merely one example, and the present disclosure is also applicable to endoscopic examinations of hollow organs other than the large intestine 28 (e.g., hollow organs such as the esophagus, stomach, duodenum, or trachea), such as upper endoscopy.

[0037] The endoscope 16 is used by a doctor 12 and inserted into a large intestine 28 of a subject 26. The endoscope system 10 causes the endoscope 16 inserted into the large intestine 28 to capture images of the inside of the large intestine 28, and performs various medical procedures on the large intestine 28 as necessary.

[0038] The endoscope 16 irradiates light 30 into the large intestine 28. The endoscope 16 captures an image of subject light, which is light that is reflected by an area including an intestinal wall 32 as a result of irradiating the light 30.

[0039] The display device 18, the control device 20, the light source device 22, and the medical support device 24 are installed on a wagon 34. The wagon 34 has a plurality of stands arranged vertically, and the medical support device 24, the light source device 22, and the control device 20 are installed from the lower stand to the upper stand. The display device 18 is installed on the top stand of the wagon 34.

[0040] The control device 20 controls the entire endoscope system 10. For example, the control device 20 is used for the endoscope 16, the display device 18, the light source device 22, the medical support device 24, etc., and the endoscope 16, the display device 18, the light source device 22, the medical support device 24, etc. are controlled by the control device 20.

[0041] The light source device 22 generates light 30 under the control of the control device 20 and supplies the generated light 30 to the endoscope 16. The endoscope 16 has a built-in light guide (not shown), and the light 30 supplied from the light source device 22 is irradiated from the tip of the endoscope 16 via the light guide.

[0042] The medical support device 24 performs various image processing on the images obtained by the endoscope 16 under the control of the control device 20 .

[0043] The display device 18 displays various information including images. Examples of the display device 18 include a liquid crystal display and an EL display. Alternatively, instead of the display device 18, or together with the display device 18, a tablet terminal with a display may be used.

[0044] The display device 18 has a screen 35. The screen 35 includes a plurality of display areas. The plurality of display areas are arranged side by side within the screen 35. In the example shown in FIG. 1, a first display area 35A and a second display area 35B are shown as examples of the plurality of display areas. The size of the first display area 35A is larger than the size of the second display area 35B. The first display area 35A is used as a main display area, and the second display area 35B is used as a sub-display area. The size relationship between the first display area 35A and the second display area 35B is not limited to this, and may be any size relationship that fits within the screen 35.

[0045] A moving image 39 is displayed in the first display area 35A. The moving image 39 is a moving image obtained by imaging the inside of the large intestine 28 of the subject 26 with the endoscope 16. In the example shown in FIG. 1, a moving image showing the intestinal wall 32 is shown as an example of the moving image 39.

[0046] The intestinal wall 32 shown in the moving image 39 (i.e., the intestinal wall 32 included as an image in the moving image 39) includes a lesion 42 (e.g., one lesion 42 in the example shown in FIG. 1 ) as a region of interest (i.e., a region to be observed) that is gazed upon by the doctor 12. Through the moving image 39, the doctor 12 can visually recognize the appearance of the intestinal wall 32, including the lesion 42. In this embodiment, the lesion 42 is an example of a "characteristic region" and a "lesion" according to the present disclosure.

[0047] There are various types of lesions 42, and examples of the types of lesions 42 include adenoma, inflammatory polyp, hyperplastic polyp, serrated polyp, lymphoid polyp, adenocarcinoma, mucinous adenocarcinoma, sclerosing adenocarcinoma, squamous cell carcinoma, and malignant lymphoma. Note that the types exemplified here are types that are anticipated in advance as types of lesions 42 when an endoscopic examination is performed on the large intestine 28, and the types of lesions 42 will differ depending on the organ that is subjected to the endoscopic examination.

[0048] In this embodiment, for the sake of convenience, an example is given in which one lesion 42 is captured in the video 39, but the present disclosure is not limited to this and applies even if multiple lesions 42 are captured in the video 39.

[0049] In this embodiment, a lesion 42 is illustrated, but this is merely one example, and the area of ​​interest (i.e., the area to be observed) that is gazed upon by the doctor 12 may be a characteristic area having some unique feature, such as an organ (e.g., the duodenal papilla), a mark, an artificial treatment device (e.g., an artificial clip), or a treated area (e.g., an area where traces remain after the removal of a polyp, etc.).

[0050] The moving image 39 displayed in the first display area 35A is a moving image including a plurality of frames 40 arranged in chronological order. That is, the plurality of frames 40 arranged in chronological order are displayed in the first display area 35A at a default frame rate (e.g., several tens of frames per second). Examples of the default frame rate include 15 frames per second, 30 frames per second, and 60 frames per second. In this embodiment, the frames 40 are an example of a "medical image" and an "endoscopic image" according to the present disclosure.

[0051] An example of a moving image displayed in first display area 35A is a moving image in a live view format. The live view format is merely one example, and a moving image that is temporarily stored in a memory or the like and then displayed, such as a moving image in a post-view format, may also be used. Furthermore, each frame included in a moving image for recording stored in a memory or the like may be reproduced and displayed on screen 35 (for example, first display area 35A) as moving image 39.

[0052] Within screen 35, second display area 35B is adjacent to first display area 35A, and is displayed in the lower right corner as viewed from the front within screen 35. The display position of second display area 35B may be anywhere within screen 35 of display device 18, but it is preferable that it be displayed in a position where it can be contrasted with moving image 39.

[0053] The second display area 35B displays auxiliary information 44 that assists the doctor 12 in making medical decisions during the endoscopic examination. The auxiliary information 44 is information that is referenced by the doctor 12. Examples of the auxiliary information 44 include various types of information about the subject 26 into whose body the endoscope 16 is inserted, and various types of information obtained by performing medical support processing, which will be described later.

[0054] Fig. 2 is a conceptual diagram showing an example of the overall configuration of the endoscope system 10. As shown in Fig. 2, the endoscope 16 includes an operation unit 46 and an insertion unit 48. The insertion unit 48 is partially curved by operating the operation unit 46. The insertion unit 48 is inserted into the large intestine 28 (see Fig. 1) while curving in accordance with the shape of the large intestine 28 (see Fig. 1) in accordance with the operation of the operation unit 46 by the doctor 12 (see Fig. 1).

[0055] A camera 52, an illumination device 54, and a treatment tool opening 56 are provided at the distal end 50 of the insertion section 48. The camera 52 and the illumination device 54 are provided at the distal end 50. An objective lens of the camera 52 is provided at a distal end surface 50A of the distal end 50.

[0056] The camera 52 is mounted on the endoscope 16 and is inserted into the body cavity of the subject 26 to capture images of an area to be observed, thereby generating frames 40. In this embodiment, the camera 52 captures images of the inside of the subject 26 (for example, the inside of the large intestine 28) to generate a moving image 39 including a plurality of frames 40 in chronological order. An example of the camera 52 is a CMOS camera. However, this is merely an example, and other types of cameras such as a CCD camera may also be used.

[0057] The illumination device 54 has illumination windows 54A and 54B. The illumination windows 54A and 54B are provided on the distal end surface 50A. The illumination device 54 emits light 30 (see FIG. 1 ) through the illumination windows 54A and 54B. Examples of the light 30 emitted from the illumination device 54 include light for WLI (e.g., white light), light for LCI (e.g., a combination of red, green, and blue light), light for BLI (e.g., blue light), and / or light for NBI (e.g., a combination of blue and green light). The camera 52 captures images of the inside of the large intestine 28 by an optical method while the illumination device 54 emits light 30 (see FIG. 1 ) into the large intestine 28.

[0058] The treatment tool opening 56 is an opening for allowing a treatment tool 58 to protrude from the distal end portion 50. The treatment tool opening 56 is also used as a suction port for sucking blood, internal waste, etc., and as a delivery port for delivering fluid.

[0059] A treatment tool insertion port 60 is formed in the operation section 46, and the treatment tool 58 is inserted into the insertion section 48 from the treatment tool insertion port 60. The treatment tool 58 passes through the insertion section 48 and protrudes to the outside from the treatment tool opening 56. In the example shown in FIG. 2, a puncture needle is shown as the treatment tool 58 protruding from the treatment tool opening 56. Here, a puncture needle is shown as the treatment tool 58, but this is merely one example, and the treatment tool 58 may also be a grasping forceps, a papillotomy knife, a snare, a catheter, a guidewire, a cannula, and / or a puncture needle with a guide sheath, etc.

[0060] The endoscope 16 is connected to a control device 20 and a light source device 22 via a universal cord 62. A medical support device 24 and a reception device 64 are connected to the control device 20. A display device 18 is also connected to the medical support device 24. That is, the control device 20 is connected to the display device 18 via the medical support device 24.

[0061] Here, the medical support device 24 is exemplified as an external device for expanding the functions performed by the control device 20, and therefore an example is given in which the control device 20 and the display device 18 are indirectly connected via the medical support device 24, but this is merely one example. For example, the display device 18 may be directly connected to the control device 20. In this case, for example, the function of the medical support device 24 may be installed in the control device 20, or the control device 20 may be equipped with a function to cause a server (not shown) to execute the same processing as that executed by the medical support device 24 (for example, the medical support processing described below) and receive and use the processing results from the server.

[0062] The reception device 64 receives instructions from the doctor 12 and outputs the received instructions as an electrical signal to the control device 20. Examples of the reception device 64 include a keyboard, a mouse, a touch panel, a foot switch, a microphone, and / or a remote control device.

[0063] The control device 20 controls the light source device 22, exchanges various signals with the camera 52, and exchanges various signals with the medical support device 24.

[0064] The light source device 22 emits light under the control of the control device 20 and supplies the light to the illumination device 54. A light guide is built into the illumination device 54, and the light supplied from the light source device 22 passes through the light guide and is irradiated from illumination windows 54A and 54B. The control device 20 causes the camera 52 to capture an image, acquires a video 39 (see FIG. 1) from the camera 52, and outputs it to a predetermined output destination (for example, the medical support device 24).

[0065] The medical support device 24 supports medical treatment (here, as an example, an endoscopic examination) by performing various image processing on the moving image 39 input from the control device 20. The medical support device 24 outputs the moving image 39 that has been subjected to various image processing to a predetermined output destination (for example, the display device 18).

[0066] Although the embodiment in which the moving image 39 output from the control device 20 is output to the display device 18 via the medical support device 24 has been described above, this is merely one example. For example, the control device 20 and the display device 18 may be connected, and the moving image 39 that has been subjected to image processing by the medical support device 24 may be displayed on the display device 18 via the control device 20.

[0067] Fig. 3 is a block diagram showing an example of the hardware configuration of the electrical system of the endoscope system 10. As shown in Fig. 3, the control device 20 includes a computer 66, a bus 68, and an external I / F 70. The computer 66 includes a processor 72, a memory 74, and a storage 76. The processor 72, the memory 74, the storage 76, and the external I / F 70 are connected to the bus 68. The processor 72 controls the entire control device 20. The memory 74 and the storage 76 are used by the processor 72.

[0068] The external I / F 70 controls the exchange of various information between the processor 72 and one or more devices (hereinafter also referred to as "first external devices") that exist outside the control device 20.

[0069] A camera 52 is connected to the external I / F 70 as one of the first external devices. The external I / F 70 controls the exchange of various information between the camera 52 and the processor 72. The processor 72 controls the camera 52 via the external I / F 70. The processor 72 also acquires, via the external I / F 70, a moving image 39 (see FIG. 1) obtained by the camera 52 capturing an image of the inside of the large intestine 28 (see FIG. 1).

[0070] The camera 52 has an optical system 52B and an image sensor 52A. Examples of the image sensor 52A include a CMOS image sensor or a CCD image sensor. The optical system 52B is an optical system that realizes so-called optical zoom, and operates under the control of the processor 72. Operation of the optical system 52B changes the magnification of the moving image 39 (see FIG. 1). In other words, the frame 40 (see FIG. 1) is optically zoomed in or out.

[0071] The image sensor 52A receives subject light incident on the optical system 52B and generates an electrical signal corresponding to the subject light by photoelectrically converting the received subject light. The camera 52 includes a signal processing circuit (not shown) connected to the image sensor 52A. The signal processing circuit acquires the electrical signal from the image sensor 52A and performs various signal processing, including A / D conversion, on the acquired electrical signal to generate frames 40 (see FIG. 1) at a predetermined frame rate (e.g., a predetermined frame rate such as 15 frames / second, 30 frames / second, or 60 frames / second).

[0072] Each time a frame 40 (see FIG. 1) is generated by the camera 52 at a predetermined frame rate, the generated frame 40 is acquired by the processor 72. That is, the processor 72 acquires from the camera 52 a moving image 39 (see FIG. 1) that is configured to include a plurality of frames 40 in time series.

[0073] Here, an example has been given in which frame 40 is generated by a signal processing circuit of camera 52 performing various signal processes on an electrical signal corresponding to subject light, but this is merely one example. For example, control device 20 may have a signal processing circuit, and the signal processing circuit of control device 20 may generate frame 40 by acquiring an electrical signal corresponding to subject light from image sensor 52A and performing various signal processes on it.

[0074] The light source device 22 is connected to the external I / F 70 as one of the first external devices, and the external I / F 70 controls the exchange of various information between the light source device 22 and the processor 72. The light source device 22 supplies light to the illumination device 54 under the control of the processor 72. The illumination device 54 irradiates the light supplied from the light source device 22.

[0075] A reception device 64 is connected to the external I / F 70 as one of the first external devices, and the processor 72 acquires instructions accepted by the reception device 64 via the external I / F 70 and executes processing according to the acquired instructions.

[0076] The medical support device 24 includes a computer 78 and an external I / F 80. The computer 78 includes a processor 82, a memory 84, and a storage 86. The processor 82, the memory 84, the storage 86, and the external I / F 80 are connected to a bus 88. In this embodiment, the computer 78 is an example of a "computer" according to the present disclosure, and the processor 82 is an example of a "processor" according to the present disclosure.

[0077] The hardware configuration of the computer 78 (i.e., the processor 82, the memory 84, and the storage 86) is basically the same as the hardware configuration of the computer 66, so a description of the hardware configuration of the computer 78 will be omitted here.

[0078] The external I / F 80 controls the exchange of various information between the processor 82 and one or more devices (hereinafter also referred to as "second external devices") that exist outside the medical support device 24.

[0079] The control device 20 is connected to the external I / F 80 as one of the second external devices. In the example shown in Fig. 3, the external I / F 70 of the control device 20 is connected to the external I / F 80. The external I / F 80 controls the exchange of various information between the processor 82 of the medical support device 24 and the processor 72 of the control device 20. For example, the processor 82 acquires a moving image 39 (see Fig. 1) from the processor 72 of the control device 20 via the external I / Fs 70 and 80, and performs various image processing on the acquired moving image 39.

[0080] The display device 18, which serves as one of the second external devices, is connected to the external I / F 80. The processor 82 controls the display device 18 via the external I / F 80, thereby causing the display device 18 to display various information (for example, a moving image 39 that has been subjected to various image processing).

[0081] Recently, there has been progress in the development of technology that recognizes the medical features of a lesion 42 captured in a frame 40 (i.e., the lesion 42 contained as an image in the frame 40) using a trained model that has undergone machine learning on the medical features of a lesion captured in an image (e.g., an endoscopic image). Here, the medical features refer to the type of the lesion 42 (e.g., adenoma, inflammatory polyp, hyperplastic polyp, serrated polyp, lymphoid polyp, adenocarcinoma, mucinous adenocarcinoma, sclerosing adenocarcinoma, squamous cell carcinoma, malignant lymphoma, etc.), the form of the lesion 42 (e.g., pedunculated, subpedunculated, sessile, elevated, flat, depressed, etc.), and / or the malignancy of the lesion 42.

[0082] For example, the recognition results by the trained model, i.e., the medical features, are displayed on the screen 35, etc. The doctor 12 can provide appropriate treatment to the subject 26 by referring to the medical features displayed on the screen 35, etc.

[0083] However, if the image input to the trained model is blurred or otherwise unclear, it is difficult to obtain highly reliable recognition results from the trained model.

[0084] Therefore, in the conventionally known technology, AI recognition processing is performed on the image of interest, which shows the lesion 42, only when the degree of blur of the image of interest in the frame 40 is less than a reference level. Below, an overview of the conventionally known technology will be described with reference to Figures 4 to 6.

[0085] 4 to 6, an object recognition process using AI is performed, and thereby lesion 42 captured in frame 40 is recognized by a bounding box method. That is, by performing object recognition process on frame 40, the image area in frame 40 capturing lesion 42 is surrounded by a bounding box 90 (a rectangular frame in the examples shown in FIGS. 4 to 6) as an image of interest 92.

[0086] In the example shown in FIG. 4, image of interest 92 includes sharp region 92A, which is a sharp image region, and unsharp region 92B, which is an unsharp image region. An example of sharp region 92A is an image region that is in focus. An example of unsharp region 92B is an image region that is out of focus (for example, an image region that shows an aspect closer to the foreground than sharp region 92A and / or an image region that shows an aspect closer to the background). Note that, although an in-focus image region is given as an example of sharp region 92A and an out-of-focus image region is given as an example of unsharp region 92B, this is merely an example, and sharp region 92A may be an image region within the depth of field, and unsharp region 92B may be an image region outside the depth of field.

[0087] In the example shown in Figure 4, lesion 42 shown in frame 40 is in focus in image of interest 92, while image areas other than lesion 42 are out of focus. That is, in image of interest 92, the high-frequency components of sharp region 92A are greater than the high-frequency components of unsharp region 92B. In the example shown in Figure 4, outer edge 42A of lesion 42 (i.e., the edge of lesion 42) in image of interest 92 is the boundary between sharp region 92A and unsharp region 92B, and has the greatest high-frequency components in image of interest 92.

[0088] In the example shown in FIG. 4, the sharpness of the target image 92 is calculated. One example of a method for calculating the sharpness of the target image 92 is a calculation method performed by the following steps (1) to (3). First, in step (1), a Laplacian filter is applied to the target image 92 to enhance edges such as the outer edge 42A. Next, in step (2), the sum (hereinafter simply referred to as the "sum") of the absolute values ​​of the pixel values ​​of each pixel of the target image 92 to which the Laplacian filter has been applied is calculated. Then, in step (3), the sum is divided by the number of pixels in the target image 92. The division result obtained in step (3) represents the average value of the edge strength of the target image 92.

[0089] The division result (i.e., the average value) obtained in step (3) is the sharpness of the target image 92. The higher the division result obtained in step (3), the sharper the target image 92 is, and the lower the division result obtained in step (3), the less sharp the target image 92 is.

[0090] 4, if the sharpness of the image of interest 92 is equal to or greater than a threshold, a feature recognition process 94 is executed on the image of interest 92. The feature recognition process 94 is a process for recognizing medical features using an AI method (i.e., a recognition process using a trained model that has undergone machine learning on the medical features of the lesion shown in the image). The processing results obtained by executing the feature recognition process 94 on the image of interest 92 or information based on the processing results are provided to the doctor 12 via the screen 35 or the like.

[0091] However, when the outer edge 42A of the image of interest 92 (i.e., the outer edge 42A included as an image in the image of interest 92) is emphasized by the Laplacian filter, the condition "sharpness of the image of interest 92 ≧ threshold" may be satisfied even though most of the image of interest 92 is an unsharp region 92B. Also, as shown in FIG. 5 as an example, an edge 96A of a halation area 96 (in other words, a light-reflecting area or a glossy area) that occurs when light 30 is irradiated onto a bubble in the lesion 42 shown in the image of interest 92 (i.e., the lesion 42 included as an image in the image of interest 92) is also emphasized by the Laplacian filter. Also, as shown in FIG. 6 as an example, when the edge 40A (i.e., the outer edge) of the frame 40 is included in the image of interest 92, the edge 40A is emphasized by the Laplacian filter. Although edges 40A and 96A are unnecessary information for feature recognition processing 94 to recognize the medical features of lesion 42, they act as factors that satisfy the condition "sharpness of target image 92≧threshold value."

[0092] In this case, even though the image of interest 92 is an image with a sharpness that would result in medical features being erroneously recognized, the feature recognition process 94 is executed on the image of interest 92, and as a result, there is a risk that the doctor 12 or the like will be provided with an unreliable processing result from the feature recognition process 94, or information based on the unreliable processing result from the feature recognition process 94.

[0093] In view of the above circumstances, in this embodiment, as shown in FIG. 7 as an example, medical support processing is performed by a processor 82 of the medical support device 24.

[0094] A medical support program 98 is stored in the storage 86. The medical support program 98 is an example of a "program" according to the present disclosure. The processor 82 reads the medical support program 98 from the storage 86 and executes the read medical support program 98 on the memory 84 to perform medical support processing. The medical support processing is realized by the processor 82 operating as a recognition unit 82A and a control unit 82B in accordance with the medical support program 98 executed on the memory 84.

[0095] The storage 86 stores a feature recognition model 100, a first area recognition model 102, and a second area recognition model 104. As will be described in detail later, the feature recognition model 100, the first area recognition model 102, and the second area recognition model 104 are used by the recognition unit 82A.

[0096] 8, the recognition unit 82A and the control unit 82B acquire each of a plurality of frames 40 in chronological order included in a moving image 39 generated by capturing images by the camera 52 at an imaging frame rate (for example, several tens of frames per second) from the camera 52, one frame at a time. The recognition unit 82A and the control unit 82B acquire the frames 40 at the same time.

[0097] The control unit 82B outputs the moving image 39 to the display device 18. For example, the control unit 82B displays the moving image 39 as a live view image in the first display area 35A. That is, each time the control unit 82B acquires a frame 40 from the camera 52, the control unit 82B displays the acquired frame 40 in the first display area 35A in order at a default frame rate. The control unit 82B also displays auxiliary information 44 in the second display area 35B. The control unit 82B also updates the display content of the second display area 35B (for example, the auxiliary information 44) in accordance with the display content of the first display area 35A.

[0098] The recognition unit 82A recognizes a lesion 42 in the moving image 39 based on the moving image 39 acquired from the camera 52. That is, the recognition unit 82A sequentially performs a first region recognition process 105 on each of a plurality of frames 40 in chronological order included in the moving image 39 acquired from the camera 52, thereby recognizing the lesion 42 appearing in the frame 40.

[0099] The first region recognition process 105 is performed by the recognition unit 82A on the acquired frame 40 each time the frame 40 is acquired. The first region recognition process 105 is a process that recognizes the lesion 42 using an AI method. Here, the first region recognition process 105 is a process that uses the first region recognition model 102.

[0100] The first region recognition model 102 is a trained model for object recognition using an AI bounding box method. The first region recognition model 102 is also a trained model obtained by optimizing a neural network by performing machine learning on the neural network using first training data. The first training data is a data set including a plurality of data (i.e., data for a plurality of frames) in which first example data and first correct answer data are associated with each other.

[0101] The first example data is an image that assumes frame 40. A first example of an image that assumes frame 40 is an image obtained by actually capturing an image of the inside of the large intestine with a camera. A second example of an image that assumes frame 40 is a virtually created image (e.g., an image generated by a generation AI). The first correct answer data is correct answer data (i.e., annotation) for the first example data. Here, an example of the first correct answer data is position information (e.g., coordinates) that indicates the position of a rectangular area surrounding an image area that includes a lesion that appears in the image used as the first example data.

[0102] The recognition unit 82A acquires a frame 40 from the camera 52 and inputs the acquired frame 40 to the first region recognition model 102. As a result, each time a frame 40 is input, the first region recognition model 102 recognizes an image region including a lesion 42 shown in the input frame 40 and generates and outputs a first region recognition result 106, which is a recognition result. The first region recognition result 106 includes position information indicating the position of a rectangular region surrounding the image region including the lesion 42, as well as a frame image 106A indicating the outer edge of the rectangular region identified from the position information. The frame image 106A corresponds to a bounding box. In other words, the geometric characteristics (e.g., size, position, and shape) of the frame image 106A match the geometric characteristics of the bounding box.

[0103] Based on first region recognition result 106 input from recognition unit 82A, control unit 82B superimposes frame image 106A on frame 40 displayed in first display region 35A, thereby surrounding lesion 42 shown in frame 40 displayed in first display region 35A with frame image 106A.

[0104] 9, the recognition unit 82A extracts a processing target image 108, which is a portion surrounded by the frame image 106A, from the same frame 40 as the frame 40 input to the first region recognition model 102 to obtain the first region recognition result 106 including the frame image 106A. The processing target image 108 shows a lesion 42 and the background side of the lesion 42 (the lumen in the example shown in FIG. 9). Because the processing target image 108 shows the lesion 42, the processing target image 108 can also be said to be an image of interest, which is an image that is attracting the attention of the doctor 12.

[0105] The recognition unit 82A recognizes the lesion 42 appearing in the processing target image 108 (i.e., the lesion 42 included as an image in the processing target image 108) by performing a second region recognition process 107 on the processing target image 108. In this embodiment, the second region recognition process 107 is an example of the "region recognition process" according to the present disclosure.

[0106] The second region recognition process 107 is performed for each processing target image 108 extracted from the same frame 40 as the frame 40 on which the first region recognition process 105 was performed. The second region recognition process 107 is a process for recognizing a lesion 42 using an AI method. The second region recognition process 107 is performed using a second region recognition model 104.

[0107] The second region recognition model 104 is a trained model for object recognition using an AI segmentation method. The second region recognition model 104 is a trained model obtained by optimizing a neural network by performing machine learning on the neural network using second training data. The second training data is a data set including multiple data (i.e., multiple frames of data) in which second example data and second answer data are associated with each other.

[0108] The second example data is an image that is an example of the processing target image 108. A first example of an image that is an example of the processing target image 108 is an image in which an image area including a lesion that appears in an image actually obtained by capturing an image of the inside of the large intestine with a camera is cut out into a rectangular shape. A second example of an image that is an example of the processing target image 108 is a virtually created image (e.g., an image generated by a generation AI). The second correct answer data is correct answer data (i.e., annotation) for the second example data. Here, an example of the second correct answer data is a lesion label assigned to each pixel of an image area that shows a lesion that appears in the image used as the second example data. A lesion label refers to a label that indicates that it is a lesion.

[0109] The recognition unit 82A inputs the processing target image 108 to the second region recognition model 104. As a result, each time the processing target image 108 is input, the second region recognition model 104 recognizes an image region representing a lesion 42 in the input processing target image 108 on a pixel-by-pixel basis, and generates and outputs a recognition result, a second region recognition result 110. The second region recognition result 110 includes position information (e.g., coordinates) indicating the position of each pixel in the image region representing the lesion 42 in the processing target image 108, as well as a segmentation mask 110A that can identify the position of the image region representing the lesion 42 in the processing target image 108. The segmentation mask 110A is a mask generated by a segmentation algorithm (e.g., U-net or Mask R-CNN).

[0110] The resolution of the segmentation mask 110A is lower than the resolution of the frame 40. Each pixel of the segmentation mask 110A is associated with position information (e.g., coordinates) indicating the corresponding position in the processing target image 108 and a confidence level that the pixel is a lesion 42. In this embodiment, the segmentation mask 110A is an example of a "segmentation mask" according to the present disclosure.

[0111] 10 , the recognition unit 82A reduces the outer shape of the segmentation mask 110A by offsetting the outer edge 110A1 of the segmentation mask 110A to the inside of the segmentation mask 110A. Within the segmentation mask 110A, the confidence level (i.e., the probability that the edge is a lesion 42 or an index indicating the likelihood of the edge being a lesion 42) on the outer edge 110A after being offset to the inside of the segmentation mask 110A is higher than the confidence level on the outer edge 110A1 before being offset to the inside of the segmentation mask 110A. This is because the outer edge 110A1 after being offset to the inside of the segmentation mask 110A is farther away from the area outside the lesion 42 and closer to the area with a higher likelihood of the edge being a lesion 42 (i.e., the confidence level) than the outer edge 110A1 before being offset to the inside of the segmentation mask 110A.

[0112] An example of the direction in which the outer edge 110A1 is offset is a direction toward the center of the circumscribed rectangular area of ​​the segmentation mask 110A, or toward the center of gravity of the segmentation mask 110A. An example of the offset amount by which the outer edge 110A1 is offset is a predetermined amount (e.g., the number of pixels) that positions the outer edge 110A1 more inward than the outer edge 42A of the lesion 42 depicted in the processing target image 108. Note that, although an example in which the outer edge 110A1 is offset has been given here, this is merely an example, and the outer shape of the segmentation mask 110A may be reduced by performing an erosion process (e.g., morphological transformation erosion process) on the segmentation mask 110A.

[0113] The recognition unit 82A fits the outer edge 110A1 of the segmentation mask 110A, which has a reduced outer shape, to the processing target image 108. That is, the recognition unit 82A maps the pixels of the outer edge 110A1 of the segmentation mask 110A to corresponding positions in the processing target image 108. Here, the corresponding positions in the processing target image 108 refer to positions indicated by position information associated with each pixel of the outer edge 110A1 of the segmentation mask 110A. When the outer edge 110A1 of the segmentation mask 110A, which has a reduced outer shape, is fitted to the processing target image 108 in this way, the outer edge 110A1 of the segmentation mask 110A is located inside the outer edge 42A of the lesion 42.

[0114] As an example, as shown in FIG. 11 , the recognition unit 82A extracts an area based on the segmentation mask 110A as an inner area 109A from the processing target image 108. The inner area 109A is an image area in the processing target image 108 that is more inward than the outer edge 42A of the lesion 42 shown in the processing target image 108. The image area in the processing target image 108 that is more inward than the outer edge 42A of the lesion 42 shown in the processing target image 108 refers to the image area inside the outer edge 110A1 of the segmentation mask 110A, which has a smaller outer shape, of the entire image area of ​​the processing target image 108. In the example shown in FIG. 11 , the image of the portion of the processing target image 108 that is surrounded by the outer edge 110A1 is extracted from the processing target image 108 by the recognition unit 82A as the inner area 109A. The outer edge 42A of the lesion 42 is not included in the inner area 109A. This means that the inner region 109A does not contain high frequency components at the same level as the outer edge 42A. In this embodiment, the inner region 109A is an example of the "inner region" according to the present disclosure.

[0115] Recognition unit 82A calculates sharpness 112A of inner region 109A. The method for calculating sharpness 112A of inner region 109A is the same as the method for calculating the sharpness of target image 92 in the example shown in FIGS. 4 to 6. Sharpness 112A can be said to be more reliable than sharpness 112A of processing target image 108. This is because processing target image 108 includes an image region corresponding to unsharp region 92B shown in FIG. 4. Therefore, when the sharpness of processing target image 108 is calculated, the calculation result is affected by edge components (i.e., high-frequency components) such as outer edge 42A of lesion 42. However, sharpness 112A of inner region 109A is hardly affected by outer edge 42A of lesion 42 (i.e., high-frequency components).

[0116] The recognition unit 82A performs a first control process 113 in accordance with the sharpness 112A. The first control process 113 is a process for controlling the feature recognition process 114. In the first control process 113, it is determined whether the sharpness 112A is equal to or greater than a threshold value TH. The threshold value TH is a value derived in advance by testing using an actual device and / or computer simulation, etc., as a lower limit value of the sharpness 112A at which the medical features of the lesion 42 captured in the inner region 109A (i.e., the lesion 42 included as an image in the inner region 109A) are not erroneously recognized by the feature recognition model 100 when the inner region 109A is input to the feature recognition model 100 (see FIGS. 7 and 12).

[0117] The first control process 113 includes a process in which the recognition unit 82A performs the feature recognition process 114 when the sharpness 112A is equal to or greater than the threshold value TH, and the recognition unit 82A does not perform the feature recognition process 114 when the sharpness 112A is less than the threshold value TH.

[0118] In this embodiment, the first control process 113 is an example of a "first process" according to the present disclosure. Also, in this embodiment, the feature recognition process 114 is an example of an "image recognition process" according to the present disclosure. Also, in this embodiment, the threshold value TH is an example of a "first threshold value" and a "second threshold value" according to the present disclosure.

[0119] As an example, as shown in Figure 12, feature recognition processing 114 is performed on frame 40 by recognition unit 82A. In the example shown in Figure 12, feature recognition processing 114 is performed on inner region 109A by recognition unit 82A using feature recognition model 100. Feature recognition processing 114 is processing that recognizes the features of lesion 42 using an AI method. Here, the features of lesion 42 refer to the medical features of lesion 42.

[0120] The feature recognition model 100 is a trained model for object recognition. The feature recognition model 100 is also a trained model obtained by optimizing a neural network by performing machine learning on the neural network using third training data. The third training data is a data set including multiple data (i.e., multiple frames of data) in which third example data and third correct answer data are associated with each other.

[0121] The third example data is an image that simulates an image showing a lesion 42 (e.g., inner region 109A). A first example of an image that simulates a lesion 42 is an image obtained by actually capturing an image of the inside of the large intestine using a camera. A second example of an image that simulates an image showing a lesion 42 is a virtually created image (e.g., an image generated by a generative AI). The third correct answer data is correct answer data (i.e., annotations) for the third example data. Here, an example of the third correct answer data is information indicating medical characteristics of the lesion shown in the image used as the third example data. Examples of medical characteristics of the lesion shown in the image used as the third example data include the type of lesion shown in the image used as the third example data, the type of lesion shown in the image used as the third example data, and / or the malignancy of the lesion shown in the image used as the third example data.

[0122] The recognition unit 82A inputs the inner region 109A to the feature recognition model 100. As a result, each time the inner region 109A is input, the feature recognition model 100 recognizes the medical features of the lesion 42 indicated by the input inner region 109A, and generates and outputs the feature recognition result 116A that is the recognition result.

[0123] As an example, as shown in FIG. 13, a halation area 117 may appear in the processing target image 108 along with a lesion 42. The halation area 117 refers to an area where halation occurs due to the reflection of light 30 by air bubbles or the like adhering to the lesion 42 (in other words, a light-reflecting area or a shiny area). When the halation area 117 is present in the lesion 42, the segmentation mask 110A included in the second region recognition result 110 has a blank area 110C corresponding to the halation area 117. The position of the blank area 110C in the segmentation mask 110A corresponds to the position of the halation area 117 in the processing target image 108.

[0124] Here, if the sharpness of the processing target image 108 containing the halation area 117 is calculated by the above-mentioned calculation method, the influence of the edge component (i.e., high-frequency component) of the halation area 117 will be reflected in the calculation result, and the reliability of the sharpness of the processing target image 108 will decrease.

[0125] Therefore, in this embodiment, as an example, as shown in FIGS. 14 and 15, the recognition unit 82A extracts an inner region 109B from the processing target image 108, taking into consideration the halation portion 117.

[0126] In this case, as shown in Fig. 14 as an example, the recognition unit 82A first offsets the outer edge 110A1 of the segmentation mask 110A and enlarges the blank region 110C of the segmentation mask 110A in the same manner as in the example shown in Fig. 10. The enlargement ratio used to enlarge the blank region 110C is determined according to the geometric characteristics (e.g., shape, size, etc.) of the segmentation mask 110A and the geometric characteristics (e.g., shape, position, size, etc.) of the blank region 110C within the segmentation mask 110A. The blank region 110C is enlarged to an area that is outer than the edge of the blank region 110C before enlargement and inner than the outer edge 110A1 of the segmentation mask 110A.

[0127] The confidence level of the region in segmentation mask 110A that is outside the edge of pre-enlargement blank region 110C and inside outer edge 110A1 of segmentation mask 110A is higher than the confidence level of the region on the edge of pre-enlargement blank region 110C in segmentation mask 110A. This is because the region in segmentation mask 110A that is outside the edge of pre-enlargement blank region 110C and inside outer edge 110A1 of segmentation mask 110A is farther from halation point 117, which is a region different from lesion 42, than the edge of pre-enlargement blank region 110C in segmentation mask 110A, and is closer to a region with a high degree of resemblance to lesion 42 (i.e., confidence level).

[0128] The recognition unit 82A fits the outer edge 110A1 of the reduced segmentation mask 110A and the edge 110C1 of the enlarged blank area 110C to the processing target image 108. That is, the recognition unit 82A maps the pixels of the outer edge 110A1 to corresponding positions in the processing target image 108, and maps the pixels of the edge 110C1 to corresponding positions in the processing target image 108. Here, the positions to which the pixels of the outer edge 110A1 are mapped are positions indicated by position information associated with each pixel of the outer edge 110A1 of the segmentation mask 110A. Furthermore, the positions to which the pixels of the edge 110C1 are mapped are positions indicated by position information associated with each pixel of the edge 110C1 of the segmentation mask 110A.

[0129] 15, recognition unit 82A extracts inner region 109B from processing target image 108. Inner region 109B is an area inward of outer edge 42A of lesion 42 shown in processing target image 108 and outward of the edge of halation portion 117. In the example shown in FIG. 15, an image of a portion surrounded by outer edge 110A1 and edge 110C1 in processing target image 108 is extracted from processing target image 108 as inner region 109B.

[0130] In this embodiment, the inner region 109B is an example of an "inner region" according to the present disclosure. Also, in this embodiment, the halation portion 117 is an example of a "halation portion" according to the present disclosure. Also, in this embodiment, the edge of the halation portion 117 is an example of an "edge of the halation portion" according to the present disclosure.

[0131] The recognition unit 82A calculates the sharpness 112B of the inner region 109B in the same manner as it calculated the sharpness 112A of the inner region 109A. The recognition unit 82A performs the first control process 113 using the sharpness 112B in the same manner as it performed the first control process 113 using the calculated sharpness 112A. If the sharpness 112B is equal to or greater than the threshold value TH, the recognition unit 82A inputs the inner region 109B to the feature recognition model 100, as shown in FIG. 16 as an example. As a result, each time the inner region 109B is input, the feature recognition model 100 recognizes the medical features of the lesion 42 indicated by the input inner region 109B and generates and outputs the feature recognition result 116B as the recognition result.

[0132] 17, when a portion of a lesion 42 captured in a frame 40 is outside the frame 40 (in other words, when a portion of the lesion 42 is out of frame), an edge 40A of the frame 40 appears together with the lesion 42 in the processing target image 108. When the edge 40A of the frame 40 appears together with the lesion 42 in the processing target image 108, the segmentation mask 110A has an edge 110D. The position of the edge 110D corresponds to the position of the edge 40A in the processing target image 108.

[0133] Here, if the sharpness of the processing target image 108 containing the edge 40A is calculated using the calculation method described above, the influence of the edge 40A (i.e., the high-frequency component) will be reflected in the calculation result, reducing the reliability of the sharpness of the processing target image 108.

[0134] 18 and 19, as an example, in this embodiment, recognition unit 82A offsets outer edge 110A1 of segmentation mask 110A to the inside of segmentation mask 110A in the same manner as the example shown in Fig. 10, enlarges blank area 110C of segmentation mask 110A in the same manner as the example shown in Fig. 14, and offsets edge 110D of segmentation mask 110A to the inside of segmentation mask 110A. As a result, the outer shape of segmentation mask 110A becomes smaller, and the occupancy rate of blank area 110C within segmentation mask 110A increases.

[0135] Within segmentation mask 110A, the confidence on outer edge 110D after it has been offset inside segmentation mask 110A is higher than the confidence on outer edge 110D before it was offset inside segmentation mask 110A because outer edge 110D after it has been offset inside segmentation mask 110A is further away from areas that are not lesions 42 (e.g., areas outside frame 40) and closer to areas that are more likely to be lesions 42 (i.e., confidence) than outer edge 110D before it was offset inside segmentation mask 110A.

[0136] An example of the direction in which outer edge 110D is offset is a direction toward the center of the circumscribed rectangular area of ​​segmentation mask 110A or the center of gravity of segmentation mask 110A. An example of the offset amount by which outer edge 110D is offset is a predetermined amount that positions outer edge 110D more inward than edge 40A shown in processing target image 108 (i.e., edge 40A included as an image in processing target image 108) and more inward than outer edge 42A of lesion 42.

[0137] Although an example in which the outer edge 110D is offset has been given here, this is merely one example, and the outer shape of the segmentation mask 110A, including the outer edge 110D, may be reduced by performing a contraction process (for example, a morphological transformation contraction process) on the segmentation mask 110A.

[0138] Recognition unit 82A fits outer edge 110A1 of segmentation mask 110A with a reduced outer shape, outer edge 110D of segmentation mask 110A with a reduced outer shape, and edge 110C1 of blank area 110C to image to be processed 108. That is, recognition unit 82A maps the pixels of outer edge 110A1 to corresponding positions in image to be processed 108, maps the pixels of edge 110C1 to corresponding positions in image to be processed 108, and maps the pixels of outer edge 110D to corresponding positions in image to be processed 108.

[0139] The positions to which the pixels of outer edge 110A1 are mapped are positions indicated by position information associated with each pixel of outer edge 110A1 of segmentation mask 110A. The positions to which the pixels of edge 110C1 are mapped are positions indicated by position information associated with each pixel of edge 110C1 of segmentation mask 110A. The positions to which the pixels of outer edge 110D are mapped are positions indicated by position information associated with each pixel of outer edge 110D of segmentation mask 110A.

[0140] In this way, when the outer edges 110A1 and 110D of the segmentation mask 110A with reduced outer dimensions are fitted to the processing target image 108, the outer edges 110A1 and 110D of the segmentation mask 110A are located more inside the lesion 42 than the outer edge 42A and edge 40A of the lesion 42.

[0141] As an example, as shown in FIG. 19 , the recognition unit 82A extracts an inner region 109C from the processing target image 108. The inner region 109C is an area inward of the outer periphery 42A and edge 40A of the lesion 42 shown in the processing target image 108 (in other words, an area in the processing target image 108 excluding the outer periphery 42A and edge 40A of the lesion 42 shown in the processing target image 108), and is an area in the processing target image 108 outward of the edge 110C1 of the blank region 110C. In the example shown in FIG. 19 , an image of a portion surrounded by the outer periphery 110A1, the outer periphery 110D, and the edge 110C1 of the blank region 110C in the processing target image 108 is extracted from the processing target image 108 as the inner region 109C. In this embodiment, the inner region 109C is an example of the “inner region” according to the present disclosure.

[0142] The recognition unit 82A calculates the sharpness 112C of the inner region 109C in the same manner as it calculated the sharpness 112A of the inner region 109A. The recognition unit 82A performs the first control process 113 using the sharpness 112C in the same manner as it performed the first control process 113 using the calculated sharpness 112A. If the sharpness 112C is equal to or greater than the threshold value TH, the recognition unit 82A inputs the inner region 109C to the feature recognition model 100, as shown in FIG. 20 as an example. As a result, each time the inner region 109C is input, the feature recognition model 100 recognizes the medical features of the lesion 42 indicated by the input inner region 109C and generates and outputs the feature recognition result 116C that is the recognition result.

[0143] Hereinafter, for convenience of explanation, when there is no need to distinguish between inner regions 109A, 109B, and 109C, they will be referred to as "inner region 109." Also, below, for convenience of explanation, when there is no need to distinguish between sharpness indices 112A, 112B, and 112C, they will be collectively referred to as "sharpness indices 112." Also, below, for convenience of explanation, when there is no need to distinguish between feature recognition results 116A, 116B, and 116C, they will be collectively referred to as "feature recognition result 116." Also, in the following, for the sake of convenience, when there is no need to distinguish between the process of offsetting the outer edge 110A1 of the segmentation mask 110A to the inside of the segmentation mask 110A, the process of enlarging the blank area 110C within the segmentation mask 110A, and the process of offsetting the outer edge 110D of the segmentation mask 110A to the inside of the segmentation mask 110A, they will be referred to as "processing to fit the outline of the segmentation mask 110A within the lesion 42 on the image to be processed 108."

[0144] 21 , the control unit 82B acquires a first area recognition result 106, a second area recognition result 110, and a feature recognition result 116 from the recognition unit 82A. The control unit 82B then outputs information based on the first area recognition result 106, information based on the second area recognition result 110, and information based on the feature recognition result 116. An example of an output destination for the information based on the first area recognition result 106, information based on the second area recognition result 110, and information based on the feature recognition result 116 is the screen 35. Other examples of output destinations include the storage 76, the storage 86, a server, a personal computer, and / or a tablet terminal.

[0145] The output timing of the information based on the first area recognition result 106, the information based on the second area recognition result 110, and the information based on the feature recognition result 116 is synchronized with the display timing of the frame 40 in the first display area 35A. For example, the information based on the first area recognition result 106, the information based on the second area recognition result 110, and the information based on the feature recognition result 116 are output in synchronization with the display of the frame 40 in the first display area 35A.

[0146] In this embodiment, the feature recognition result 116 is an example of the "processing result of the image recognition processing" according to the present disclosure. Also, in this embodiment, information based on the feature recognition result 116 is an example of the "processing result of the image recognition processing" according to the present disclosure. Also, in this embodiment, the screen 35 is an example of the "screen" according to the present disclosure.

[0147] The first display area 35A displays a frame 40, and a frame image 106A is superimposed on the frame 40 as an example of information based on the first area recognition result 106. The second display area 35B displays information based on the second area recognition result 110 and information based on the feature recognition result 116 as part of the information included in the auxiliary information 44. The second display area 35B displays a segmentation mask 110A as an example of information based on the second area recognition result 110. The second display area 35B also displays information indicating the type of lesion 42 captured in the frame 40 displayed in the first display area 35A, information indicating the form of the lesion 42, and information indicating the malignancy of the lesion 42 as an example of information based on the feature recognition result 116. The second display area 35B also displays information indicating the size of the lesion 42 captured in the frame 40 displayed in the first display area 35A as an example of information based on the second area recognition result 110. The size of the lesion 42 is calculated based on the number of pixels in the segmentation mask 110A.

[0148] The display of information based on first area recognition result 106, information based on second area recognition result 110, and information based on feature recognition result 116 is updated in synchronization with the display timing of each frame 40 included in moving image 39 displayed in first display area 35A. In other words, the display of information based on first area recognition result 106, information based on second area recognition result 110, and information based on feature recognition result 116 is updated according to the default frame rate applied to the display of frames 40.

[0149] In the following, for the sake of convenience, when there is no need to distinguish between information based on the first area recognition result 106, information based on the second area recognition result 110, and information based on the feature recognition result 116, they will be collectively referred to as "processing result information."

[0150] Next, the operation of the portion of the endoscope system 10 according to the present disclosure will be described with reference to Fig. 22. The flowchart shown in Fig. 22 is an example of the "medical support method" according to the present disclosure.

[0151] 22, first, in step ST10, the recognition unit 82A determines whether or not one frame of image data has been captured by the camera 52 inside the large intestine 28. If one frame of image data has not been captured by the camera 52 inside the large intestine 28 in step ST10, the determination is negative, and the medical support process proceeds to step ST26. If one frame of image data has been captured by the camera 52 inside the large intestine 28 in step ST10, the determination is positive, and the medical support process proceeds to step ST12.

[0152] In step ST12, the recognition unit 82A and the control unit 82B acquire a frame 40 obtained by capturing an image of the inside of the large intestine 28 (for example, the intestinal wall 32) with the camera 52 (see FIG. 8). Then, the control unit 82B displays the frame 40 in the first display area 35A (see FIG. 8). Note that, if the frame 40 is already displayed in the first display area 35A, the control unit 82B updates the frame 40 displayed in the first display area 35A. That is, by repeatedly executing the processing of step ST12, the frame 40 is displayed in the first display area 35A in a live view format. After the processing of step ST12 is executed, the medical support processing proceeds to step ST14.

[0153] In step ST14, the recognition unit 82A executes first region recognition processing 105 on the frame 40 acquired in step ST12 to recognize an image region including the lesion 42, and generates a first region recognition result 106, which is the recognition result (see FIG. 8). After the processing of step ST14 is executed, the medical support processing proceeds to step ST16.

[0154] In step ST16, the recognition unit 82A extracts a processing target image 108 surrounded by a frame image 106A included in the first region recognition result 106 acquired in step ST14 from the frame 40 targeted for the first region recognition process 105, and performs a second region recognition process 107 on the processing target image 108 (see FIG. 9). By performing the second region recognition process 107 on the processing target image 108, the recognition unit 82A generates a second region recognition result 110 including a segmentation mask 110A (see FIG. 9). The recognition unit 82A performs a processing process to fit the outline of the segmentation mask 110A within the lesion 42 on the processing target image 108 (see FIGS. 10, 14, and 18). The recognition unit 82A extracts, from the processing target image 108, an image region surrounded by the contour of the processed segmentation mask 110A as an inner region 109 (see FIGS. 11, 15, and 19). After the processing of step ST16 is executed, the medical support processing proceeds to step ST18.

[0155] In step ST18, the recognition unit 82A calculates the sharpness 112 of the inner region 109 (see FIGS. 11, 15, and 19). After the processing of step ST18 is executed, the medical support processing proceeds to step ST20.

[0156] In step ST20, the recognition unit 82A determines whether the sharpness 112 calculated in step ST18 is equal to or greater than the threshold value TH (see FIGS. 11, 15, and 19). If the sharpness 112 is less than the threshold value TH in step ST20, the determination is negative, and the medical support processing proceeds to step ST26. If the sharpness 112 is equal to or greater than the threshold value TH in step ST20, the determination is positive, and the medical support processing proceeds to step ST22.

[0157] In step ST22, the recognition unit 82A executes feature recognition processing 114 on the inner region 109 (see FIGS. 12, 16, and 20). After the processing in step ST22 is executed, the medical support processing proceeds to step ST24.

[0158] In step ST24, the control unit 82B acquires the first area recognition result 106, the second area recognition result 110, and the feature recognition result 116 from the recognition unit 82A, and generates processing result information to display on the screen 35 (see FIG. 21). After the processing of step ST24 is executed, the medical support processing proceeds to step ST26.

[0159] In step ST26, the control unit 82B determines whether a condition for terminating the medical support process is satisfied. An example of the condition for terminating the medical support process is a condition that an instruction to terminate the medical support process is given to the endoscope system 10 (for example, a condition that an instruction to terminate the medical support process is accepted by the acceptance device 64).

[0160] In step ST26, if the condition for terminating the medical support process is not satisfied, the determination is negative and the medical support process proceeds to step ST 10. In step ST26, if the condition for terminating the medical support process is satisfied, the determination is positive and the medical support process ends.

[0161] As described above, for example, if the sharpness of the processing target image 108 containing the lesion 42 is calculated, there is a concern that an excessively high sharpness will be calculated because the high-frequency components of the outer edge 42A of the lesion 42 will affect the sharpness. If an excessively high sharpness is calculated, the processing target image 108 will be determined to be an image suitable for execution of the feature recognition process 114, and the feature recognition process 114 will be executed on the processing target image 108. However, this may result in the medical features of the lesion 42 being erroneously recognized by the feature recognition process 114 on the processing target image 108.

[0162] Therefore, in this embodiment, the sharpness 112 of the inner region 109, which is an area inside the outer edge 42A of the lesion 42 shown in the frame 40, is calculated. The inner region 109 is an image region extracted from the processing target image 108 using the contour of the segmentation mask 110A obtained by performing processing to fit the contour of the segmentation mask 110A within the lesion 42 in the processing target image 108. Therefore, the sharpness 112 of the inner region 109 is calculated as a value that is less affected by the high-frequency components of the outer edge 42A of the lesion 42, preventing the sharpness 112 from becoming an excessively high value due to the high-frequency components of the outer edge 42A of the lesion 42. In this embodiment, the feature recognition process 114 is or is not performed on the inner region 109 depending on the sharpness 112. This prevents information reflecting the results of erroneous recognition of the medical features of the lesion 42 by the feature recognition process 114 due to low accuracy of the sharpness 112 from being provided to the doctor 12 or the like.

[0163] Furthermore, in this embodiment, the recognition unit 82A performs a process included in the first control process 113, in which the feature recognition process 114 is performed when the sharpness 112 is equal to or greater than the threshold value TH, and the feature recognition process 114 is not performed when the sharpness 112 is less than the threshold value TH. When the sharpness 112 is equal to or greater than the threshold value TH, the feature recognition process 114 is performed on the inner region 109, which does not include blurred or other unsharp image regions within the processing target image 108. This makes it possible to provide information to the doctor 12 or the like that reflects the results of highly accurate recognition of the medical features of the lesion 42 by the feature recognition process 114. On the other hand, when the sharpness 112 is less than the threshold value TH, the feature recognition process 114 is not performed on the inner region 109, which is affected by blurred or other unsharp image regions. This makes it possible to prevent information that reflects the results of erroneous recognition of the medical features of the lesion 42 by the feature recognition process 114 from being provided to the doctor 12 or the like. Furthermore, if the sharpness 112 is less than the threshold value TH, it is known that the inner region 109 is affected by an unsharp image region such as blur, so unnecessary execution of the feature recognition process 114 can be suppressed.

[0164] Furthermore, in this embodiment, the inner region 109 is generated based on the segmentation mask 110A obtained by performing the second region recognition process 107 on the processing target image 108. That is, the inner region 109 is an image region cut out from the processing target image 108 by the outer contour obtained by offsetting the outer edge 110A1 of the segmentation mask 110A to the inside of the segmentation mask 110A. Therefore, compared to when an image region including the outer edge 42A of the lesion 42 shown in the processing target image 108 is used as the inner region 109, it is possible to easily obtain an image region that is less affected by high-frequency components as the inner region 109 used to calculate the sharpness 112. Furthermore, compared to when an image region including the outer edge 42A of the lesion 42 shown in the processing target image 108 is used as the inner region 109, it is possible to easily obtain an image region with a high certainty of being the lesion 42 as the inner region 109.

[0165] Furthermore, in this embodiment, the resolution of the segmentation mask 110A is lower than the resolution of the frame 40, and there is no need to perform processing to increase the resolution of the segmentation mask 110A to the same resolution as the frame 40 in order to generate the inner region 109. Therefore, the processing load on the processor 82 can be reduced compared to when processing is performed to increase the resolution of the segmentation mask 110A to the same resolution as the frame 40 in order to generate the inner region 109.

[0166] Furthermore, in this embodiment, if a portion of the lesion 42 shown in the processing target image 108 is outside the processing target image 108, the image region of the entire image region of the processing target image 108 that does not include the edge 40A of the frame 40 (in other words, the image region of the entire image region of the processing target image 108 that does not include the edge 40A of the frame 40) is used as the inner region 109C. Therefore, the sharpness 112C of the inner region 109C is calculated as a value that is less susceptible to the influence of high-frequency components of the edge 40A of the frame 40, preventing the sharpness 112C from becoming an excessively high value due to the high-frequency components of the edge 40A of the frame 40. In this embodiment, the feature recognition process 114 is or is not performed on the inner region 109C depending on the sharpness 112C. This prevents information reflecting the result of erroneous recognition of the medical features of the lesion 42 by the feature recognition process 114 due to low accuracy of the sharpness 112C from being provided to the doctor 12 or the like.

[0167] Furthermore, in this embodiment, when a halation portion 117 is present in the lesion 42 captured in the processing target image 108, the image region of the entire image region of the processing target image 108 that is outside the edge of the halation portion 117 captured in the processing target image 108 (i.e., the edge of the halation portion 117 included as an image in the processing target image 108) is used as the inner region 109B. Therefore, the sharpness 112B of the inner region 109B is calculated as a value that is less susceptible to the influence of high-frequency components of the edge of the halation portion 117, thereby preventing the sharpness 112B from becoming an excessively high value due to the high-frequency components of the edge of the halation portion 117. In this embodiment, the feature recognition process 114 is or is not performed on the inner region 109B depending on the sharpness 112B. This makes it possible to prevent information reflecting the result of erroneous recognition of the medical features of the lesion 42 by the feature recognition process 114 due to low accuracy of the sharpness 112B from being provided to the doctor 12 or the like.

[0168] In the above embodiment, an example was given in which the image region of the entire image region of the processing target image 108 that is surrounded by the outer edge 110A1 of the segmentation mask 110A with a reduced outer shape is defined as the inner region 109, but the present disclosure is not limited to this. For example, instead of the inner region 109, the image region within the entire image region of the processing target image 108 that is a second frame that is a frame obtained by narrowing the first frame that surrounds the lesion 42 to the inside of the lesion 42 relative to the outer edge 42A of the lesion 42 may be applied.

[0169] For example, as shown in FIG. 23 , the image region surrounded by frame image 106B obtained by reducing frame image 106A to the inside of lesion 42 relative to the outer edge 42A of lesion 42 may be extracted by recognition unit 82A as inner region 109D from the entire image region of image 108 to be processed. In this case, sharpness 112D of inner region 109D may be calculated in a manner similar to that used to calculate sharpness 112A-112C. An example of frame image 106A is a frame image corresponding to a bounding box. While the example described above illustrates a form in which frame image 106A is reduced to the inside of lesion 42 relative to the outer edge 42A of lesion 42, this is merely an example. Frame image 106B may also be defined by offsetting the outer edge of frame image 106A to the inside of lesion 42 by a number of pixels determined according to the geometric characteristics of lesion 42 and frame image 106A.

[0170] In this way, even when inner region 109D is extracted from processing target image 108 using frame image 106B obtained by narrowing frame image 106A in processing target image 108 to the inside of lesion 42 relative to outer edge 42A of lesion 42, the same effect as in the above embodiment can be obtained. Furthermore, in the example shown in Fig. 23, execution of second region recognition process 107 is unnecessary, and therefore the processing load is reduced by the amount that execution of second region recognition process 107 is unnecessary.

[0171] In the example shown in Fig. 23, inner region 109D is an example of an "inner region" according to the present disclosure. Also, in the example shown in Fig. 23, frame image 106A is an example of a "first frame" according to the present disclosure. Also, in the example shown in Fig. 23, frame image 106B is an example of a "second frame" according to the present disclosure.

[0172] Hereinafter, for convenience of explanation, when there is no need to distinguish between sharpness levels 112A, 112B, 112C, and 112D, they will be collectively referred to as "sharpness level 112." Also, hereinafter, for convenience of explanation, when there is no need to distinguish between inner regions 109A, 109B, 109C, and 109D, they will be collectively referred to as "inner region 109."

[0173] In the above embodiment, an example was given in which the first control process 113 was executed by the recognition unit 82A, but the present disclosure is not limited to this. For example, as shown in Fig. 24, instead of the first control process 113 executed by the recognition unit 82A, a second control process 118 executed by the control unit 82B may be applied. The second control process 118 is an example of a "second process" according to the present disclosure.

[0174] 12, control unit 82B performs second control processing 118. Second control processing 118 is processing for controlling the output of processing result information (for example, processing for increasing or decreasing the output level of processing result information) in accordance with sharpness 112. As processing included in second control processing 118, control unit 82B performs processing for outputting processing result information when sharpness 112 is equal to or greater than threshold value TH, and for not outputting processing result information when sharpness 112 is less than threshold value TH.

[0175] For example, output of the processing result information is realized by displaying the processing result information on the screen 35. That is, when the sharpness 112 is equal to or greater than the threshold value TH, the processing result information is displayed on the screen 35, and when the sharpness 112 is less than the threshold value TH, the processing result information is not displayed on the screen 35. Here, the concept of "the processing result information is not displayed on the screen 35" includes the meaning that the display intensity of the processing result information is weakened compared to the display intensity of the processing result information when the sharpness 112 is equal to or greater than the threshold value TH (for example, the meaning that the processing result information is displayed with a transparency that is not visually perceptible), and the meaning that the processing result information is displayed in a visually imperceptible state by being masked.

[0176] Fig. 25 shows an example of the flow of the medical support process when the second control process 118 is executed by the control unit 82B. The flowchart shown in Fig. 25 differs from the flowchart shown in Fig. 22 in that it includes processes of steps ST100 to ST106 instead of the processes of steps ST18 to ST24.

[0177] In step ST100, the recognition unit 82A executes feature recognition processing 114 on the inner region 109 in the same manner as in the above embodiment. After the processing in step ST100 is executed, the medical support processing proceeds to step ST102.

[0178] In step ST102, the recognition unit 82A calculates the sharpness 112 of the inner region 109 in the same manner as in the above embodiment. After the processing in step ST102 is executed, the medical support processing proceeds to step ST104.

[0179] In step ST104, the control unit 82B determines whether the sharpness 112 is equal to or greater than the threshold value TH. If the sharpness 112 is less than the threshold value TH in step ST104, the determination is negative, and the medical support processing proceeds to step ST26. If the sharpness 112 is equal to or greater than the threshold value TH in step ST104, the determination is positive, and the medical support processing proceeds to step ST106.

[0180] In step ST106, the control unit 82B acquires the first area recognition result 106, the second area recognition result 110, and the feature recognition result 116 from the recognition unit 82A in the same manner as in the above embodiment, and generates processing result information to display on the screen 35. After the processing of step ST106 is executed, the medical support processing proceeds to step ST26.

[0181] 24 and 25, the feature recognition process 114 is executed on the inner region 109 regardless of the sharpness 112. This differs from the above embodiment in that the processing result information is displayed or not displayed on the screen 35 depending on the sharpness 112. By displaying or not displaying the processing result information on the screen 35 depending on the sharpness 112 in this way, it is possible to prevent the doctor 12 or the like from being provided with information that reflects the result of erroneous recognition of the medical features of the lesion 42 by the feature recognition process 114 due to low accuracy of the sharpness 112.

[0182] 24 and 25, when the sharpness 112 is equal to or greater than the threshold value TH, the feature recognition process 114 is performed on the inner region 109 of the processing target image 108, which does not include blurred or other unsharp image regions. The resulting processing result information is displayed on the screen 35. This allows the doctor 12 and other users to receive information reflecting the results of the feature recognition process 114 accurately recognizing the medical features of the lesion 42. On the other hand, when the sharpness 112 is less than the threshold value TH, the feature recognition process 114 is performed on the inner region 109, which is affected by blurred or other unsharp image regions. Therefore, the doctor 12 and other users can be prevented from receiving information reflecting the results of the feature recognition process 114 incorrectly recognizing the medical features of the lesion 42. When the sharpness 112 is less than the threshold value TH, it is known that the inner region 109 is affected by blurred or other unsharp image regions. Therefore, unnecessary output of the processing result information (e.g., display on the screen 35) can be prevented.

[0183] In the above embodiment, an example was given in which the first control process 113 is executed in accordance with the sharpness 112 obtained from a single processing target image 108, but this is merely one example. For example, the first control process 113 may be executed in accordance with a plurality of sharpnesses 112 obtained from a plurality of processing target images 108 corresponding to a plurality of frames 40 in a time series. Here, a specific example of this embodiment will be described.

[0184] The feature recognition process 114 included in the first control process 113 is executed for each of a plurality of inner regions 109 obtained from a plurality of processing target images 108 (i.e., three processing target images 108) corresponding to a plurality of frames 40 (i.e., three frames 40) obtained after t seconds, t-1 seconds, and t-2 seconds. As a result, a plurality of malignancy levels 120 are obtained, as shown in FIG. 26 as an example. The malignancy levels 120 are information included in the feature recognition result 116.

[0185] The latest feature recognition result 116 (i.e., the feature recognition result 116 obtained by executing the feature recognition process 114 on the processing target image 108 corresponding to the frame 40 after t seconds) among the multiple feature recognition results 116 (i.e., three feature recognition results 116) obtained after t seconds, t-1 seconds, and t-2 seconds is assigned a weight determined based on the multiple sharpness indices 112 corresponding to the multiple processing target images 108. For example, as shown in Fig. 26, a weight to be assigned to the malignancy level 120 after t seconds is determined based on the three sharpness indices 112 obtained after t seconds, t-1 seconds, and t-2 seconds and the three malignancy levels 120 included in the three feature recognition results 116, and the weight is assigned to the malignancy level 120 after t seconds, thereby updating the malignancy level 120 after t seconds.

[0186] For example, if three sharpness indices 112 and three malignancy levels 120 are obtained after t seconds, t-1 seconds, and t-2 seconds as shown in Table 1 below, the malignancy level 120 after t seconds is calculated as "0.756" using the following formula (1).

[0187] [Table 1]

[0188]

number

[0189] That is, when feature recognition process 114 is performed on inner region 109 obtained from processing target image 108 corresponding to frame 40 obtained after t seconds, the malignancy level 120 included in feature recognition result 116 is "0.7", but by assigning a weight of "1.08" to "0.7", the malignancy level 120 after t seconds is updated from "0.7" to "0.756". Note that malignancy level 120 calculated by formula (1) may be classified into any one of grades 1 to 3 and output to screen 35, etc.

[0190] By doing so, it is possible to increase the contribution of the inner region 109 with a high sharpness 112 among the three sharpnesses 112 after t seconds, t-1 seconds, and t-2 seconds to the feature recognition result 116 (here, as an example, the malignancy level 120), and to decrease the contribution of the inner region 109 with a low sharpness 112 among the three sharpnesses 112 after t seconds, t-1 seconds, and t-2 seconds to the feature recognition result 116 (here, as an example, the reliability of the malignancy level 120 included in the feature recognition result 116 after t seconds).

[0191] Here, we have given an example in which a weight is assigned to the malignancy 120 contained in the feature recognition result 116 after t seconds, but this is merely one example, and a weight may be assigned to the malignancy 120 contained in each of multiple feature recognition results 116.

[0192] Furthermore, while the above embodiment presents an example in which the first control process 113 is performed in accordance with the plurality of sharpness indices 112 obtained from the plurality of processing target images 108 corresponding to the plurality of frames 40 in a time series, this is merely an example. For example, the second control process 118 may be performed in accordance with the plurality of sharpness indices 112 obtained from the plurality of processing target images 108 corresponding to the plurality of frames 40 in a time series. In this case, the reliability of the feature recognition result 116 corresponds to the level of the sharpness 112 (i.e., the higher the sharpness 112, the higher the reliability of the feature recognition result 116). Therefore, information based on the feature recognition result 116 may or may not be displayed depending on the reliability of the feature recognition result 116. For example, if the reliability of the feature recognition result 116 is above a certain level, the information based on the feature recognition result 116 may be displayed, and if the reliability of the feature recognition result 116 is below the certain level, the information based on the feature recognition result 116 may not be displayed.

[0193] In the above embodiment, the threshold value TH is exemplified, but instead of the threshold value TH, a first threshold value and a second threshold value that is a value less than the first threshold value may be used. An example of the first threshold value is the same as the threshold value TH. When the first threshold value and the second threshold value that is a value less than the first threshold value are used, it is determined whether the sharpness 112 is equal to or greater than the first threshold value, and if the sharpness 112 is equal to or greater than the first threshold value, the processing from step ST22 onwards shown in FIG. 22 is executed, or the processing from step ST106 onwards shown in FIG. 25 is executed.

[0194] On the other hand, if the sharpness 112 is less than the first threshold and also less than the second threshold, the determination in step ST20 shown in Fig. 22 is negative and the process in step ST26 is executed, or the determination in step ST104 shown in Fig. 25 is negative and the process in step ST26 is executed. Also, if the sharpness 112 is equal to or greater than the second threshold and less than the first threshold, the doctor 12 or the like may decide whether to cause the recognition unit 82A to execute the feature recognition process 114 or whether to cause the control unit 82B to output processing result information.

[0195] Furthermore, when sharpness 112 is equal to or greater than the second threshold and less than the first threshold, the recognition unit 82A may be caused to execute feature recognition processing 114 or the control unit 82B may be caused to output processing result information, depending on a predetermined condition. For example, when sharpness 112 is equal to or greater than the second threshold and less than the first threshold, the recognition unit 82A may be caused to execute feature recognition processing 114, provided that a part shown in frame 40 displayed in first display area 35A (i.e., a part included as an image in frame 40) is an important part. Furthermore, when sharpness 112 is equal to or greater than the second threshold and less than the first threshold, the control unit 82B may be caused to output processing result information, provided that a part shown in frame 40 displayed in first display area 35A is an important part.

[0196] In the above embodiment, a calculation method incorporating a process of filtering the inner region 109 with a Laplacian filter has been exemplified as a calculation method of the sharpness 112, but this is merely one example, and a calculation method incorporating a process of filtering the inner region 109 with a Sobel filter may also be used. Alternatively, a calculation method incorporating a process of converting the inner region 109 into a frequency domain by performing a fast Fourier transform on the inner region 109, emphasizing high-frequency components in the frequency domain by applying a high-pass filter to the frequency domain, and converting the frequency domain into a spatial domain by performing an inverse fast Fourier transform on the frequency domain with the emphasized high-frequency components may also be used.

[0197] In the above embodiment, the sharpness 112 is calculated by averaging the sum of the absolute values ​​of the pixel values ​​of each pixel in the inner region 109 to which a Laplacian filter has been applied. However, this is merely an example. Instead of averaging the sum of the absolute values ​​of the pixel values ​​of each pixel in the inner region 109 to which a Laplacian filter has been applied, a statistical value other than the average value may be calculated as the sharpness 112. Examples of statistical values ​​other than the average value include variance, standard deviation, median, maximum value, and minimum value. Furthermore, the statistical value may be a combination of two or more of the average value, variance, standard deviation, median, maximum value, and minimum value.

[0198] In the above embodiment, an endoscopic image is exemplified as frame 40, but this is merely an example, and the present disclosure will be valid even if a medical image such as an MRI image, a CT image, or an X-ray image is applied instead of frame 40.

[0199] In the above embodiment, first area recognition processing 105 using bounding box AI has been exemplified, but this is merely one example, and for example, object recognition processing using segmentation AI may be performed instead of first area recognition processing 105 using bounding box AI. Also, instead of AI-based recognition processing, non-AI-based recognition processing (e.g., template matching) may be performed, or recognition processing that combines non-AI and AI-based methods may be performed.

[0200] In the above embodiment, an example in which the medical support processing is performed by the computer 78 has been described, but the present disclosure is not limited to this, and at least a part of the processing included in the medical support processing may be performed by a device provided outside the computer 78. An example of this case will be described below with reference to FIG.

[0201] 27 is a conceptual diagram showing an example of the configuration of an endoscope system 122. The endoscope system 122 is an example of the "endoscope system" according to the present disclosure. The endoscope system 122 differs from the endoscope system 10 described in the above embodiment in that it includes an external device 124.

[0202] External device 124 is communicatively coupled to computer 78 via a network 126 (eg, a WAN and / or LAN, etc.).

[0203] An example of the external device 124 is at least one server that directly or indirectly transmits and receives data to and from the computer 78 via the network 126. The external device 124 receives a processing execution instruction provided from the processor 82 of the computer 78 via the network 126. The external device 124 then executes processing in accordance with the received processing execution instruction and transmits the processing result to the computer 78 via the network 126. In the computer 78, the processor 82 receives the processing result transmitted from the external device 124 via the network 126 and executes processing using the received processing result.

[0204] An example of the processing execution instruction is an instruction to cause the external device 124 to execute at least a part of the medical support processing. A first example of at least a part of the medical support processing (i.e., a processing to be executed by the external device 124) is a first area recognition processing 105. In this case, the external device 124 executes the first area recognition processing 105 in accordance with the processing execution instruction provided from the processor 82 via the network 126, and transmits a first area recognition result 106 to the computer 78 via the network 126. In the computer 78, the processor 82 receives the first area recognition result 106 and executes processing similar to that in the above embodiment using the received first area recognition result 106.

[0205] A second example of at least a part of the medical support processing (i.e., processing to be executed by the external device 124) is second area recognition processing 107. In this case, the external device 124 executes the second area recognition processing 107 in accordance with a processing execution instruction provided from the processor 82 via the network 126, and transmits a second area recognition result 110 to the computer 78 via the network 126. In the computer 78, the processor 82 receives the second area recognition result 110 and executes processing similar to that in the above embodiment using the received second area recognition result 110.

[0206] A third example of at least a part of the medical support processing (i.e., processing to be executed by the external device 124) is processing by the first control processing 113. In this case, the external device 124 executes the first control processing 113 in accordance with a processing execution instruction provided from the processor 82 via the network 126, and transmits the processing results (e.g., information indicating whether the feature recognition processing 114 has been executed and / or the feature recognition result 116 obtained when the feature recognition processing 114 has been executed) to the computer 78 via the network 126. In the computer 78, the processor 82 receives the processing results and executes processing similar to that of the above embodiment (e.g., display using the display device 18) using the received processing results.

[0207] A fourth example of at least a part of the medical support processing (i.e., processing to be executed by the external device 124) includes extraction of the processing target image 108 from the frame 40, extraction of the inner region 109 from the processing target image 108, and / or calculation of the sharpness 112. In this case, the external device 124 extracts the processing target image 108 from the frame 40, extraction of the inner region 109 from the processing target image 108, and / or calculation of the sharpness 112 in accordance with a processing execution instruction provided from the processor 82 via the network 126, and transmits the processing results (e.g., the processing target image 108, the inner region 109, and / or the sharpness 112) to the computer 78 via the network 126. In the computer 78, the processor 82 receives the processing results and executes processing similar to that of the above embodiment using the received processing results.

[0208] The external device 124 may be realized by cloud computing. Cloud computing is merely an example, and the external device 124 may be realized by network computing such as fog computing, edge computing, or grid computing.

[0209] In the above embodiment, an example has been described in which the medical support program 98 is stored in the storage 86, but the present disclosure is not limited to this. For example, the medical support program 98 may be stored in a portable, computer-readable, non-transitory storage medium such as an SSD or a USB memory. The medical support program 98 stored in the non-transitory storage medium is installed in the computer 78 of the endoscope system 10. The processor 82 executes medical support processing in accordance with the medical support program 98.

[0210] Alternatively, the medical support program 98 may be stored in a storage device such as another computer or server connected to the endoscopic system 10 via a network, and the medical support program 98 may be downloaded and installed on the computer 78 in response to a request from the endoscopic system 10.

[0211] It is not necessary to store the entire medical support program 98 in a storage device such as another computer or server device connected to the endoscopic system 10, or to store the entire medical support program 98 in the storage 86; only a portion of the medical support program 98 may be stored.

[0212] The hardware resources that execute the medical support processing can be various processors, as listed below. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource that executes medical support processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs, PLDs, or ASICs, which are processors with circuit configurations specifically designed to execute specific processes. Each processor has built-in or connected memory, and executes medical support processing by using the memory.

[0213] The hardware resource that executes the medical support processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the medical support processing may be a single processor.

[0214] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes medical support processing. Second, there is a system that uses a processor that realizes the functions of the entire system, including multiple hardware resources that execute medical support processing, on a single IC chip, as typified by SoCs. In this way, medical support processing is realized using one or more of the above-mentioned various processors as hardware resources.

[0215] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The above medical support process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the process.

[0216] The above-described description and illustrations are a detailed explanation of the parts related to the present disclosure and are merely an example of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or elements may be replaced with other parts from the above-described description and illustrations, as long as they do not deviate from the gist of the present disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the parts related to the present disclosure, the above-described description and illustrations omit explanations of common general technical knowledge that do not require particular explanation to enable the implementation of the present disclosure.

[0217] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0218] 10,122 Endoscopy Systems 12 Doctors 14. Nurse 16 Endoscope 18 Display device 20 Control device 22 Light source device 24 Medical support equipment 26 Subject 28 Large intestine 30 light 32 Intestinal wall 34 Wagon 35 screens 35A 1st display area 35B 2nd display area 39 Video 40 frames 40A Edge 42 Lesions 42A,110A1 outer edge 44 Supplementary Information 46 Control section 48 Insertion section 50 Tip 50A tip surface 52 Camera 52A image sensor 52B Optical system 54 Lighting equipment 54A, 54B Lighting window 56 Opening for treatment instruments 58 Treatment tools 60 Treatment tool insertion port 62 Universal Code 64 Reception device 66,78 Computer 68,88 bus 70,80 External I / F 72,82 processor 74,84 memory 76,86 storage 82A recognition part 82B Control section 90 Bounding Box 92 Featured Images 92A sharp area 92B Unsharp area 94,114 Feature Recognition Processing 96,117 halation areas 96A,110D Edge 98 Medical Assistance Program 100 Feature Recognition Model 102 First Area Recognition Model 104 Second Domain Recognition Model 105 First area recognition processing 106 First area recognition result 106A Frame Image 107 Second Area Recognition Processing 108 Processing target image 109,109A,109B,109C,109D Inner area 110 Second area recognition results 110A Segmentation Mask 110C blank area 110C1 Edge 112,112A,112B,112C,112D Sharpness 113 First control process 116, 116A, 116B, 116C Feature recognition results 118 Second control process 120 malignant grade 124 External device 126 Network TH threshold

Claims

1. a processor; The processor: A medical image is obtained by capturing an image of a region including the characteristic region; performing a first process according to the sharpness of an inner region that is a region inside an outer edge of a feature region included in the medical image, or performing a second process according to the sharpness; the first processing is image recognition processing that can be executed on the medical image, and is processing for controlling image recognition processing that recognizes features of the feature region; The second processing is processing for controlling output of information based on the processing result of the image recognition processing. Medical support equipment.

2. The first processing includes a processing of executing the image recognition processing when the sharpness is equal to or greater than a first threshold, and not executing the image recognition processing when the sharpness is less than a second threshold that is equal to or less than the first threshold. The medical support device according to claim 1 .

3. The second process includes a process of outputting the information when the sharpness is equal to or greater than a first threshold, and not outputting the information when the sharpness is less than a second threshold that is equal to or less than the first threshold. The medical support device according to claim 1 .

4. a region recognition process for recognizing the characteristic region is performed on the medical image; The inner region is a region based on a segmentation mask obtained by executing the region recognition process. The medical support device according to claim 1 .

5. The resolution of the segmentation mask is lower than the resolution of the medical image. The medical support device according to claim 4.

6. The inner region is a region within a second frame, which is a frame formed by narrowing a first frame that is a frame surrounding the feature region and is located inside the feature region more than the outer edge. The medical support device according to claim 1 .

7. a region recognition process for recognizing the characteristic region is performed on the medical image; In the area recognition process, a bounding box is used, The first frame corresponds to the bounding box. The medical support device according to claim 6.

8. When a part of the characteristic region included in the medical image is outside the medical image, the inner region is a region outside the edge of the medical image. The medical support device according to claim 1 .

9. When a halation portion exists in the feature region included in the medical image, the inner region is a region outside the edge of the halation portion. The medical support device according to claim 1 .

10. The first processing and / or the second processing are performed according to a plurality of sharpness indices obtained from a plurality of the medical images in time series. The medical support device according to claim 1 .

11. At least one of the processing results obtained by executing the image recognition processing on the medical images is assigned a weight determined based on the sharpness indices. The medical support device according to claim 1 .

12. Outputting the information includes displaying the information on a screen. The medical support device according to claim 1 .

13. The feature is a medical feature of the feature region. The medical support device according to claim 1 .

14. The characteristic region is a lesion. The medical support device according to any one of claims 1 to 13.

15. The medical image is an endoscopic image. The medical support device according to claim 1 .

16. A medical support device according to any one of claims 1 to 15; an endoscope equipped with an image sensor that captures an image of the region; Endoscopy system.

17. acquiring a medical image obtained by imaging a site including the characteristic region; performing a first process according to a sharpness of an inner region that is a region inside an outer edge of a feature region included in the medical image, or performing a second process according to the sharpness; the first processing is image recognition processing that can be executed on the medical image, and is processing for controlling image recognition processing that recognizes features of the feature region; The second processing is processing for controlling output of information based on the processing result of the image recognition processing. Medical support methods.

18. A program for causing a computer to execute medical support processing, The medical support process includes: acquiring a medical image obtained by imaging a site including the characteristic region; performing a first process according to a sharpness of an inner region that is a region inside an outer edge of a feature region included in the medical image, or performing a second process according to the sharpness; the first processing is image recognition processing that can be executed on the medical image, and is processing for controlling image recognition processing that recognizes features of the feature region; The second processing is processing for controlling output of information based on the processing result of the image recognition processing. program.

Images