Medical assistance device, endoscope, medical assistance method, and program

Abstract

This medical assistance device comprises a processor. The processor uses a medical image to recognize a to-be-observed region shown in the medical image, measures the size corresponding to the characteristics of the to-be-observed region on the basis of the medical image, and outputs the size.

Description

Medical support device, endoscope, medical support method, and program

[0001] The technology of the present disclosure relates to a medical support device, an endoscope, a medical support method, and a program.

[0002] Japanese Patent Laid-Open Publication No. 2008-061704 discloses an image display device that displays a group of images captured inside a subject in chronological order. The image display device disclosed in Japanese Patent Laid-Open Publication No. 2008-061704 includes an image detection unit, a mark display unit, and a display control unit.

[0003] The image detection means detects lesion images included in the image group. The mark display means displays lesion marks indicating the time positions of lesion images on a time bar that indicates the overall time position of the image group. The display control means calculates the number of images per unit pixel of the time bar based on the number of pixels in the time axis direction that form the time bar and the number of images in the image group. The display control means also counts the number of lesion images for each consecutive image group in the image group that contains consecutive images of the same number per unit pixel. The display control means then controls the display of a lesion mark having a display mode corresponding to the result of counting the number of lesion images for each consecutive image group that includes one or more lesion images.

[0004] International Publication No. 2020 / 165978 discloses an image recording device having an acquisition unit that acquires time-series images of an endoscopic examination, a lesion appearance identification unit that identifies the appearance of a lesion related to the acquired time-series images, and a recording unit that starts recording the time-series images from the point in time when the appearance of a lesion is identified by the lesion appearance identification unit.

[0005] In the image recording device described in International Publication No. 2020 / 165978, the lesion appearance identification unit includes a lesion detection unit that detects a lesion based on the acquired time-series images. The lesion appearance identification unit further includes a lesion information calculation unit that calculates information related to the lesion based on the lesion detected by the lesion detection unit. The lesion information calculation unit calculates size information related to the lesion detected by the lesion detection unit.

[0006] One embodiment of the technology disclosed herein provides a medical support device, an endoscope, a medical support method, and a program that enable a user or the like to accurately grasp the size of an observation area shown in a medical image.

[0007] A first aspect of the technology of the present disclosure is a medical support device that includes a processor, and that uses the processor and a medical image to recognize an observation area that appears in the medical image, measures a size according to the characteristics of the observation area based on the medical image, and outputs the size.

[0008] A second aspect of the technology of the present disclosure is a medical support device according to the first aspect, in which the characteristics include the shape of the observation area, the type of the observation area, the clarity of the contour of the observation area, and / or the overlap between the observation area and the surrounding area.

[0009] A third aspect of the technique of the present disclosure is the medical support device according to the first or second aspect, in which the processor recognizes the characteristics based on a medical image.

[0010] A fourth aspect of the technology of the present disclosure is a medical support device according to any one of the first to third aspects, in which the size is the long side, short side, radius, and / or diameter of the observation area.

[0011] A fifth aspect of the technology of the present disclosure is a medical support device according to any one of the first to fourth aspects, in which the observation area is recognized using an AI method and the size is measured based on a probability map obtained from the AI.

[0012] A sixth aspect of the technique of the present disclosure is the medical support device according to the fifth aspect, in which the size is measured based on a closed region obtained by dividing the probability map according to a threshold value.

[0013] A seventh aspect of the technology of the present disclosure is a medical support device according to the fifth or sixth aspect, in which the size is measured based on a plurality of partitioned regions obtained by partitioning the probability map according to a plurality of thresholds.

[0014] An eighth aspect of the technique of the present disclosure is the medical support device according to the seventh aspect, in which the size has a range, and the width is specified based on a plurality of partitioned regions.

[0015] A ninth aspect of the technology of the present disclosure is a medical support device according to the eighth aspect, in which the lower limit of the width is measured based on a first divided area that is the narrowest of the multiple divided areas, and the upper limit of the width is measured based on a second divided area that is outside the first divided area of ​​the multiple divided areas.

[0016] A tenth aspect of the technology of the present disclosure is a medical support device according to any one of the first to ninth aspects, in which a processor measures a plurality of first sizes of an observation area based on a medical image, and the size is a representative value of the plurality of first sizes.

[0017] An eleventh aspect of the technique of the present disclosure is the medical support device according to the tenth aspect, wherein the representative value includes a maximum value, a minimum value, an average value, a median value, and / or a variance value.

[0018] A twelfth aspect of the technology of the present disclosure is a medical support device according to any one of the first to eleventh aspects, in which the characteristics include overlap between the observation area and the surrounding area, and the size is the size of the observation area including the overlap and / or the size of the observation area excluding the overlap.

[0019] A thirteenth aspect of the technology of the present disclosure is a medical support device according to any one of the first to twelfth aspects, in which the output of the size is achieved by displaying the size on a screen.

[0020] A fourteenth aspect of the technology of the present disclosure is a medical support device according to any one of the first to thirteenth aspects, in which the medical image is an endoscopic image obtained by capturing an image using an endoscope.

[0021] A fifteenth aspect of the technique of the present disclosure is the medical support device according to any one of the first to fourteenth aspects, in which the observation target region is a lesion.

[0022] A sixteenth aspect of the technology of the present disclosure is an endoscope comprising a medical support device according to any one of the first to fifteenth aspects and a module that is inserted into a body including an observation target area and acquires medical images by capturing images of the observation target area.

[0023] A seventeenth aspect of the technology of the present disclosure is a medical support method that includes using a medical image to recognize an observation area that appears in the medical image, measuring a size of the observation area based on the medical image in accordance with the characteristics of the observation area, and outputting the size.

[0024] An eighteenth aspect of the technology of the present disclosure is a program for causing a computer to execute medical support processing, the medical support processing including using a medical image to recognize an observation target area shown in the medical image, measuring a size according to the characteristics of the observation target area based on the medical image, and outputting the size.

[0025] 1 is a conceptual diagram showing an example of an aspect in which an endoscopic system is used. FIG. 1 is a conceptual diagram showing an example of the overall configuration of an endoscope. FIG. 2 is a block diagram showing an example of the hardware configuration of an electrical system of an endoscope. FIG. 3 is a block diagram showing an example of main functions of a processor included in an endoscope according to a first embodiment, and an example of information according to the first embodiment stored in an NVM. FIG. 4 is a conceptual diagram showing an example of processing content of a recognition unit and a control unit according to the first embodiment. FIG. 5 is a conceptual diagram showing an example of processing content of a measurement unit when a minimum size is measured. FIG. 6 is a conceptual diagram showing an example of processing content of a measurement unit when a maximum size is measured. FIG. 7 is a conceptual diagram showing an example of an aspect in which an endoscopic image is displayed in a first display area, and a size is displayed in a map in a second display area. FIG. 8 is a flowchart showing an example of the flow of medical support processing. FIG. 9 is a conceptual diagram showing an example of an aspect in which a size is displayed in an endoscopic image. FIG. 10 is a conceptual diagram showing an example of an aspect in which sizes in multiple directions are displayed in a probability map. FIG. 11 is a block diagram showing an example of main functions of a processor included in an endoscope according to a second embodiment, and an example of information according to the second embodiment stored in an NVM. FIG. 12 is a conceptual diagram showing an example of processing content of a recognition unit and a control unit according to the second embodiment. FIG. 13 is a conceptual diagram showing an example of processing content of a generation unit. FIG. 14 is a conceptual diagram showing an example of processing content of a recognition unit according to the second embodiment. FIG. 10 is a conceptual diagram showing an example of a mode in which the control unit according to the second embodiment displays the apparent size in the second display area. FIG. 11 is a conceptual diagram showing an example of a mode in which the control unit according to the second embodiment displays a predicted size in the second display area. FIG. 12 is a conceptual diagram showing an example of a mode in which the control unit displays the apparent size of a lesion in the second display area when the lesion shown in an endoscopic image is pedunculated. FIG. 13 is a conceptual diagram showing an example of an output destination of the size.

[0026] Hereinafter, exemplary embodiments of a medical support device, an endoscope, a medical support method, and a program according to the techniques of the present disclosure will be described with reference to the accompanying drawings.

[0027] First, the terms used in the following description will be explained.

[0028] CPU is an abbreviation for "Central Processing Unit." GPU is an abbreviation for "Graphics Processing Unit." RAM is an abbreviation for "Random Access Memory." NVM is an abbreviation for "Non-volatile memory." EEPROM is an abbreviation for "Electrically Erasable Programmable Read-Only Memory." ASIC is an abbreviation for "Application Specific Integrated Circuit." PLD is an abbreviation for "Programmable Logic Device." FPGA is an abbreviation for "Field-Programmable Gate Array." SoC is an abbreviation for "System-on-a-chip." SSD is an abbreviation for "Solid State Drive." USB is an abbreviation for "Universal Serial Bus." HDD is an abbreviation for "Hard Disk Drive." EL is an abbreviation for "Electro-Luminescence". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". AI is an abbreviation for "Artificial Intelligence". BLI is an abbreviation for "Blue Light Imaging". LCI is an abbreviation for "Linked Color Imaging". I / F is an abbreviation for "Interface". SSL is an abbreviation for "Sessile Serrated Lesion". GANs is an abbreviation for "Generative Adversarial Networks". VAE is an abbreviation for "Variational Autoencoder".

[0029] First Embodiment As shown in Fig. 1 as an example, an endoscope system 10 includes an endoscope 12 and a display device 14. The endoscope 12 is used by a doctor 16 in an endoscopic examination. The endoscopic examination is assisted by staff such as a nurse 17. In this first embodiment, the endoscope 12 is an example of an "endoscope" according to the technology of the present disclosure.

[0030] The endoscope 12 is communicatively connected to a communication device (not shown), and information obtained by the endoscope 12 is transmitted to the communication device. An example of the communication device is a server and / or a client terminal (e.g., a personal computer and / or a tablet terminal) that manages various information such as electronic medical records. The communication device receives the information transmitted from the endoscope 12 and executes processing using the received information (e.g., processing to store the information in an electronic medical record, etc.).

[0031] The endoscope 12 includes an endoscope body 18. The endoscope 12 is a device for performing medical examinations on a large intestine 22 contained in the body of a subject 20 (e.g., a patient) using the endoscope body 18. In this first embodiment, the large intestine 22 is an object to be observed by a doctor 16.

[0032] The endoscope body 18 is inserted into the large intestine 22 of the subject 20. The endoscope 12 causes the endoscope body 18 inserted into the large intestine 22 of the subject 20 to capture images of the inside of the large intestine 22 inside the body of the subject 20, and also performs various medical procedures on the large intestine 22 as necessary.

[0033] The endoscope 12 acquires and outputs images showing the state inside the body by imaging the inside of the large intestine 22 of the subject 20. In the first embodiment, the endoscope 12 is an endoscope having an optical imaging function that irradiates the inside of the large intestine 22 with light 26 and captures an image of the reflected light obtained by reflection on the intestinal wall 24 of the large intestine 22.

[0034] Although an endoscopic examination of the large intestine 22 is illustrated here, this is merely one example, and the technology disclosed herein can also be applied to endoscopic examination of hollow organs such as the esophagus, stomach, duodenum, or trachea.

[0035] The endoscope 12 is equipped with a control device 28, a light source device 30, and a medical support device 32. The control device 28, the light source device 30, and the medical support device 32 are installed on a wagon 34. The wagon 34 has a plurality of stands arranged vertically, and the medical support device 32, the control device 28, and the light source device 30 are installed from the lower stand to the upper stand. In addition, the display device 14 is installed on the top stand of the wagon 34.

[0036] The control device 28 controls the entire endoscope 12. The medical support device 32, under the control of the control device 28, performs various image processing on the images obtained by the endoscope body 18 capturing images of the intestinal wall 24.

[0037] The display device 14 displays various information including images. Examples of the display device 14 include a liquid crystal display and an EL display. Alternatively, a tablet terminal with a display may be used instead of the display device 14 or together with the display device 14.

[0038] A screen 35 is displayed on the display device 14. In the first embodiment, the screen 35 is an example of a "screen" according to the technology of the present disclosure. The screen 35 includes a plurality of display areas. The plurality of display areas are arranged side by side within the screen 35. In the example shown in FIG. 1 , a first display area 36 and a second display area 38 are shown as examples of the plurality of display areas. The size of the first display area 36 is larger than the size of the second display area 38. The first display area 36 is used as a main display area, and the second display area 38 is used as a sub-display area.

[0039] An endoscopic image 40 is displayed in the first display area 36. The endoscopic image 40 is an image acquired by imaging the intestinal wall 24 inside the large intestine 22 of the subject 20 using the endoscope main body 18. In the example shown in Fig. 1, an image showing the intestinal wall 24 is shown as an example of the endoscopic image 40. In this first embodiment, the endoscopic image 40 is an example of a "medical image" and an "endoscopic image" according to the technology of the present disclosure.

[0040] The intestinal wall 24 shown in the endoscopic image 40 includes a lesion 42 (for example, one lesion 42 in the example shown in FIG. 1 ) as a region of interest (i.e., an observation target region) that is gazed upon by the physician 16, and the physician 16 can visually recognize the state of the intestinal wall 24, including the lesion 42, through the endoscopic image 40. In this first embodiment, the lesion 42 is an example of an "observation target region" and a "lesion" according to the technology of the present disclosure.

[0041] There are various types of lesions 42, and examples of the types of lesions 42 include neoplastic polyps and non-neoplastic polyps. Examples of the types of neoplastic polyps include adenomatous polyps (e.g., SSL). Examples of the types of non-neoplastic polyps include hamartomatous polyps, hyperplastic polyps, and inflammatory polyps. Note that the types exemplified here are types that are anticipated in advance as types of lesions 42 when an endoscopic examination is performed on the large intestine 22, and the types of lesions will differ depending on the organ that is subjected to the endoscopic examination.

[0042] In this first embodiment, for the sake of convenience of explanation, an example is given in which one lesion 42 is shown in the endoscopic image 40, but the technology of the present disclosure is not limited to this, and the technology of the present disclosure is applicable even when multiple lesions 42 are shown in the endoscopic image 40.

[0043] In this first embodiment, a lesion 42 is illustrated, but this is merely one example, and the area of ​​interest (i.e., the area to be observed) that is gazed upon by the doctor 16 may be an organ (e.g., the duodenal papilla), a marked area, an artificial treatment device (e.g., an artificial clip), or a treated area (e.g., an area where traces remain after removal of a polyp, etc.), etc.

[0044] A moving image is displayed in the first display area 36. The endoscopic image 40 displayed in the first display area 36 is one frame included in a moving image that includes multiple frames in chronological order. In other words, the first display area 36 displays multiple frames of the endoscopic image 40 at a default frame rate (e.g., 30 frames / second or 60 frames / second).

[0045] An example of a moving image displayed in the first display area 36 is a moving image in a live view format. The live view format is merely one example, and a moving image that is temporarily stored in a memory or the like and then displayed, such as a moving image in a post-view format, may also be used. Furthermore, each frame included in a moving image for recording that is stored in a memory or the like may be reproduced and displayed in the first display area 36 as the endoscopic image 40.

[0046] Within the screen 35, the second display area 38 is adjacent to the first display area 36 and is displayed in the lower right corner of the screen 35 when viewed from the front. The display position of the second display area 38 may be anywhere within the screen 35 of the display device 14, but it is preferable that it be displayed in a position where it can be compared with the endoscopic image 40.

[0047] A probability map 45 including a segmentation image 44 is displayed in the second display area 38. The segmentation image 44 is an image area that identifies the position within the endoscopic image 40 of a lesion 42 that has been recognized by performing object recognition processing using AI segmentation on the endoscopic image 40 (i.e., an image displayed in a display manner that enables identification of the position within the endoscopic image 40 where the lesion 42 is most likely to exist).

[0048] The segmentation image 44 displayed in the second display area 38 is an image corresponding to the endoscopic image 40 and is referenced by the physician 16 to identify the location of the lesion 42 within the endoscopic image 40 .

[0049] Although the segmentation image 44 is shown here as an example, if a lesion 42 is recognized by performing bounding box-based object recognition processing using AI on the endoscopic image 40, a bounding box is displayed instead of the segmentation image 44. Alternatively, the segmentation image 44 and the bounding box may be used together. The segmentation image 44 and the bounding box are merely examples, and any image may be used as long as it allows the position of the lesion 42 within the endoscopic image 40 to be identified.

[0050] 2, the endoscope body 18 includes an operating section 46 and an insertion section 48. The insertion section 48 is partially curved by operating the operating section 46. The insertion section 48 is inserted into the large intestine 22 (see FIG. 1) while curving in accordance with the shape of the large intestine 22 (see FIG. 1) in accordance with the operation of the operating section 46 by the physician 16 (see FIG. 1).

[0051] The distal end portion 50 of the insertion section 48 is provided with a camera 52, an illumination device 54, and a treatment tool opening 56. The camera 52 and the illumination device 54 are provided on the distal end surface 50A of the distal end portion 50. Note that, although an example in which the camera 52 and the illumination device 54 are provided on the distal end surface 50A of the distal end portion 50 is given here, this is merely one example, and the camera 52 and the illumination device 54 may be provided on the side surface of the distal end portion 50, so that the endoscope 12 is configured as a side-viewing endoscope.

[0052] The camera 52 is a device that captures an image of the inside of the subject 20 (for example, the inside of the large intestine 22) to obtain an endoscopic image 40 as a medical image. An example of the camera 52 is a CMOS camera. However, this is merely an example, and other types of cameras such as a CCD camera may also be used. The camera 52 is an example of a "module" according to the technology of the present disclosure.

[0053] The illumination device 54 has illumination windows 54A and 54B. The illumination device 54 emits light 26 (see FIG. 1 ) through the illumination windows 54A and 54B. Examples of the light 26 emitted from the illumination device 54 include visible light (e.g., white light) and invisible light (e.g., near-infrared light). The illumination device 54 also emits special light through the illumination windows 54A and 54B. Examples of the special light include light for BLI and / or light for LCI. The camera 52 captures images of the inside of the large intestine 22 by an optical method while the light 26 is being emitted from the illumination device 54 inside the large intestine 22.

[0054] The treatment tool opening 56 is an opening for allowing a treatment tool 58 to protrude from the distal end portion 50. The treatment tool opening 56 is also used as a suction port for sucking blood, internal waste, etc., and as a delivery port for delivering fluid.

[0055] A treatment tool insertion port 60 is formed in the operation section 46, and the treatment tool 58 is inserted into the insertion section 48 from the treatment tool insertion port 60. The treatment tool 58 passes through the insertion section 48 and protrudes to the outside from the treatment tool opening 56. In the example shown in Fig. 2, a puncture needle is shown as the treatment tool 58 protruding from the treatment tool opening 56. Here, a puncture needle is shown as the treatment tool 58, but this is merely one example, and the treatment tool 58 may also be a grasping forceps, a papillotomy knife, a snare, a catheter, a guidewire, a cannula, and / or a puncture needle with a guide sheath, etc.

[0056] The endoscope main body 18 is connected to a control device 28 and a light source device 30 via a universal cord 62. A medical support device 32 and a reception device 64 are connected to the control device 28. The display device 14 is also connected to the medical support device 32. That is, the control device 28 is connected to the display device 14 via the medical support device 32.

[0057] Note that, because the medical support device 32 is exemplified here as an external device for expanding the functions performed by the control device 28, an example in which the control device 28 and the display device 14 are indirectly connected via the medical support device 32 is given, but this is merely one example. For example, the display device 14 may be directly connected to the control device 28. In this case, for example, the functions of the medical support device 32 may be installed in the control device 28, or the control device 28 may be equipped with a function for causing a server (not shown) to execute the same processing as that executed by the medical support device 32 (e.g., the medical support processing described below) and receiving and using the processing results from the server.

[0058] The reception device 64 receives instructions from the doctor 16 and outputs the received instructions as an electrical signal to the control device 28. Examples of the reception device 64 include a keyboard, a mouse, a touch panel, a foot switch, a microphone, and / or a remote control device.

[0059] The control device 28 controls the light source device 30 , exchanges various signals with the camera 52 , and exchanges various signals with the medical support device 32 .

[0060] The light source device 30 emits light under the control of the control device 28 and supplies the light to the illumination device 54. The illumination device 54 has a built-in light guide, and the light supplied from the light source device 30 passes through the light guide and is irradiated from illumination windows 54A and 54B. The control device 28 causes the camera 52 to capture an image, acquires an endoscopic image 40 (see FIG. 1 ) from the camera 52, and outputs the image to a predetermined output destination (for example, the medical support device 32).

[0061] The medical support device 32 performs various image processing on the endoscopic image 40 input from the control device 28. The medical support device 32 outputs the endoscopic image 40 that has been subjected to various image processing to a predetermined output destination (for example, the display device 14).

[0062] Although the embodiment in which the endoscopic image 40 output from the control device 28 is output to the display device 14 via the medical support device 32 has been described above, this is merely one example. For example, the control device 28 and the display device 14 may be connected, and the endoscopic image 40 that has been subjected to image processing by the medical support device 32 may be displayed on the display device 14 via the control device 28.

[0063] 3 , the control device 28 includes a computer 66, a bus 68, and an external I / F 70. The computer 66 includes a processor 72, a RAM 74, and an NVM 76. The processor 72, the RAM 74, the NVM 76, and the external I / F 70 are connected to the bus 68.

[0064] For example, the processor 72 has at least one CPU and at least one GPU, and controls the entire control device 28. The GPU operates under the control of the CPU and is responsible for executing various graphic processing operations and performing calculations using neural networks. The processor 72 may be one or more CPUs that have integrated GPU functionality, or one or more CPUs that do not have integrated GPU functionality. In the example shown in FIG. 3 , the computer 66 is equipped with one processor 72, but this is merely an example, and the computer 66 may be equipped with multiple processors 72.

[0065] The RAM 74 is a memory that temporarily stores information and is used as a work memory by the processor 72. The NVM 76 is a nonvolatile storage device that stores various programs, various parameters, and the like. An example of the NVM 76 is a flash memory (for example, an EEPROM and / or an SSD). Note that the flash memory is merely one example, and the NVM 76 may be another nonvolatile storage device such as an HDD, or may be a combination of two or more types of nonvolatile storage devices.

[0066] The external I / F 70 controls the exchange of various information between one or more devices (hereinafter also referred to as “first external devices”) existing outside the control device 28 and the processor 72. An example of the external I / F 70 is a USB interface.

[0067] The external I / F 70 is connected to the camera 52 as one of the first external devices, and the external I / F 70 controls the exchange of various information between the camera 52 and the processor 72. The processor 72 controls the camera 52 via the external I / F 70. The processor 72 also acquires, via the external I / F 70, endoscopic images 40 (see FIG. 1 ) obtained by the camera 52 capturing an image of the inside of the large intestine 22 (see FIG. 1 ).

[0068] The light source device 30 is connected to the external I / F 70 as one of the first external devices, and the external I / F 70 controls the exchange of various information between the light source device 30 and the processor 72. The light source device 30 supplies light to the illumination device 54 under the control of the processor 72. The illumination device 54 irradiates the light supplied from the light source device 30.

[0069] A reception device 64 is connected to the external I / F 70 as one of the first external devices, and the processor 72 acquires instructions accepted by the reception device 64 via the external I / F 70 and executes processing according to the acquired instructions.

[0070] The medical support device 32 includes a computer 78 and an external I / F 80. The computer 78 includes a processor 82, a RAM 84, and an NVM 86. The processor 82, the RAM 84, the NVM 86, and the external I / F 80 are connected to a bus 88. In the first embodiment, the medical support device 32 is an example of a "medical support device" according to the technology of the present disclosure, the computer 78 is an example of a "computer" according to the technology of the present disclosure, and the processor 82 is an example of a "processor" according to the technology of the present disclosure.

[0071] The hardware configuration of the computer 78 (i.e., the processor 82, RAM 84, and NVM 86) is basically the same as the hardware configuration of the computer 66, so a description of the hardware configuration of the computer 78 will be omitted here.

[0072] The external I / F 80 controls the exchange of various information between the processor 82 and one or more devices (hereinafter also referred to as "second external devices") that exist outside the medical support device 32. An example of the external I / F 80 is a USB interface.

[0073] The control device 28 is connected to the external I / F 80 as one of the second external devices. In the example shown in Fig. 3, the external I / F 70 of the control device 28 is connected to the external I / F 80. The external I / F 80 controls the exchange of various information between the processor 82 of the medical support device 32 and the processor 72 of the control device 28. For example, the processor 82 acquires the endoscopic image 40 (see Fig. 1 ) from the processor 72 of the control device 28 via the external I / Fs 70 and 80, and performs various image processing on the acquired endoscopic image 40.

[0074] The display device 14, which serves as one of the second external devices, is connected to the external I / F 80. The processor 82 controls the display device 14 via the external I / F 80, thereby causing the display device 14 to display various information (e.g., an endoscopic image 40 that has been subjected to various image processing).

[0075] During an endoscopic examination, the doctor 16 checks the endoscopic image 40 via the display device 14 and determines whether or not medical treatment is required for the lesion 42 shown in the endoscopic image 40, and if necessary, performs medical treatment on the lesion 42. The size of the lesion 42 is an important factor in determining whether or not medical treatment is required.

[0076] In recent years, advances in machine learning have made it possible to use AI to detect and differentiate lesions 42 based on endoscopic images 40. Applying this technology makes it possible to measure the size of lesions 42 from endoscopic images 40. Measuring the size of lesions 42 with high accuracy and presenting the measurement results to physician 16 is extremely useful for physician 16 in performing medical treatment on the lesion.

[0077] In view of the above circumstances, in the first embodiment, as shown in FIG. 4 as an example, medical support processing is performed by a processor 82 of the medical support device 32.

[0078] A medical support program 90 is stored in the NVM 86. The medical support program 90 is an example of a "program" according to the technology of the present disclosure. The processor 82 performs medical support processing by reading the medical support program 90 from the NVM 86 and executing the read medical support program 90 on the RAM 84. The medical support processing is realized by the processor 82 operating as a recognition unit 82A, a measurement unit 82B, and a control unit 82C in accordance with the medical support program 90 executed on the RAM 84.

[0079] The NVM 86 stores a recognition model 92 and a distance derivation model 94. As will be described in detail later, the recognition model 92 is used by the recognition unit 82A, and the distance derivation model 94 is used by the measurement unit 82B. In the first embodiment, the recognition model 92 is an example of "AI" according to the technology of the present disclosure.

[0080] As an example, as shown in FIG. 5, the recognition unit 82A and the control unit 82C acquire the endoscopic image 40 generated by the camera 52 capturing images at an imaging frame rate (e.g., several tens of frames per second) from the camera 52 on a frame-by-frame basis.

[0081] The control unit 82C displays the endoscopic image 40 as a live view image in the first display area 36. That is, each time the control unit 82C acquires an endoscopic image 40 frame by frame from the camera 52, the control unit 82C displays the acquired endoscopic image 40 in sequence in the first display area 36 at a display frame rate (e.g., several tens of frames per second).

[0082] The recognition unit 82A uses the endoscopic image 40 acquired from the camera 52 to recognize the lesion 42 in the endoscopic image 40. That is, the recognition unit 82A recognizes the characteristics of the lesion 42 shown in the endoscopic image 40 by performing a recognition process 96 on the endoscopic image 40 acquired from the camera 52. Here, the recognition unit 82A recognizes the characteristics of the lesion 42, such as the shape of the lesion 42, the type of the lesion 42, the lesion 42 type (e.g., pedunculated, subpedunculated, sessile, surface-protruding, surface-flat, surface-depressed, etc.), and the clarity of the outline of the lesion 42. The recognition of the shape of the lesion 42 and the clarity of the outline of the lesion 42 is achieved by recognizing the position of the lesion 42 in the endoscopic image 40 (i.e., the position of the lesion 42 shown in the endoscopic image 40).

[0083] The recognition process 96 is performed by the recognition unit 82A on the acquired endoscopic image 40 each time the endoscopic image 40 is acquired. The recognition process 96 is a process that recognizes the lesion 42 using an AI-based method. In the first embodiment, for example, the recognition process 96 uses object recognition processing using an AI segmentation method (e.g., semantic segmentation, instance segmentation, and / or panoptic segmentation).

[0084] Here, processing using a recognition model 92 is performed as the recognition processing 96. The recognition model 92 is a trained model for object recognition using an AI segmentation method. An example of a trained model for object recognition using an AI segmentation method is a model for semantic segmentation. An example of a model for semantic segmentation is a model with an encoder-decoder structure. An example of a model with an encoder-decoder structure is U-Net or HRNet.

[0085] The recognition model 92 is optimized by performing machine learning on the neural network using first training data. The first training data is a data set including a plurality of data (i.e., a plurality of frames of data) in which first example data and first correct answer data are associated with each other.

[0086] The first example data is an image corresponding to the endoscopic image 40. The first correct answer data is correct answer data (i.e., annotations) for the first example data. Here, an annotation that identifies the position, type, and model of a lesion shown in the image used as the first example data is used as an example of the first correct answer data.

[0087] The recognition unit 82A acquires an endoscopic image 40 from the camera 52 and inputs the acquired endoscopic image 40 to the recognition model 92. As a result, each time an endoscopic image 40 is input, the recognition model 92 identifies the position of a segmentation image 44 identified by the segmentation method as the position of a lesion 42 appearing in the input endoscopic image 40, and outputs position identification information 98 that can identify the position of the segmentation image 44. An example of the position identification information 98 is coordinates that identify the segmentation image 44 within the endoscopic image 40. Furthermore, each time an endoscopic image 40 is input, the recognition model 92 recognizes the type of lesion 42 appearing in the input endoscopic image 40 and outputs type information 100 that indicates the recognized type. Furthermore, each time an endoscopic image 40 is input, the recognition model 92 recognizes the type of lesion 42 appearing in the input endoscopic image 40 and outputs type information 102 that indicates the recognized type. The position identification information 98, type information 100, and type information 102 are associated with the segmentation image 44.

[0088] The control unit 82C displays a probability map 45 indicating the distribution of the positions of the lesions 42 in the second display area 38 for each endoscopic image 40 in accordance with the segmentation image 44 and the position identification information 98. The probability map 45 is a map that expresses the distribution of the positions of the lesions 42 in the endoscopic image 40 using probability, which is an example of an index indicating likelihood. The probability map 45 is obtained by the recognition unit 82A from the recognition model 92 for each endoscopic image 40. Note that the probability map 45 is generally also called a reliability map, a certainty map, or the like.

[0089] The probability map 45 displayed in the second display area 38 is updated according to the display frame rate applied to the first display area 36. That is, the display of the probability map 45 in the second display area 38 (i.e., the display of the segmentation image 44) is updated in synchronization with the display timing of the endoscopic image 40 displayed in the first display area 36. This allows the doctor 16 to grasp the general position of the lesion 42 in the endoscopic image 40 displayed in the first display area 36 by referring to the probability map 45 displayed in the second display area 38 while observing the endoscopic image 40 displayed in the first display area 36. In the first embodiment, the probability map 45 is an example of a "probability map" according to the technology of the present disclosure.

[0090] In the probability map 45, the location of the lesion 42 is divided according to probability. In the example shown in FIG. 5 , the probability map 45 is divided into three closed regions, a first divided region 105, a second divided region 106, and a third divided region 108, according to thresholds α and β. The thresholds α and β have a relationship of "α > β." In the probability map 45, a closed region whose probability is equal to or greater than the threshold α is the first divided region 105, which corresponds to the segmentation image 44. In the probability map 45, a closed region whose probability is equal to or greater than the threshold β but less than the threshold α is the second divided region 106. In the probability map 45, a closed region whose probability is less than the threshold β is the third divided region 108. Examples of the thresholds α and β include values ​​determined according to instructions received by the reception device 64 or values ​​determined according to various conditions. The thresholds α and / or β may be fixed values ​​or variable values ​​that change according to given instructions and various conditions.

[0091] In the first embodiment, the thresholds α and β are an example of "plurality of thresholds" according to the technology of the present disclosure. Also, in the first embodiment, the first divided area 105, the second divided area 106, and the third divided area 108 are an example of "plurality of divided areas" according to the technology of the present disclosure. Also, in the first embodiment, the first divided area 105 is an example of a "closed area" and a "first divided area" according to the technology of the present disclosure, and the second divided area 106 is an example of a "closed area" and a "second divided area" according to the technology of the present disclosure.

[0092] The position specifying information 98 is broadly divided into position specifying information 98A and 98B. The position specifying information 98A is associated with the segmentation image 44 (i.e., the first demarcated region 105), and the position specifying information 98B is associated with the second demarcated region 106. The position specifying information 98A is a plurality of coordinates that specify the position of the contour of the segmentation image 44 within the endoscopic image 40. The position specifying information 98B is a plurality of coordinates that specify the position of the contour (e.g., outer contour and inner contour) of the second demarcated region 106 within the endoscopic image 40.

[0093] Although the thresholds α and β are exemplified here, the technology of the present disclosure is not limited to this and may have three or more thresholds. Also, although the first divided area 105, the second divided area 106, and the third divided area 108 are exemplified here, the technology of the present disclosure is not limited to this and may have one divided area or three or more divided areas, and the number of divided areas can vary depending on the number of thresholds.

[0094] As an example, as shown in FIG. 6 , the measurement unit 82B measures a minimum size 112A of the lesion 42 based on the endoscopic image 40 acquired from the camera 52. To measure the minimum size 112A of the lesion 42, the measurement unit 82B acquires distance information 114 of the lesion 42 based on the endoscopic image 40 acquired from the camera 52. The distance information 114 is information indicating the distance from the camera 52 (i.e., the observation position) to the intestinal wall 24 (see FIG. 1 ) including the lesion 42. Note that while the distance from the camera 52 to the intestinal wall 24 including the lesion 42 is illustrated here, this is merely an example, and instead of the distance, a numerical value indicating the depth from the camera 52 to the intestinal wall 24 including the lesion 42 (e.g., a plurality of numerical values ​​specifying the depth in stages (e.g., numerical values ​​ranging from several stages to several tens of stages)) may be used.

[0095] The distance information 114 is acquired for each of all pixels constituting the endoscopic image 40. Note that the distance information 114 may also be acquired for each block of the endoscopic image 40 that is larger than a pixel (for example, a pixel group made up of several to several hundred pixels).

[0096] The measurement unit 82B acquires the distance information 114, for example, by deriving the distance information 114 using an AI method. In the first embodiment, a distance derivation model 94 is used to derive the distance information 114.

[0097] The distance derivation model 94 is optimized by performing machine learning on the neural network using second training data. The second training data is a data set including a plurality of data (i.e., a plurality of frames of data) in which second example data and second answer data are associated with each other.

[0098] The second example data is an image corresponding to the endoscopic image 40. The second supervised data is supervised data (i.e., annotations) for the second example data. Here, an annotation that specifies the distance corresponding to each pixel in the image used as the second example data is used as an example of the second supervised data.

[0099] The measurement unit 82B acquires the endoscopic image 40 from the camera 52 and inputs the acquired endoscopic image 40 to the distance derivation model 94. As a result, the distance derivation model 94 outputs distance information 114 in pixel units of the input endoscopic image 40. That is, in the measurement unit 82B, information indicating the distance from the position of the camera 52 (for example, the position of an image sensor or objective lens mounted on the camera 52) to the intestinal wall 24 shown in the endoscopic image 40 is output from the distance derivation model 94 as distance information 114 in pixel units of the endoscopic image 40.

[0100] The measurement unit 82B generates a distance image 116 based on the distance information 114 output from the distance derivation model 94. The distance image 116 is an image in which the distance information 114 is distributed for each pixel included in the endoscopic image 40.

[0101] The measurement unit 82B acquires the position identification information 98A assigned to the segmentation image 44 in the probability map 45 obtained by the recognition unit 82A. The measurement unit 82B references the position identification information 98A and extracts, from the distance image 116, distance information 114 corresponding to the position identified from the position identification information 98A. Examples of the distance information 114 extracted from the distance image 116 include distance information 114 corresponding to the position (e.g., center of gravity) of the lesion 42, or a statistical value (e.g., median, average, or mode) of the distance information 114 for multiple pixels (e.g., all pixels) included in the lesion 42.

[0102] The measurement unit 82B extracts a pixel count 118 from the endoscopic image 40. The pixel count 118 is the number of pixels on a line segment 120 that crosses an image region (i.e., an image region showing the lesion 42) at a position identified by the position identification information 98A among the entire image region of the endoscopic image 40 input to the distance derivation model 94. An example of the line segment 120 is the longest line segment parallel to the long side of a circumscribing rectangular frame 122 for the image region showing the lesion 42. Note that the line segment 120 is merely an example, and instead of the line segment 120, the longest line segment parallel to the short side of the circumscribing rectangular frame 122 for the image region showing the lesion 42 may be used.

[0103] In this first embodiment, line segment 120 is an example of the "long side of the observation area" according to the technology of the present disclosure, and the longest line segment parallel to the short side of circumscribing rectangular frame 122 for the image area showing lesion 42 is an example of the "short side of the observation area" according to the technology of the present disclosure.

[0104] The measurement unit 82B calculates a minimum size 112A of the lesion 42 in real space based on the distance information 114 extracted from the distance image 116 and the number of pixels 118 extracted from the endoscopic image 40. The minimum size 112A refers to, for example, the smallest size expected for the lesion 42 in real space. In the example shown in FIG. 6 , the minimum size 112A is the size in real space of the first delimited region 105, which is the smallest of the multiple delimited regions, i.e., the size in real space of the segmentation image 44 (i.e., the actual size within the body).

[0105] A calculation formula 124 is used to calculate minimum size 112A. Measurement unit 82B inputs distance information 114 extracted from distance image 116 and number of pixels 118 extracted from endoscopic image 40 into calculation formula 124. A calculation formula 124 is an arithmetic expression in which distance information 114 and number of pixels 118 are independent variables and minimum size 112A is a dependent variable. A calculation formula 124 outputs minimum size 112A corresponding to the input distance information 114 and number of pixels 118.

[0106] Although the length of the lesion 42 in real space is exemplified as minimum size 112A here, the technology of the present disclosure is not limited to this, and minimum size 112A may be the surface area or volume of the lesion 42 in real space. In this case, for example, an arithmetic expression 124 is used in which the number of pixels in the entire image region representing the lesion 42 and distance information 114 are independent variables, and the surface area or volume of the lesion 42 in real space is a dependent variable.

[0107] 7 , the measurement unit 82B measures the maximum size 112B of the lesion 42 based on the endoscopic image 40 acquired from the camera 52. The maximum size 112B is measured in the same manner as the minimum size 112A. While the minimum size 112A was measured using the segmentation image 44 and the position identification information 98A, the maximum size 112B is measured using the second partitioned region 106 and the position identification information 98B. This will be described in more detail below.

[0108] The measurement unit 82B acquires the position identification information 98B assigned to the second divided region 106 in the probability map 45 obtained by the recognition unit 82A. The measurement unit 82B references the position identification information 98B and extracts, from the distance image 116, distance information 114 corresponding to the position identified from the position identification information 98B. The distance information 114 extracted from the distance image 116 may be, for example, distance information 114 corresponding to the annular closed region (e.g., the inner contour and / or the outer contour) identified from the position identification information 98B in the endoscopic image 40, or a statistical value (e.g., median, average, or mode) of the distance information 114 for a plurality of pixels included in the annular closed region identified from the position identification information 98B in the endoscopic image 40 (e.g., all pixels constituting the annular closed region, all pixels constituting the inner contour of the annular closed region, or all pixels constituting the outer contour of the annular closed region).

[0109] The measurement unit 82B extracts an image region 128 from the endoscopic image 40. The image region 128 is a region surrounded by the outer contour of a closed ring-shaped region identified from the position identification information 98B within the entire image region of the endoscopic image 40 input to the distance derivation model 94. The measurement unit 82B then extracts a pixel count 126 from the image region 128. The pixel count 126 is the number of pixels on a line segment 130 that crosses the image region 128. An example of the line segment 130 is the longest line segment parallel to the long side of a circumscribing rectangular frame 132 for the image region 128. Note that the line segment 130 is merely an example, and the longest line segment parallel to the short side of the circumscribing rectangular frame 132 for the image region 128 may be used instead of the line segment 130.

[0110] The measurement unit 82B calculates a maximum size 112B of the lesion 42 in real space based on the distance information 114 extracted from the distance image 116 and the number of pixels 126 extracted from the endoscopic image 40. The maximum size 112B refers to, for example, the largest size that is predicted as the size of the lesion 42 in real space. In the example shown in FIG. 7 , the size in real space of the second demarcated region 106, which is outside the first demarcated region 105 among the multiple demarcated regions, (i.e., the actual size within the body) is shown as an example of the maximum size 112B.

[0111] A calculation formula 134 is used to calculate maximum size 112B. Measurement unit 82B inputs distance information 114 extracted from distance image 116 and number of pixels 126 extracted from endoscopic image 40 into calculation formula 134. Calculation formula 134 is an arithmetic expression in which distance information 114 and number of pixels 126 are independent variables and maximum size 112B is a dependent variable. Calculation formula 134 outputs maximum size 112B corresponding to the input distance information 114 and number of pixels 126.

[0112] Furthermore, although the maximum size 112B is exemplified here as the length of the second demarcated region 106 in real space, the technology of the present disclosure is not limited to this, and the maximum size 112B may be the surface area or volume in real space of the second demarcated region 106. In this case, for example, an arithmetic expression 134 is used in which the number of pixels of the entire image region indicating the second demarcated region 106 and the distance information 114 are used as independent variables, and the surface area or volume of the second demarcated region 106 in real space is used as a dependent variable.

[0113] 6 and 7, two sizes, a minimum size 112A and a maximum size 112B, are measured by the measuring unit 82B, but this is merely an example, and three or more sizes may be measured by the measuring unit 82B. In this case, three or more divided areas are obtained using three or more thresholds, and the size of each divided area is measured in the manner described above.

[0114] In the first embodiment, the minimum size 112A and the maximum size 112B are examples of "multiple first sizes of the observation target region." Also, in the first embodiment, the minimum size 112A is an example of a "lower limit value of width" according to the technology of the present disclosure. Also, in the first embodiment, the maximum size 112B is an example of an "upper limit value of width" according to the technology of the present disclosure. Also, in the first embodiment, the size 112C is an example of a "representative value of multiple first sizes" according to the technology of the present disclosure.

[0115] 8 , the measurement unit 82B measures a size 112C of the lesion 42 using a minimum size 112A and a maximum size 112B measured based on the endoscopic image 40. In the first embodiment, the "size 112C" is an example of the "size of the observation region" according to the technology of the present disclosure.

[0116] The size 112C is a size according to the characteristics of the lesion 42. Here, the characteristics of the lesion 42 refer to, for example, the shape, type, and contour clarity of the lesion 42. The shape and contour clarity of the lesion 42 (e.g., the range of variation in the contour of the lesion 42 due to the way the lesion 42 appears in the endoscopic image 40) are identified from the minimum size 112A and the maximum size 112B. The type of the lesion 42 is identified from the type information 100, and the shape of the lesion 42 is identified from the type information 102.

[0117] In the first embodiment, the minimum size 112A and the maximum size 112B are sizes according to the characteristics of the lesion 42, and the size 112C is measured by deriving the size 112C from the minimum size 112A and the maximum size 112B.

[0118] Size 112C is calculated from equation 135, which has minimum size 112A and maximum size 112B as independent variables and size 112C as a dependent variable. Size 112C may also be derived from a table, which has minimum size 112A and maximum size 112B as inputs and size 112C as an output.

[0119] The size 112C is, for example, the average value of the minimum size 112A and the maximum size 112B. The average value is merely an example, and any representative value of the minimum size 112A and the maximum size 112B may be used. An example of the representative value is a statistical value. The statistical value refers to a maximum value, a minimum value, a median value, an average value, and / or a variance value, etc.

[0120] However, with conventionally known techniques, the size of the lesion 42 within one frame may not be uniquely determined due to factors such as the appearance of the lesion 42 in the endoscopic image 40, the shape of the lesion 42 in the endoscopic image 40, the structure of the AI, and / or insufficient learning of the AI. For example, the measured size may have a range (i.e., a fluctuation range), or multiple sizes may be measured.

[0121] Therefore, the measurement unit 82B derives width information 136 based on the minimum size 112A and the maximum size 112B. The width information 136 is information indicating the width of the size of the lesion 42 in real space (hereinafter also referred to as the "actual size of the lesion 42"). The width of the actual size of the lesion 42 is determined based on the first demarcated region 105 (i.e., the segmentation image 44) and the second demarcated region 106. For example, the width of the actual size of the lesion 42 is determined using the minimum size 112A and the maximum size 112B.

[0122] The measurement unit 82B derives information (e.g., text information or an image) capable of identifying both the maximum size 112B and the minimum size 112A as the width information 136. Note that this is merely one example, and the measurement unit 82B may derive, as the width information 136, information using the absolute value of the difference between the maximum size 112B and the minimum size 112A, in addition to or instead of information capable of identifying both the maximum size 112B and the minimum size 112A.

[0123] The control unit 82C acquires the size 112C from the measurement unit 82B. The control unit 82C also acquires the probability map 45 from the recognition unit 82A. The control unit 82C displays the probability map 45 acquired from the recognition unit 82A in the second display area 38. The control unit 82C then displays the size 112C acquired from the measurement unit 82B within the probability map 45. For example, the size 112C is displayed superimposed on the probability map 45. Note that the superimposed display is merely an example, and an embedded display may also be used.

[0124] The control unit 82C displays a dimension line 138 in the probability map 45 as a mark that enables identification of which portion of the lesion 42 the size 112C displayed in the probability map 45 corresponds to (i.e., the length). The control unit 82C acquires the position identification information 98A from the recognition unit 82A, and creates and displays the dimension line 138 based on the position identification information 98A. The dimension line 138 may be created, for example, in a manner similar to that used to create the line segment 120 (i.e., in a manner similar to that used with the circumscribing rectangular frame 122).

[0125] The control unit 82C acquires the type information 100 and the type information 102 from the recognition unit 82A, and displays the type of lesion 42 indicated by the type information 100 and the type of lesion 42 indicated by the type information 102 on the screen 35. In the example shown in Fig. 8 , the type information 100 and the type information 102 are displayed in text format on the screen 35. The type information 100 and the type information 102 may also be displayed on the screen 35 in a format other than text format (for example, an image, etc.).

[0126] The control unit 82C acquires width information 136 from the measurement unit 82B and displays the actual width of the lesion 42 indicated by the width information 136 on the screen 35. In the example shown in FIG. 8 , the width information 136 is displayed in text format on the screen 35. The width information 136 may also be displayed on the screen 35 in a format other than text (e.g., an image, etc.). For example, a curve obtained by offsetting the outer contour of the segmentation image 44 outward by the width indicated by the width information 136 may be displayed on the periphery of the segmentation image 44, or the width information 136 may be displayed as an image along the periphery of the segmentation image 44 in a specific display mode (e.g., a display mode that is distinguishable from other regions in the probability map 45).

[0127] Next, the operation of the portion of the endoscope system 10 related to the technology of the present disclosure will be described with reference to Fig. 9. The flow of medical support processing shown in Fig. 9 is an example of a "medical support method" related to the technology of the present disclosure.

[0128] 9 , first, in step ST10, the recognition unit 82A determines whether or not one frame of image data has been captured by the camera 52 inside the large intestine 22. If one frame of image data has not been captured by the camera 52 inside the large intestine 22 in step ST10, the determination is negative, and the determination in step ST10 is made again. If one frame of image data has been captured by the camera 52 inside the large intestine 22 in step ST10, the determination is positive, and the medical support process proceeds to step ST12.

[0129] In step ST12, the recognition unit 82A and the control unit 82C acquire one frame of the endoscopic image 40 obtained by capturing an image of the large intestine 22 with the camera 52 (see FIG. 5). For ease of explanation, the following description will be given on the assumption that the endoscopic image 40 shows a lesion 42. After the processing of step ST12 is executed, the medical support processing proceeds to step ST14.

[0130] In step ST14, the control unit 82C displays the endoscopic image 40 acquired in step ST12 in the first display area 36 (see FIGS. 1, 5, and 8). After the processing of step ST14 is executed, the medical support processing proceeds to step ST16.

[0131] In step ST16, the recognition unit 82A performs a recognition process 96 using the endoscopic image 40 acquired in step ST12 to recognize the position, type, and model of the lesion 42 in the endoscopic image 40, and acquires position identification information 98, type information 100, and model information 102 (see FIG. 5 ). After the process of step ST16 is executed, the medical support process proceeds to step ST18.

[0132] In step ST18, the recognition unit 82A acquires the probability map 45 from the recognition model 92 used in step ST16 to recognize the position, type, and form of the lesion 42. Then, the control unit 82C displays the probability map 45 acquired from the recognition model 92 by the recognition unit 82A in the second display area 38 (see FIGS. 1, 5, and 8). After the processing of step ST18 is executed, the medical support processing proceeds to step ST20.

[0133] In step ST20, the measurement unit 82B measures a minimum size 112A of the lesion 42 based on the endoscopic image 40 used in step ST16 and position identification information 98A obtained by performing the recognition process 96 in step ST16 (see FIG. 6). The measurement unit 82B also measures a maximum size 112B of the lesion 42 based on the endoscopic image 40 used in step ST16 and position identification information 98B obtained by performing the recognition process 96 in step ST16 (see FIG. 7). After the process of step ST20 is executed, the medical support process proceeds to step ST22.

[0134] In step ST22, the measurement unit 82B derives the size 112C and the width information 136 based on the minimum size 112A and the maximum size 112B measured in step ST20. After the process of step ST22 is executed, the medical support process proceeds to step ST24.

[0135] In step ST24, the control unit 82C displays on the screen 35 the type of lesion 42 indicated by the type information 100 acquired in step ST16 and the type of lesion 42 indicated by the type information 102 acquired in step ST16 (see FIG. 8). The control unit 82C also displays on the screen 35 the width indicated by the width information 136 derived in step ST22 (see FIG. 8). Furthermore, the control unit 82C displays the size 112C derived in step ST22 in the probability map 45. After the processing of step ST24 is executed, the medical support processing proceeds to step ST26.

[0136] In step ST26, the control unit 82C determines whether a condition for terminating the medical support process is satisfied. One example of the condition for terminating the medical support process is that an instruction to terminate the medical support process has been given to the endoscope system 10 (for example, that an instruction to terminate the medical support process has been accepted by the acceptance device 64).

[0137] In step ST26, if the condition for terminating the medical support process is not satisfied, the determination is negative and the medical support process proceeds to step ST10. In step ST24, if the condition for terminating the medical support process is satisfied, the determination is positive and the medical support process ends.

[0138] As described above, in the endoscopic system 10 according to the first embodiment, the recognition unit 82A uses the endoscopic image 40 to recognize the lesion 42 shown in the endoscopic image 40. The measurement unit 82B measures the size 112C of the lesion 42 based on the endoscopic image 40. The control unit 82C then displays the size 112C on the screen 35.

[0139] Here, the size 112C is a size according to the characteristics of the lesion 42. The characteristics of the lesion 42 refer to, for example, the shape, type, and contour clarity of the lesion 42. Therefore, the physician 16 can accurately determine the size 112C of the lesion 42 shown in the endoscopic image 40 compared to when the size of the lesion 42 shown in the endoscopic image 40 is measured without any consideration of the characteristics of the lesion 42. Furthermore, the physician 16 can accurately determine the size 112C of the lesion 42 shown in the endoscopic image 40 compared to when the size of the lesion 42 shown in the endoscopic image 40 is measured without any consideration of the shape, type, and contour clarity of the lesion 42.

[0140] Furthermore, in the endoscopic system 10 according to the first embodiment, the recognition unit 82A recognizes the characteristics of the lesion 42 based on the endoscopic image 40. Therefore, the characteristics of the lesion 42 shown in the endoscopic image 40 can be identified with high accuracy.

[0141] Furthermore, in the endoscopic system 10 according to the first embodiment, the size 112C of the range corresponding to the line segment 120 is measured and displayed on the screen 35. The line segment 120 is the longest line segment parallel to the long side of the circumscribing rectangular frame 122 for the image region showing the lesion 42. Therefore, the doctor 16 can grasp the length in real space of the longest range that crosses the lesion 42 along the longest line segment parallel to the long side of the circumscribing rectangular frame 122 for the image region showing the lesion 42.

[0142] Furthermore, in the endoscopic system 10 according to the first embodiment, the lesion 42 is recognized by a method using the recognition model 92, and the size 112C is measured based on the probability map 45 obtained from the recognition model 92. Therefore, the size 112C of the lesion 42 shown in the endoscopic image 40 can be measured with high accuracy.

[0143] Furthermore, in the endoscopic system 10 according to the first embodiment, measurements are made based on closed regions obtained by dividing the probability map 45 in accordance with the thresholds α and β. Therefore, the position of the lesion 42 in the endoscopic image 40 can be identified with high accuracy, and the size 112C of the lesion 42 shown in the endoscopic image 40 (i.e., the size in real space) can be measured with high accuracy.

[0144] Furthermore, in the endoscopic system 10 according to the first embodiment, the size 112C of the lesion 42 is measured based on the first divided region 105 and the second divided region 106 obtained by dividing the probability map 45 in accordance with the thresholds α and β. Therefore, even if the outline of the lesion 42 shown in the endoscopic image 40 is unclear due to body movement and / or movement of the camera 52, the size 112C of the lesion 42 can be measured with high accuracy.

[0145] Furthermore, in the endoscopic system 10 according to the first embodiment, width information 136 is measured based on the minimum size 112A and the maximum size 112B and displayed on the screen 35. This allows the doctor 16 to accurately grasp the range of variation in the actual size of the lesion 42 shown in the endoscopic image 40. In other words, the doctor 16 can accurately grasp the lower and upper limits of the actual size of the lesion 42 shown in the endoscopic image 40. As a result, the doctor 16 can predict that the actual size of the lesion 42 is likely to be within the range indicated by the width information 136.

[0146] Furthermore, in the endoscope system 10 according to the first embodiment, a representative value (e.g., a maximum value, a minimum value, an average value, a median value, and / or a variance value) of the minimum size 112A and the maximum size 112B is used as the size 112C displayed on the screen 35. Therefore, compared to when multiple sizes are displayed on the screen 35, the doctor 16 can grasp the actual size of the lesion 42 without any confusion.

[0147] In the first embodiment, an example was given in which the size 112C and the dimension line 138 are displayed in the probability map 45, but the technology of the present disclosure is not limited to this. For example, as shown in Fig. 10 , the size 112C and the dimension line 138 may be displayed in the endoscopic image 40. The size 112C and / or the dimension line 138 may be superimposed on the endoscopic image 40 using an alpha blending method, or the display mode, such as the display position, display size, and / or display color, in the endoscopic image 40 may be changed in accordance with an instruction received by the reception device 64.

[0148] In the first embodiment, the length in real space of the longest range crossing the lesion 42 along the line segment 120 is measured as the size 112C. However, the technology of the present disclosure is not limited to this. For example, as shown in FIG. 11 , the size 112D of the range corresponding to the longest line segment parallel to the short side of the circumscribing rectangular frame 122 for the image region showing the lesion 42 may be measured and displayed on the screen 35. In this case, the physician 16 can grasp the length in real space of the longest range crossing the lesion 42 along the longest line segment parallel to the short side of the circumscribing rectangular frame 122 for the image region showing the lesion 42.

[0149] Additionally, the actual size of the lesion 42 in terms of the radius and / or diameter of the circumscribing circle for the image region showing the lesion 42 may be displayed on the measured screen 35. In this case, the actual size of the lesion 42 in terms of the radius and / or diameter of the circumscribing circle for the image region showing the lesion 42 can be known to the physician 16.

[0150] Second Embodiment In the first embodiment, an example was given in which the lesion 42 was imaged by the camera 52 when there was no obstruction between the camera 52 and the lesion 42. However, in the second embodiment, a case will be described in which an obstruction exists between the camera 52 and the lesion 42 and an area including the obstruction and a portion of the lesion 42 that is not obstructed by the obstruction (i.e., the inside of the large intestine 22) is imaged by the camera 52. For ease of explanation, the following description will focus mainly on the differences from the first embodiment, with the same components as those in the first embodiment being assigned the same reference numerals and their descriptions omitted.

[0151] As an example, as shown in FIG. 12 , a medical support program 90A is stored in the NVM 86. The medical support program 90A is an example of a "program" according to the technology of the present disclosure. The processor 82 reads the medical support program 90A from the NVM 86 and executes the read medical support program 90A on the RAM 84. The medical support processing according to the second embodiment is realized by the processor 82 operating as a recognition unit 82A, a measurement unit 82B, a control unit 82C, and a generation unit 82D in accordance with the medical support program 90A executed on the RAM 84.

[0152] The NVM 86 stores an image generation model 140. As will be described in detail later, the image generation model 140 is used by the generation unit 82D.

[0153] 13 , an endoscopic image 40 shows a lesion 42, but part of the lesion 42 is obscured by folds 43. In other words, when the lesion 42 is observed from the position of the camera 52, part of the lesion 42 overlaps with the folds 43, which are the peripheral area of ​​the lesion 42.

[0154] In the second embodiment, the fold 43 is an example of a "peripheral region" according to the technology of the present disclosure. Also, in the second embodiment, the overlap between the lesion 42 and the fold 43 when the lesion 42 is observed from the position of the camera 52 is an example of a "characteristic" and an "overlap between the observation region and the peripheral region" according to the technology of the present disclosure.

[0155] 13 , the lesion 42 shown in the endoscopic image 40 is roughly divided into a visible portion 42 A and an invisible portion 42 B. The visible portion 42 A is the portion of the lesion 42 shown in the endoscopic image 40 that is not obstructed by the folds 43 when the lesion 42 is observed from the position of the camera 52 (i.e., the portion that does not overlap with the folds 43), and is visually recognized by the physician 16 through the endoscopic image 40.

[0156] In contrast, the invisible portion 42B is the portion of the lesion 42 that is obscured by the folds 43 (i.e., the portion that overlaps with the folds 43) that appears in the endoscopic image 40 when the lesion 42 is observed from the position of the camera 52, and is not visible to the doctor 16 through the endoscopic image 40.

[0157] The recognition unit 82A performs a recognition process 96 on the endoscopic image 40 containing the visible portion 42A in the same manner as in the first embodiment, thereby acquiring position identification information 99, type information 100A, and model information 102A for the visible portion 42A. The position identification information 99 corresponds to the position identification information 98 described in the first embodiment, the type information 100A corresponds to the type information 100 described in the first embodiment, and the model information 102A corresponds to the model information 102 described in the first embodiment.

[0158] The position specifying information 99 is roughly divided into position specifying information 99 A and position specifying information 99 B. The position specifying information 99 A corresponds to the position specifying information 98 A described in the first embodiment, and the position specifying information 99 B corresponds to the position specifying information 98 B described in the first embodiment.

[0159] The recognition unit 82A acquires a probability map 45A for the visible portion 42A from the recognition model 92 in the same manner as in the first embodiment. The probability map 45A corresponds to the probability map 45 described in the first embodiment. The probability map 45A includes a segmentation image 44A corresponding to the visible portion 42A. The segmentation image 44A corresponds to the segmentation image 44 described in the first embodiment.

[0160] The probability map 45A is divided into three closed regions, a first divided region 105A, a second divided region 106A, and a third divided region 108A, in the same manner as in the first embodiment. In the example shown in FIG. 13 , the probability map 45A is divided into three closed regions, a first divided region 105A, a second divided region 106A, and a third divided region 108A, in accordance with thresholds α1 and β1. The thresholds α1 and β1 correspond to the thresholds α and β described in the first embodiment. The first divided region 105A corresponds to the first divided region 105 described in the first embodiment. The second divided region 106A corresponds to the second divided region 106 described in the first embodiment. The third divided region 108A corresponds to the third divided region 108 described in the first embodiment. As in the first embodiment, the segmentation image 44A is formed by the first divided region 105A.

[0161] The control unit 82C displays the endoscopic image 40 in the first display area 36 and also displays the probability map 45A in the second display area 38, similar to the first embodiment.

[0162] 14 , the generation unit 82D performs image generation processing 142 on an endoscopic image 40 acquired from the camera 52 (here, as an example, an endoscopic image 40 that has been subjected to recognition processing 96). In the image generation processing 142, an image generation model 140 is used. The image generation model 140 is a generation model using a neural network, and is a trained model obtained by performing machine learning on the neural network using second training data. An example of the image generation model 140 is an autoencoder such as a GAN or a VAE.

[0163] The second training data used in machine learning performed on the neural network to create the image generation model 140 is a dataset including multiple data (i.e., multiple frames of data) in which second example data and second correct answer data are associated with each other. The second example data is an image corresponding to the endoscopic image 40 showing a lesion that is partially overlapped with the surrounding area (e.g., a fold, an artificial treatment device, and / or an organ). The second correct answer data is an image corresponding to the endoscopic image 40 showing a lesion that is not overlapped with the surrounding area.

[0164] In the example shown in FIG. 14 , generation unit 82D inputs endoscopic image 40 acquired from camera 52 to image generation model 140. Based on the input endoscopic image 40, image generation model 140 then generates a pseudo image 144 that simulates endoscopic image 40. Pseudo image 144 is an image obtained by supplementing visible portion 42A with predicted invisible image 146A, which corresponds to invisible portion 42B that is not visible in endoscopic image 40 because it overlaps with folds 43. Image generation model 140 predicts invisible portion 42B based on the input endoscopic image 40, generates predicted invisible image 146A indicating the prediction result, and combines it with visible portion 42A to generate predicted lesion image 146, thereby generating pseudo image 144 including predicted lesion image 146.

[0165] 15 , the recognition unit 82A performs a recognition process 96 on the pseudo image 144 including the predicted lesion image 146 in the same manner as in the first embodiment, thereby obtaining position identification information 101, type information 100B, and pattern information 102B for the predicted lesion image 146. The position identification information 101 corresponds to the position identification information 98 described in the first embodiment, the type information 100B corresponds to the type information 100 described in the first embodiment, and the pattern information 102B corresponds to the pattern information 102 described in the first embodiment.

[0166] The position specifying information 101 is roughly divided into position specifying information 101A and position specifying information 101B. The position specifying information 101A corresponds to the position specifying information 98A described in the first embodiment, and the position specifying information 101B corresponds to the position specifying information 98B described in the first embodiment.

[0167] Recognition unit 82A acquires a probability map 45B for predicted lesion image 146 from recognition model 92 in the same manner as in the first embodiment. Probability map 45B corresponds to probability map 45 described in the first embodiment. Probability map 45B includes segmentation image 44B corresponding to predicted lesion image 146. Segmentation image 44B corresponds to segmentation image 44 described in the first embodiment.

[0168] In the probability map 45B, the location of the predicted lesion image 146 is divided by probability in the same manner as in the first embodiment. In the example shown in FIG. 15 , the probability map 45B is divided into three closed regions, a first divided region 105B, a second divided region 106B, and a third divided region 108B, according to thresholds α2 and β2. The thresholds α2 and β2 correspond to the thresholds α and β described in the first embodiment. The first divided region 105B corresponds to the first divided region 105 described in the first embodiment. The second divided region 106B corresponds to the second divided region 106 described in the first embodiment. The third divided region 108B corresponds to the third divided region 108 described in the first embodiment. As in the first embodiment, the segmentation image 44B is formed by the first divided region 105B.

[0169] As an example, as shown in FIG. 16, the measurement unit 82B derives a revealed size 112C1, a predicted size 112C2, and width information 136A and 136B in the same manner as in the first embodiment.

[0170] The apparent size 112C1 is the size of the visible portion 42A in real space. The apparent size 112C1 is derived based on the minimum size 112A1 and the maximum size 112B1. The minimum size 112A1 corresponds to the minimum size 112A described in the first embodiment and is derived based on the position specifying information 99A. The maximum size 112B1 corresponds to the maximum size 112B described in the first embodiment and is derived based on the position specifying information 99B.

[0171] The width information 136A corresponds to the width information 136 described in the first embodiment, and is derived based on the minimum size 112A1 and the maximum size 112B1 in the same manner as in the first embodiment.

[0172] The predicted size 112C2 is the size of the predicted lesion image 146 in real space (i.e., the size predicted as the actual size of the lesion 42). The predicted size 112C2 is derived based on the minimum size 112A2 and the maximum size 112B2. The minimum size 112A2 corresponds to the minimum size 112A described in the first embodiment and is derived based on the position identification information 101A. The maximum size 112B2 corresponds to the maximum size 112B described in the first embodiment and is derived based on the position identification information 101B.

[0173] The width information 136B corresponds to the width information 136 described in the first embodiment, and is derived based on the minimum size 112A2 and the maximum size 112B2 in the same manner as in the first embodiment.

[0174] 17 , control unit 82C displays probability map 45A in second display area 38, and also displays appearance size 112C1 within probability map 45A. Control unit 82C also displays type information 100A, shape information 102A, and width information 136A on screen 35.

[0175] 18 , when an instruction 147 (e.g., an instruction from doctor 16) to switch the display content of screen 35 is received by reception device 64, control unit 82C switches probability map 45A displayed in second display area 38 to probability map 45B, and displays predicted size 112C2 within probability map 45B. Control unit 82C also switches type information 100A, model information 102A, and width information 136A displayed on screen 35 to type information 100B, model information 102B, and width information 136B.

[0176] In this way, in the second embodiment, the apparent size 112C1 and the predicted size 112C2 are displayed on the screen 35. This allows the doctor 16 to grasp the actual size of the lesion 42 when the overlap between the lesion 42 and the folds 43 is included (i.e., the size of the visible portion 42A in real space), and the actual size of the lesion 42 when the overlap between the lesion 42 and the folds 43 is not included (i.e., the size of the lesion 42 made up of the visible portion 42A and the invisible portion 42B in real space).

[0177] In the second embodiment, an example was given in which the display is switched on the condition that instruction 147 is accepted by acceptance device 64. However, this is merely an example, and the display content shown in FIG. 17 may be switched to the display content shown in FIG. 18 when a specified condition is satisfied (for example, a condition that a predetermined time (for example, 10 seconds) has elapsed since the display of apparent size 112C1). Furthermore, the display content shown in FIG. 17 and the display content shown in FIG. 18 may be displayed in parallel on one or more screens. In this case, for example, information corresponding to the display content shown in FIG. 17 and information corresponding to the display content shown in FIG. 18 may be displayed or hidden depending on the instruction accepted by acceptance device 64 and / or various conditions.

[0178] [Other Embodiments] In the above embodiments, the surface flat type is exemplified as the type of lesion 42, but this is merely an example. For example, as shown in FIG. 19 , the technology of the present disclosure can also be applied to pedunculated lesions 42. When the lesion 42 is pedunculated, the lesion 42 is divided into a tip 42C and a stalk 42D. The size of the tip 42C may be measured and displayed on the screen 35 as the revealed size 112C1. Alternatively, the revealed size 112C1a of the long side of the tip 42C and the revealed size 112C1b of the short side of the tip 42C may be measured and displayed on the screen 35. In this case, dimension lines 138 and the like may also be displayed on the screen 35 so that the size of each portion can be determined.

[0179] Furthermore, the revealed size 112C1a and the revealed size 112C1b may be selectively displayed in accordance with instructions and / or various conditions accepted by the accepting device 64. When sizes in multiple directions are measured in this manner, the measured sizes in multiple directions may be selectively displayed on the screen 35 in accordance with instructions and / or various conditions accepted by the accepting device 64. Note that the multiple directions may be determined in accordance with instructions accepted by the accepting device 64 or in accordance with various conditions.

[0180] In the first embodiment, an example was given in which dimension lines 138 were displayed in association with the segmentation image 44 as information for identifying the lesion 42 corresponding to the size 112C displayed in the probability map 45. However, the technology of the present disclosure is not limited to this. For example, a circumscribing rectangular frame for the segmentation image 44, which can identify the position in the endoscopic image 40 of the lesion 42 corresponding to the size 112C displayed in the probability map 45, may be displayed in the probability map 45. In this case, the dimension lines 138 may also be displayed in the probability map 45 together with the circumscribing rectangular frame. Furthermore, when the position of the lesion 42 is recognized using AI based on a bounding box method, for example, a bounding box may be used as the circumscribing rectangular frame. The same applies to the second embodiment.

[0181] In the first embodiment described above, an example was given in which the lesion 42 shown in the endoscopic image 40 is identified from the size and position of the segmentation image 44 in the probability map 45, but the technology of the present disclosure is not limited to this. For example, the segmentation image 44 may be superimposed on the endoscopic image 40. In this case, for example, the segmentation image 44 may be superimposed on the endoscopic image 40 using alpha blending. Alternatively, the outer contour of the segmentation image 44 may be superimposed on the endoscopic image 40. In this case, too, the outer contour of the segmentation image 44 may be superimposed on the endoscopic image 40 using alpha blending.

[0182] In the first embodiment, an example was given in which the minimum size 112A was calculated using the line segment 120 defined by the circumscribing rectangular frame 122, but the technology of the present disclosure is not limited to this. The line segment 120 may be set according to instructions received by the reception device 64. The same applies to the line segment 130 used when calculating the maximum size 112B. In this way, the size 112C of the range specified by the doctor 16 is measured and displayed on the screen 35. The same applies to the second embodiment.

[0183] In the first embodiment, the width information 136 is displayed on the screen 35 in text format. However, this is merely an example. For example, the second delimited region 106 may be displayed in the probability map 45 in a manner that allows it to be distinguished from the segmentation image 44, thereby allowing the doctor 16 to visually recognize the width information 136. Alternatively, the second delimited region 106 may be displayed in the endoscopic image 40. In this case, the second delimited region 106 may be displayed in a manner that allows it to be distinguished from the lesion 42 shown in the endoscopic image 40. In this case, for example, the second delimited region 106 may be superimposed on the endoscopic image 40 using alpha blending. The same applies to the second embodiment.

[0184] In the first embodiment, an example was given in which the size 112C is displayed in the second display area 38, but this is merely one example, and the size 112C may be displayed in a pop-up format from within the second display area 38 to outside the second display area 38, or the size 112C may be displayed in a location other than the second display area 38 on the screen 35. Furthermore, various information such as the type of lesion 42, the kind of lesion 42, and the width of the lesion 42 may also be displayed in the first display area 36 and / or the second display area 38, or may be displayed on a screen other than the screen 35. The same can be said for the second embodiment.

[0185] In the above embodiments, an example was given in which the size of one lesion 42 was measured and the measurement result was presented to the doctor 16, but if multiple lesions 42 are shown in the endoscopic image 40, medical support processing may be performed for each of the multiple lesions 42. In this case, a mark or the like may be added to the image area of ​​the lesion 42 corresponding to the information displayed on the screen 35 so that it is possible to identify which lesion 42 information (size, type, kind, and width) is displayed on the screen 35.

[0186] In the first embodiment, an example was given in which the size 112C was measured frame by frame, but this is merely one example, and statistical values ​​(e.g., average, median, or mode) of the size 112C measured for multiple frames of endoscopic images 40 in time series may be displayed in the same display manner as in the first embodiment. For example, the size 112C may be measured when the amount of displacement of the position of the lesion 42 between multiple frames is less than a threshold, and the measured size 112C itself or statistical values ​​of the size 112C measured for multiple frames of endoscopic images 40 in time series may be displayed on the screen 35. The same can be said for the second embodiment.

[0187] In the above-described embodiments, an example was described in which the position of the lesion 42 was recognized for each endoscopic image 40 using an AI segmentation method, but the technology of the present disclosure is not limited to this. For example, the position of the lesion 42 may be recognized for each endoscopic image 40 using an AI bounding box method.

[0188] In this case, the amount of change in the bounding box is calculated by the processor 82, and a decision as to whether or not to measure the size of the lesion 42 is made based on the amount of change in the bounding box in the same manner as in the above embodiment.

[0189] For example, the amount of change in the bounding box refers to the amount of change in the position of the lesion 42. The amount of change in the position of the lesion 42 may be the amount of change in the position of the lesion 42 between adjacent endoscopic images 40 in time series, or may be the amount of change in the position of the lesion 42 between three or more frames of endoscopic images 40 in time series (e.g., a statistical value such as the average, median, mode, or maximum value of the amount of change between three or more frames of endoscopic images 40 in time series). Alternatively, it may be the amount of change in the position of the lesion 42 between multiple frames in time series spaced apart by one or more frames.

[0190] In each of the above embodiments, an AI-based object recognition process is exemplified as the recognition process 96, but the technology disclosed herein is not limited to this, and the lesion 42 shown in the endoscopic image 40 may be recognized by the recognition unit 82A by performing a non-AI-based object recognition process (e.g., template matching, etc.).

[0191] In the first embodiment, the display device 14 is exemplified as an output destination of the size 112C, but the technology of the present disclosure is not limited to this, and the output destination of the size 112 may be a destination other than the display device 14. As an example, as shown in Fig. 20, output destinations of the size 112C include an audio playback device 148, a printer 150, and / or an electronic medical record management device 152.

[0192] The size 112C may be output as sound by an audio playback device 148. The size 112C may also be printed as text on a medium (e.g., paper) by a printer 150. The size 112C may also be stored in an electronic medical record 154 managed by an electronic medical record management device 152. The same applies to the second embodiment.

[0193] In the first embodiment, an example was described in which arithmetic expressions 124, 134, and 135 were used to calculate size 112C. However, the technology of the present disclosure is not limited to this. Size 112C may be measured by performing AI processing on endoscopic image 40. In this case, for example, a trained model may be used that outputs size 112C of lesion 42 when endoscopic image 40 including lesion 42 is input. When creating the trained model, deep learning may be performed on a neural network using training data in which annotations indicating the size of the lesion are added as ground truth data for lesions shown in images used as example data. The same applies to the second embodiment.

[0194] In the above-described embodiments, the recognition unit 82A performs the recognition process 96 on the endoscopic image 40 acquired from the camera 52 to recognize the characteristics of the lesion 42 shown in the endoscopic image 40. However, the technology of the present disclosure is not limited to this. For example, the characteristics of the lesion 42 shown in the endoscopic image 40 may be provided to the processor 82 by the physician 16 or the like via the reception device 64 or the like, or may be acquired by the processor 82 from an external device (e.g., a server, a personal computer, and / or a tablet terminal). In these cases, the size of the lesion 42 may be measured by the measurement unit 82B in accordance with the characteristics of the lesion 42 in the same manner as in the above-described embodiments.

[0195] In the above embodiments, an example of deriving the distance information 114 using the distance derivation model 94 has been described, but the technology of the present disclosure is not limited to this. For example, other methods of deriving the distance information 114 using an AI method include a method of combining segmentation and depth estimation (for example, regression learning that provides distance information 114 for the entire image (for example, all pixels that make up the image), or unsupervised learning that learns the distance for the entire image in an unsupervised manner).

[0196] In the above embodiments, an example was given in which the distance from the camera 52 to the intestinal wall 24 was derived using an AI system, but the distance from the camera 52 to the intestinal wall 24 may also be measured. In this case, for example, a distance measuring sensor may be provided at the tip portion 50 (see FIG. 2 ) so that the distance from the camera 52 to the intestinal wall 24 is measured by the distance measuring sensor.

[0197] In each of the above embodiments, an endoscopic image 40 is exemplified, but the technology of the present disclosure is not limited to this, and the technology of the present disclosure can also be applied to medical images other than the endoscopic image 40 (for example, images obtained by a modality other than the endoscope 12, such as a radiological image or an ultrasound image).

[0198] In each of the above embodiments, an example of measuring the size 112 of a lesion 42 shown in a moving image is given, but this is merely one example, and the technology of the present disclosure can also be applied to frame-by-frame or still images showing the lesion 42.

[0199] In the above embodiments, an example was given in which distance information 114 extracted from distance image 116 was input to arithmetic expressions 124 and 134. However, the technology of the present disclosure is not limited to this. For example, without generating distance image 116, distance information 114 corresponding to a position identified from position identification information 98 may be extracted from all distance information 114 output from distance derivation model 94, and the extracted distance information 114 may be input to arithmetic expressions 124 and 134.

[0200] In each of the above embodiments, an example has been described in which medical support processing is performed by the processor 82 of the computer 78 included in the endoscope 12, but the technology of the present disclosure is not limited to this, and the device that performs medical support processing may be provided external to the endoscope 12. Examples of devices that may be provided external to the endoscope 12 include at least one server and / or at least one personal computer that are communicatively connected to the endoscope 12. Furthermore, medical support processing may be performed in a distributed manner by multiple devices.

[0201] In the above embodiments, an example has been described in which the medical support programs 90 and 90A (hereinafter referred to as "medical support programs" without reference numerals) are stored in the NVM 86, but the technology of the present disclosure is not limited to this. For example, the medical support programs may be stored in a portable, computer-readable, non-transitory storage medium such as an SSD or USB memory. The medical support programs stored in the non-transitory storage medium are installed in the computer 78 of the endoscope 12. The processor 82 executes medical support processing in accordance with the medical support programs.

[0202] Alternatively, the medical support program may be stored in a storage device such as another computer or server connected to the endoscope 12 via a network, and the medical support program may be downloaded and installed on the computer 78 in response to a request from the endoscope 12.

[0203] It is not necessary to store all of the medical support program in a storage device such as another computer or server device connected to the endoscope 12, or to store all of the medical support program in the NVM 86; only a portion of the medical support program may be stored.

[0204] The hardware resources that execute the medical support processing can be various processors, as listed below. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource that executes medical support processing by executing software, i.e., a program. Examples of processors include dedicated electrical circuits, such as FPGAs, PLDs, or ASICs, which are processors with a circuit configuration specifically designed to execute specific processing. Each processor has built-in or connected memory, and executes medical support processing by using the memory.

[0205] The hardware resource for executing the medical support processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource for executing the medical support processing may be a single processor.

[0206] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes medical support processing. Second, there is a system that uses a processor that realizes the functions of the entire system, including multiple hardware resources that execute medical support processing, on a single IC chip, as typified by SoC. In this way, medical support processing is realized using one or more of the various processors described above as hardware resources.

[0207] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The above medical support process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the process.

[0208] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0209] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0210] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. a processor; The processor: Using a medical image, a region to be observed that appears in the medical image is recognized; measuring a size according to the characteristics of the observation region based on the medical image; Output the size, The characteristics include the shape of the observation region, the type of the observation region, the category of the observation region, the clarity of the contour of the observation region, and / or the overlap of the observation region with a surrounding region. Medical support equipment.

2. The processor recognizes the characteristic based on the medical image. The medical support device according to claim 1 .

3. The size is the long side, short side, radius, and / or diameter of the observation area. The medical support device according to claim 1 .

4. The observation target area is recognized using an AI method, The size is measured based on a probability map obtained from the AI. The medical support device according to claim 1 .

5. The size is measured based on a closed region obtained by dividing the probability map according to a threshold. The medical support device according to claim 4.

6. The size is measured based on a plurality of partitioned regions obtained by partitioning the probability map according to a plurality of thresholds. The medical support device according to claim 4.

7. The size ranges, The width is determined based on the plurality of partitioned regions. The medical support device according to claim 6.

8. the lower limit of the width is measured based on a first divided area that is narrowest among the plurality of divided areas; The upper limit of the width is measured based on a second divided area that is located outside the first divided area among the plurality of divided areas. The medical support device according to claim 7.

9. The processor measures a plurality of first sizes of the region of interest based on the medical image; The size is a representative value of the plurality of first sizes. The medical support device according to claim 1 .

10. The representative value includes a maximum value, a minimum value, a mean value, a median value, and / or a variance value. The medical support device according to claim 9.

11. the characteristics include an overlap between the observation region and a surrounding region; The size is the size of the observation area including the overlap and / or the size of the observation area excluding the overlap. The medical support device according to claim 1 .

12. The output of the size is realized by displaying the size on the screen. The medical support device according to claim 1 .

13. The medical image is an endoscopic image obtained by imaging using an endoscope. The medical support device according to claim 1 .

14. The observation target area is a lesion. The medical support device according to claim 1 .

15. A medical support device according to any one of claims 1 to 14; a module that is inserted into a body including the observation target region and captures the observation target region to obtain the medical image. Endoscope.

16. Using a medical image, recognizing an observation target area shown in the medical image; measuring a size according to characteristics of the observation area based on the medical image; and outputting the size. Medical support methods.

17. A program for causing a computer to execute medical support processing, The medical support process includes: Using a medical image, recognizing an observation target area shown in the medical image; measuring a size according to characteristics of the observation area based on the medical image; and outputting the size. program.