Image processing device, endoscope, image processing method, and program

Abstract

An image processing apparatus includes a processor. The processor performs: a process for recognizing, from a medical image reflecting an existing observation target region, a position of the observation target region within the medical image; judging whether size output is performed or not according to the position; and measuring the size on the basis of the medical image when it is determined that output is performed.

Description

Technical Field

[0001] The technology of the present invention relates to an image processing device, an endoscope, an image processing method, and a program. Background Art

[0002] Japanese Patent Publication No. 2022-535873 discloses a method for processing colon images and videos. The method described in Japanese Patent Publication No. 2022-535873 generates commands that prompt a graphical user interface (GUI) for dynamically tracking at least one polyp within multiple endoscopic images of a patient's colon.

[0003] The method described in Japanese Patent Application Publication No. 2022-535873 includes the step of tracking the position of a polyp delineation region. Furthermore, the method described in Japanese Patent Application Publication No. 2022-535873 includes repeatedly performing the following steps for a plurality of endoscopic images: when the region is located outside each endoscopic image, calculating a vector from within each endoscopic image to the position of the region outside each endoscopic image; creating a supplemented endoscopic image by supplementing each endoscopic image with the display of the vector; and generating a command for displaying the supplemented endoscopic image in a GUI.

[0004] Japanese Patent Gazette No. 2020-093076 discloses a medical image processing device comprising: an acquisition unit for acquiring a tomographic image of an eye to be examined; and a first processing unit for performing a first detection process for detecting at least one of a plurality of retinal layers in the acquired tomographic image using a learned model, wherein the learned model learns data of at least one of a plurality of retinal layers displayed in the tomographic image of the eye to be examined.

[0005] An endoscope system is disclosed in International Publication No. 2020 / 110214, which comprises: an image input unit for sequentially inputting a plurality of observation images obtained by photographing a subject using an endoscope; a lesion detection unit for detecting the observation object of the endoscope, i.e., the lesion, from the observation image; a missed risk analysis unit for judging the risk of the operator missing the lesion, i.e., the degree of missed risk, based on the observation image; a notification control unit for controlling the notification mechanism and notification method of lesion detection according to the degree of missed risk; and a notification unit for notifying the operator of the detection of the lesion under the control of the notification control unit.

[0006] Furthermore, in the endoscope system described in International Publication No. 2020 / 110214, the omission risk analysis unit includes a lesion analysis unit that analyzes the omission risk based on the state of the lesion. Furthermore, the lesion analysis unit includes a lesion size analysis unit that estimates the size of the lesion itself. Furthermore, the lesion analysis unit includes a lesion position analysis unit that analyzes the position of the lesion in the observed image. Summary of the Invention

[0007] One embodiment of the technology according to the present invention provides an image processing device, an endoscope, an image processing method, and a program that enable a user or the like to accurately grasp the size of an observation target region appearing in a medical image.

[0008] Means for solving technical problems

[0009] The first method involved in the technology of the present invention is an image processing device, which includes a processor, wherein the processor performs the following processing: identifying the position of the observation object area within the medical image based on the medical image in which the observation object area is reflected; determining whether to output the size of the observation object area based on the position; and outputting the size when it is determined that the output is to be performed.

[0010] The second method involved in the technology of the present invention is the image processing device involved in the first method, wherein the medical image is a plurality of frames in a time series, and the processor performs the following processing: identifying a position in each of the plurality of frames; and using the change in position between the plurality of frames to determine whether to output.

[0011] A third aspect according to the technology of the present invention is the image processing device according to the second aspect, wherein the amount of change in position between the plurality of frames is defined based on a distance between the positions of the plurality of frames.

[0012] A fourth aspect of the technology of the present invention is the image processing device according to the second aspect or the third aspect, wherein the amount of position change between the plurality of frames is specified based on a degree of overlap of observation target areas between the plurality of frames.

[0013] A fifth aspect according to the technology of the present invention is the image processing device according to any one of the first to fourth aspects, wherein the processor measures the size by performing AI-based processing on the medical image.

[0014] The sixth embodiment of the technology of the present invention is an image processing device according to any one of the first to fourth embodiments, wherein the processor performs the following processing: deriving the distance from the observation position to the observation object area by processing the medical image using AI; and measuring the size based on the distance and the number of pixels of the range of the measurement object within the observation object area.

[0015] The seventh method involved in the technology of the present invention is an image processing device involved in any one of the first to sixth methods, wherein the processor performs the following processing: when the position exists in the first area within the medical image, it is determined that output is to be performed; and when the position exists in the second area further outside than the first area within the medical image, it is determined that no output is to be performed.

[0016] An eighth aspect of the present invention is the image processing device according to any one of the first to sixth aspects, wherein the medical image is a plurality of frames in time series, and the processor performs the following processing:

[0017] Identifying a position in each of a plurality of frames; and determining whether to output based on the amount of position change between the plurality of frames and whether the position exists in a first region within the medical image or a second region within the medical image that is further outside the first region.

[0018] The ninth method involved in the technology of the present invention is an image processing device involved in any one of the first to eighth methods, wherein the medical image is a plurality of frames in a time series, and the processor performs the following processing: identifying the position in each of the plurality of frames using an AI-based bounding box method; and using the change in the bounding box to determine whether to output.

[0019] The 10th method involved in the technology of the present invention is an image processing device involved in any one of the 1st to 8th methods, wherein the medical image is a plurality of frames in a time series, and the processor performs the following processing: identifying the position in each of the plurality of frames in an AI-based segmentation method; and using the change amount of the segmented area to determine whether to perform measurement.

[0020] An eleventh aspect according to the technology of the present invention is the image processing apparatus according to any one of the first to tenth aspects, wherein the processor determines whether to output based on whether the position is at an edge of the medical image.

[0021] The 12th method involved in the technology of the present invention is an image processing device involved in any one of the 1st to 7th methods, wherein the medical image is a plurality of frames in a time series, and the processor determines whether to output based on the position within the first frame selected according to the instruction issued among the multiple frames and the position within at least one second frame obtained earlier than the first frame among the multiple frames.

[0022] A thirteenth aspect according to the technology of the present invention is the image processing apparatus according to any one of the first to twelfth aspects, wherein the medical image is a moving image.

[0023] A fourteenth aspect according to the technology of the present invention is the image processing device according to any one of the first to twelfth aspects, wherein the processor outputs the size when determining to output.

[0024] A fifteenth aspect according to the technology of the present invention is the image processing device according to the fourteenth aspect, wherein the size output is achieved by displaying the size on the first screen.

[0025] A sixteenth aspect according to the technology of the present invention is the image processing device according to any one of the first to fifteenth aspects, wherein the processor outputs a past result of the measured size when determining not to output.

[0026] A seventeenth aspect according to the technology of the present invention is the image processing device according to the sixteenth aspect, wherein the output of the past results is achieved by displaying the past results on the second screen.

[0027] The 18th method involved in the technology of the present invention is the image processing device involved in the 17th method, wherein the processor performs the following processing: when it is determined that output is to be performed, the current result of the measured size is displayed on the second screen; and when it is determined that output is not to be performed and when it is determined that output is to be performed, the past results and the current results are displayed in a distinguishable manner on the second screen.

[0028] A nineteenth aspect of the present invention is the image processing device according to any one of the first to eighteenth aspects, wherein, when it is determined that output is not to be performed, the processor outputs non-output specifying information that specifies non-output.

[0029] A 20th aspect according to the technology of the present invention is the image processing device according to the 19th aspect, wherein the output of the non-output specific information is realized by displaying the non-output specific information on the third screen.

[0030] A 21st aspect according to the technology of the present invention is the image processing device according to any one of the 1st to 20th aspects, wherein the medical image is an endoscopic image obtained by imaging with an endoscope.

[0031] A 22nd aspect according to the technology of the present invention is the image processing device according to any one of the 1st to 21st aspects, wherein the observation target region is a lesion.

[0032] A 23rd aspect of the technology of the present invention is an endoscope comprising: the image processing device according to any one of the first to 22nd aspects; and a module that acquires a medical image by being inserted into a body including an observation target area and photographing the observation target area.

[0033] The 24th method involved in the technology of the present invention is an image processing method, which includes the following steps: identifying the position of the observation object area in the medical image based on the medical image mapping the observation object area; determining whether to output the size of the observation object area based on the position; and outputting the size when it is determined to be output.

[0034] The 25th method involved in the technology of the present invention is a program, which is used to enable a computer to perform processing including the following steps: identifying the position of the observation object area within the medical image based on the medical image in which the observation object area is mapped; determining whether to output the size of the observation object area based on the position; and outputting the size when it is determined that output is to be performed. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a conceptual diagram showing an example of how to use an endoscope.

[0036] Figure 2 This is a conceptual diagram showing an example of the overall structure of an endoscope.

[0037] Figure 3 This is a block diagram showing an example of the hardware configuration of the electrical system of the endoscope.

[0038] Figure 4 This is a block diagram showing an example of the main functions of a processor included in the endoscope and an example of information stored in the NVM.

[0039] Figure 5 This is a conceptual diagram showing an example of the processing contents of the recognition unit and the control unit.

[0040] Figure 6 This is a conceptual diagram showing an example of the processing contents of the recognition unit and the determination unit.

[0041] Figure 7 This is a conceptual diagram showing an example of the processing content of the determination unit when a lesion is included in the edge portion.

[0042] Figure 8 This is a conceptual diagram showing an example of the processing content of the measurement unit.

[0043] Figure 9 This is a conceptual diagram showing an example of a mode in which an endoscopic image is displayed on a first screen and dimensions are displayed on a second screen.

[0044] Figure 10 This is a conceptual diagram showing an example of a mode in which an endoscopic image is displayed on a first screen and a mode in which no specific information is output is displayed on a second screen.

[0045] Figure 11 This is a flowchart showing an example of a medical support processing flow.

[0046] Figure 12 This is a conceptual diagram showing a first modified example of the processing contents of the recognition unit and the determination unit.

[0047] Figure 13 This is a flowchart showing a modified example of the medical support processing flow.

[0048] Figure 14 This is a conceptual diagram showing an example of a mode in which an endoscopic image is displayed on a first screen and past results are displayed on a second screen.

[0049] Figure 15 This is a conceptual diagram showing a second modified example of the processing contents of the recognition unit and the determination unit.

[0050] Figure 16 This is a conceptual diagram showing an example of a size output terminal. DETAILED DESCRIPTION

[0051] Hereinafter, an example of an embodiment of an image processing device, an endoscope, an image processing method, and a program according to the technology of the present invention will be described with reference to the accompanying drawings.

[0052] First, the terms used in the following description are explained.

[0053] CPU stands for "Central Processing Unit." GPU stands for "Graphics Processing Unit." RAM stands for "Random Access Memory." NVM stands for "Non-volatile Memory." EEPROM stands for "Electrically Erasable Programmable Read-Only Memory." ASIC stands for "Application Specific Integrated Circuit." PLD stands for "Programmable Logic Device." FPGA stands for "Field-Programmable Gate Array." SoC stands for "System-on-a-Chip." SSD stands for "Solid State Drive." USB stands for "Universal Serial Bus." HDD stands for "Hard Disk Drive." EL stands for "Electro-Luminescence." CMOS stands for "Complementary Metal Oxide Semiconductor." CCD stands for "Charge Coupled Device." AI stands for "Artificial Intelligence." BLI stands for "Blue Light Imaging." LCI stands for "Linked Color Imaging." I / F stands for "Interface." IoU stands for "Intersection over Union." FIFO stands for "First In First Out."

[0054] As an example, Figure 1 As shown, the endoscope system 10 includes an endoscope 12 and a display device 14. The endoscope 12 is used by a doctor 16 during endoscopic examination. The endoscopic examination is assisted by staff such as a nurse 17. In this embodiment, the endoscope 12 is an example of an "endoscope" according to the present invention.

[0055] The endoscope 12 is communicatively connected to a communication device (not shown), and information obtained by the endoscope 12 is transmitted to the communication device. Examples of the communication device include a server and / or a client terminal (e.g., a personal computer and / or a tablet terminal) that manages various information such as electronic medical records. The communication device receives information transmitted from the endoscope 12 and performs processing using the received information (e.g., processing stored in the electronic medical record, etc.).

[0056] The endoscope 12 includes an endoscope body 18. The endoscope 12 is a device for diagnosing and treating a large intestine 22 contained in a subject 20 (eg, a patient) using the endoscope body 18. In this embodiment, the large intestine 22 is an object to be observed by the doctor 16.

[0057] The endoscope body 18 is inserted into the large intestine 22 of the subject 20. The endoscope 12 images the interior of the large intestine 22 of the subject 20 while the endoscope body 18 is inserted into the large intestine 22 of the subject 20 and performs various medical procedures on the large intestine 22 as needed.

[0058] The endoscope 12 captures and outputs images showing the internal state of the subject 20 by imaging the inside of the large intestine 22. In this embodiment, the endoscope 12 is an endoscope having an optical imaging function for capturing light reflected from the intestinal wall 24 of the large intestine 22 by irradiating light 26 into the large intestine 22.

[0059] In addition, although the endoscope inspection of the large intestine 22 is illustrated here, this is only an example, and the technology of the present invention is also applicable to the endoscope inspection of a luminal organ such as the esophagus, stomach, duodenum, or trachea.

[0060] The endoscope 12 includes a control device 28, a light source device 30, and an image processing device 32. The control device 28, light source device 30, and image processing device 32 are mounted on a carriage 34. The carriage 34 has a plurality of stages arranged in the vertical direction, with the image processing device 32, the control device 28, and the light source device 30 being mounted from the lower stage to the upper stage. Furthermore, the display device 14 is mounted on the uppermost stage of the carriage 34.

[0061] The control device 28 controls the entire endoscope 12. The image processing device 32 performs various image processing on images obtained by imaging the intestinal wall 24 with the endoscope body 18 under the control of the control device 28.

[0062] The display device 14 displays various information including images. Examples of the display device 14 include a liquid crystal display and an EL display. Alternatively, a tablet terminal with a display may be used in place of or in combination with the display device 14.

[0063] A plurality of screens are displayed in an array on the display device 14. Figure 1 In the example shown, a first screen 36 and a second screen 38 are shown as examples of a plurality of screens.

[0064] An endoscopic image 40 is displayed on the first screen 36. The endoscopic image 40 is a circular image. That is, the endoscopic image 40 is an image obtained by the endoscope body 18 capturing the intestinal wall 24 in the large intestine 22 of the subject 20. Figure 1 In the example shown, an image showing the intestinal wall 24 is shown as an example of the endoscopic image 40. The intestinal wall 24 shown in the endoscopic image 40 includes a lesion 42. Figure 1 In the example shown, the lesion 42, which is the observation target area focused on by the doctor 16, is also shown in the endoscopic image 40. There are many types of lesions 42, and examples of the types of lesions 42 include tumorous polyps and non-tumorous polyps.

[0065] In this embodiment, endoscopic image 40 is an example of a "medical image," "frame," and "endoscopic image" within the scope of the present invention. Furthermore, in this embodiment, lesion 42 is an example of an "observation target region" and a "lesion" within the scope of the present invention. While lesion 42 is shown here as an example, the present invention is not limited thereto. The observation target region may also be an organ (e.g., the duodenal papilla), a marked region, or a treated region (e.g., a region with traces of polyp removal).

[0066] A moving image is displayed on the first screen 36. The endoscopic image 40 displayed on the first screen 36 is a single frame included in a moving image composed of multiple frames in a time series. That is, multiple frames of the endoscopic image 40 are displayed on the first screen 36 at a predetermined frame rate (e.g., 30 frames / second or 60 frames / second).

[0067] An example of a dynamic image displayed on the first screen 36 is a live preview dynamic image. The live preview method is merely one example, and as illustrated by a post-view dynamic image, a dynamic image may be temporarily stored in a memory or the like before being displayed. Furthermore, each frame included in a recording dynamic image stored in a memory or the like may be played back and displayed as the endoscopic image 40 on the first screen 36.

[0068] The second screen 38 is a rectangular screen that is smaller than the first screen 36. Figure 1In the example shown, the second screen 38 is displayed overlappingly on the lower right side of the main view of the first screen 36. Here, an overlapping display is illustrated, but this is only an example, and an embedded display may also be used. Furthermore, the display position of the second screen 38 may be any position as long as it is within the screen of the display device 14, and it is preferably displayed at a position where it can be compared with the endoscopic image 40. A position-specific image 44 is displayed on the second screen 38. The position-specific image 44 is an image corresponding to the endoscopic image 40, and is an image that a user (e.g., a doctor 16, etc.) refers to when determining the position of the lesion 42 within the endoscopic image 40.

[0069] The position identification image 44 includes an outer frame 44A, a target mark 44B, and a lesion image 44C. The outer frame 44A is a circular frame obtained by reducing the annular outline of the endoscopic image 40 , with the upper and lower parts of the circular frame cut off by the upper and lower sides of the second screen 38 .

[0070] The target mark 44B is a mark that intersects in a cross shape at the center of the display area of ​​the position identification image 44. The intersection point of the target mark 44B corresponds to the center point of the endoscopic image 40.

[0071] The lesion image 44C is an image corresponding to the lesion 42 within the endoscopic image 40 and is displayed in a display mode appropriate to the size, shape, and type of the lesion 42. An example of the lesion image 44C is an image representing the segmented region of the lesion 42 identified by the AI-based segmentation method for each endoscopic image 40 or an image having a similar relationship to the segmented region.

[0072] As an example, Figure 2 As shown, the endoscope body 18 includes an operating portion 46 and an insertion portion 48. The insertion portion 48 is partially bent by being operated by the operating portion 46. The insertion portion 48 is bent according to the doctor 16 (refer to Figure 1 ) The operation of the operating unit 46 is performed according to the large intestine 22 (reference Figure 1 ) is inserted into the large intestine 22 while being bent into the shape of the uterus.

[0073] A camera 52, an illumination device 54, and a treatment instrument opening 56 are provided at the distal end portion 50 of the insertion portion 48. The camera 52 and the illumination device 54 are provided on the distal end surface 50A of the distal end portion 50. While the camera 52 and the illumination device 54 are provided on the distal end surface 50A of the distal end portion 50, this is merely an example. The camera 52 and the illumination device 54 may also be provided on the side surface of the distal end portion 50, thereby configuring the endoscope 12 as a side view mirror.

[0074] The camera 52 is a device that obtains the endoscopic image 40 as a medical image by photographing the interior of the subject 20 (e.g., the large intestine 22). As an example of the camera 52, a CMOS camera can be cited. However, this is only one example, and other types of cameras such as a CCD camera may also be used. The camera 52 is an example of a "module" involved in the technology of the present invention.

[0075] The lighting device 54 has lighting windows 54A and 54B. The lighting device 54 irradiates the light 26 (see Figure 1 ). Examples of the type of light 26 emitted from the lighting device 54 include visible light (e.g., white light) and non-visible light (e.g., near-infrared light). Furthermore, the lighting device 54 emits special light through the lighting windows 54A and 54B. Examples of special light include light for BLI and / or light for LCI. The camera 52 optically captures the interior of the large intestine 22 while the lighting device 54 is irradiating the large intestine 22 with light 26.

[0076] The treatment instrument opening 56 is an opening for allowing a treatment instrument 58 to protrude from the distal end portion 50. The treatment instrument opening 56 can also be used as a suction port for sucking blood and body waste, and a delivery port for delivering fluid.

[0077] The operation portion 46 is formed with a treatment instrument insertion port 60, and the treatment instrument 58 is inserted into the insertion portion 48 through the treatment instrument insertion port 60. The treatment instrument 58 passes through the insertion portion 48 and protrudes from the treatment instrument opening 56 to the outside. Figure 2 In the example shown, a puncture needle is shown as the treatment instrument 58 protruding from the treatment instrument opening 56. While a puncture needle is shown as the treatment instrument 58, this is merely an example, and the treatment instrument 58 may also be a grasping forceps, a papillotomy knife, a snare, a catheter, a guide wire, a cannula, and / or a puncture needle with a guide sheath.

[0078] The endoscope body 18 is connected to the control device 28 and the light source device 30 via a universal cord 62. The control device 28 is connected to an image processing device 32 and a receiving device 64. The image processing device 32 is also connected to the display device 14. That is, the control device 28 is connected to the display device 14 via the image processing device 32.

[0079] In addition, the image processing device 32 is shown here as an example of an external device used to expand the functions performed by the control device 28. Therefore, the control device 28 and the display device 14 are indirectly connected via the image processing device 32. However, this is merely an example. For example, the display device 14 may be directly connected to the control device 28. In this case, for example, the functions of the image processing device 32 can be implemented in the control device 28, or the control device 28 can implement a function that causes a server (not shown) to perform the same processing as that performed by the image processing device 32 (for example, medical support processing described later) and receive and use the processing results based on the server.

[0080] The receiving device 64 receives instructions from the doctor 16 and outputs the received instructions as electrical signals to the control device 28. Examples of the receiving device 64 include a keyboard, a mouse, a touch panel, a foot switch, a microphone, and / or a remote control device.

[0081] The control device 28 controls the light source device 30 , or transmits and receives various signals with the camera 52 , or transmits and receives various signals with the image processing device 32 .

[0082] The light source device 30 emits light under the control of the control device 28 and supplies light to the lighting device 54. The lighting device 54 has a built-in light guide, and the light supplied from the light source device 30 is irradiated from the lighting windows 54A and 54B via the light guide. The control device 28 causes the camera 52 to capture an image, and the camera 52 acquires the endoscopic image 40 (see FIG. 4 ). Figure 1 ) and outputs it to a predetermined output terminal (for example, the image processing device 32).

[0083] The image processing device 32 performs various image processing on the endoscopic image 40 input from the control device 28. The image processing device 32 outputs the endoscopic image 40 subjected to the various image processing to a predetermined output destination (for example, the display device 14).

[0084] In addition, although the example described here is a method in which the endoscopic image 40 output from the control device 28 is output to the display device 14 via the image processing device 32, this is merely an example. For example, a method may also be used in which the control device 28 is connected to the display device 14, and the endoscopic image 40 subjected to image processing by the image processing device 32 is displayed on the display device 14 via the control device 28.

[0085] As an example, Figure 3 As shown, the control device 28 includes a computer 66 , a bus 68 , and an external I / F 70 . The computer 66 includes a processor 72 , a RAM 74 , and an NVM 76 . The processor 72 , the RAM 74 , the NVM 76 , and the external I / F 70 are connected to the bus 68 .

[0086] For example, the processor 72 includes at least one CPU and at least one GPU, and controls the entire control device 28. The GPU operates under the control of the CPU and is responsible for executing various processes of the graphics system and performing calculations using neural networks. In addition, the processor 72 may be one or more CPUs with integrated GPU functions, or one or more CPUs without integrated GPU functions. Figure 3 In the example shown, a mode in which one processor 72 is mounted in the computer 66 is shown. However, this is only an example, and a plurality of processors 72 may be mounted in the computer 66 .

[0087] RAM 74 is a memory for temporarily storing information and is used as working memory by processor 72. NVM 76 is a nonvolatile storage device that stores various programs and parameters. An example of NVM 76 is a flash memory (e.g., EEPROM and / or SSD). Flash memory is merely an example, and other nonvolatile storage devices such as HDDs may be used, or a combination of two or more nonvolatile storage devices may be used.

[0088] The external I / F 70 is responsible for transmitting and receiving various information between one or more devices (hereinafter also referred to as "first external devices") existing outside the control device 28 and the processor 72. An example of the external I / F 70 is a USB interface.

[0089] The camera 52 is connected to the external I / F 70 as one of the first external devices. The external I / F 70 is responsible for sending and receiving various information between the camera 52 and the processor 72. The processor 72 controls the camera 52 via the external I / F 70. In addition, the processor 72 obtains the image of the large intestine 22 (see FIG. 2 ) captured by the camera 52 via the external I / F 70. Figure 1 ) and the endoscopic image 40 (reference Figure 1 ).

[0090] The light source device 30 is connected to the external I / F 70 as one of the first external devices. The external I / F 70 is responsible for transmitting and receiving various information between the light source device 30 and the processor 72. Under the control of the processor 72, the light source device 30 supplies light to the lighting device 54. The lighting device 54 irradiates the light supplied from the light source device 30.

[0091] The receiving device 64 as one of the first external devices is connected to the external I / F 70 . The processor 72 acquires an instruction received by the receiving device 64 via the external I / F 70 and executes a process corresponding to the acquired instruction.

[0092] Image processing device 32 includes a computer 78 and an external I / F 80. Computer 78 includes a processor 82, RAM 84, and NVM 86. Processor 82, RAM 84, NVM 86, and external I / F 80 are connected to bus 88. In this embodiment, image processing device 32 is an example of an "image processing device" involved in the technology of the present invention, computer 78 is an example of a "computer" involved in the technology of the present invention, and processor 82 is an example of a "processor" involved in the technology of the present invention.

[0093] In addition, the hardware structure of the computer 78 (ie, the processor 82, RAM 84, and NVM 86) is substantially the same as the hardware structure of the computer 66, and thus the description of the hardware structure of the computer 78 is omitted here.

[0094] External I / F 80 is responsible for transmitting and receiving various information between one or more devices (hereinafter also referred to as "second external devices") existing outside image processing device 32 and processor 82. An example of external I / F 80 is a USB interface.

[0095] The control device 28 is connected to the external I / F 80 as one of the second external devices. Figure 3 In the example shown, the external I / F 70 of the control device 28 is connected to the external I / F 80. The external I / F 80 is responsible for sending and receiving various information between the processor 82 of the image processing device 32 and the processor 72 of the control device 28. For example, the processor 82 obtains the endoscopic image 40 (see FIG. 4 ) from the processor 72 of the control device 28 via the external I / Fs 70 and 80. Figure 1 ) and perform various image processing on the acquired endoscopic image 40.

[0096] The display device 14 is connected as one of the second external devices to the external I / F 80. The processor 82 controls the display device 14 via the external I / F 80, thereby causing the display device 14 to display various information (eg, the endoscopic image 40 subjected to various image processing).

[0097] During endoscopic examination, the doctor 16 determines whether medical treatment is required for the lesion 42 while viewing the endoscopic image 40 on the display device 14, and performs the medical treatment on the lesion 42 if necessary. The size of the lesion 42 is an important factor in determining whether medical treatment is required.

[0098] In recent years, with the development of machine learning, it has become possible to detect and identify lesions 42 using AI from endoscopic images 40. By applying this technology, the size of lesions 42 can be measured from endoscopic images 40.

[0099] However, even if the size of the lesion 42 is measured, the measured size may vary greatly depending on the shooting state of the camera 52. For example, when the camera 52 shakes violently or the body moves violently, the endoscopic image 40 becomes blurred, making it difficult to accurately measure the size of the lesion 42 by using an image processing method using AI. In addition, due to the optical influence (e.g., aberration) of the objective lens of the camera 52, the edge of the endoscopic image 40 is deformed. Therefore, when the lesion 42 is located at the edge of the endoscopic image 40, the lesion 42 is mismeasured, and when the doctor 16 determines whether medical treatment is required based on the mismeasured size, it may result in medical treatment being performed when it is not required, or not being performed when it is required.

[0100] Therefore, in view of this situation, in this embodiment, as an example, Figure 4 As shown, medical auxiliary processing is performed by the processor 82 of the image processing device 32.

[0101] NVM 86 stores a medical assistance program 90. Medical assistance program 90 is an example of a "program" within the scope of the present invention. Processor 82 reads medical assistance program 90 from NVM 86 and executes it on RAM 84, thereby performing medical assistance processing. Medical assistance processing is achieved by the recognition unit 82A, determination unit 82B, measurement unit 82C, and control unit 82D operating in accordance with medical assistance program 90 executed by processor 82 on RAM 84.

[0102] NVM 86 stores a recognition model 92 and a distance derivation model 94. These models are examples of "AI" within the technology of the present invention. Details will be described later. Recognition model 92 is used by recognition unit 82A, while distance derivation model 94 is used by measurement unit 82C.

[0103] As an example, Figure 5 As shown, the recognition unit 82A and the control unit 82D acquire, from the camera 52 , the endoscopic image 40 generated by imaging at an imaging frame rate (eg, several tens of frames / second) in units of one frame.

[0104] The control unit 82D displays the endoscopic image 40 as a live preview image on the first screen 36. That is, each time the control unit 82D acquires an endoscopic image 40 from the camera 52 in units of one frame, it sequentially displays the acquired endoscopic image 40 on the first screen 36 at a display frame rate (e.g., several tens of frames per second).

[0105] The recognition unit 82A performs recognition processing 96 on the endoscopic image 40 acquired from the camera 52 to thereby recognize the position of the lesion 42 within the endoscopic image 40 (i.e., the position of the lesion 42 reflected in the endoscopic image 40). Each time the recognition unit 82A acquires an endoscopic image 40, the recognition processing 96 is performed on the acquired endoscopic image 40.

[0106] The recognition process 96 is an image recognition process of a segmentation method based on AI. Here, the recognition process 96 uses the recognition model 92.

[0107] Recognition model 92 is a segmented AI-based object detection model that is optimized by machine learning of a neural network using first training data. The first training data is a dataset containing multiple data (i.e., multiple frames of data) obtained by associating first example problem data with first correct answer data.

[0108] The first example data is an image corresponding to the endoscopic image 40. The first correct answer data is correct answer data (i.e., annotation) for the first example data. Here, as an example of the first correct answer data, an annotation that identifies a lesion that appears in the image used as the first example data is used.

[0109] The recognition unit 82A acquires the endoscopic image 40 from the camera 52 and inputs the acquired endoscopic image 40 into the recognition model 92. Thus, each time the endoscopic image 40 is input, the recognition model 92 identifies the position of the segmented region 100 identified in a segmented manner as the position of the lesion 42 reflected in the input endoscopic image 40, and outputs position identification information 98 that identifies the position of the segmented region 100. An example of the position identification information 98 is the coordinates that identify the segmented region 100 within the endoscopic image 40.

[0110] As an example, Figure 6 As shown, the determination unit 82B performs recognition processing 96 (see FIG. 1 ) on each endoscopic image 40 unit by the recognition unit 82A. Figure 5 ), the position specific information 98 is obtained from the recognition unit 82A. Then, the determination unit 82B determines whether to output the size of the lesion 42 based on the position specific information 98. In this embodiment, the size of the lesion 42 is output when the size of the lesion 42 is measured, and the size of the lesion 42 is not output when the size of the lesion 42 is not measured. Figure 6 In the example shown, the determination unit 82B determines whether to output the size of the lesion 42 by determining whether to measure the size of the lesion 42. That is, measuring the size of the lesion 42 means outputting the size of the lesion 42, and not measuring the size of the lesion 42 means not outputting the size of the lesion 42.

[0111] The determination unit 92B determines whether the position of the lesion 42 is within the edge portion 40A of the endoscopic image 40 or in an area outside the edge portion 40A. The edge portion 40A refers to an annular area having the outer edge of the endoscopic image 40 as the outer periphery and a circle offset from the outer edge of the endoscopic image 40 toward the center of the endoscopic image 40 by a length α as the inner periphery. The length α may be a fixed value predetermined as the length of a prescribed annular area, which may be affected by the aberration of the objective lens of the camera 52 and thus may not be able to accurately measure the size of the lesion 42. Alternatively, the length α may be a variable value that can be changed by a user or the like according to instructions and / or shooting conditions received by the receiving device 64. Here, the area outside the edge portion 40A is an example of the "first area" involved in the technology of the present invention, and the edge portion 40A is an example of the "second area" and "edge portion" involved in the technology of the present invention.

[0112] The determination unit 82B determines whether all of the segmented regions 100 are included in the edge portion 40A based on the position identification information 98 , thereby determining whether the position of the lesion 42 is within the edge portion 40A of the endoscopic image 40 .

[0113] Here, when the edge portion 40A does not include the entire segmented region 100 , it is determined that the position of the lesion 42 is not in the edge portion 40A of the endoscopic image 40 ; when the edge portion 40A includes the entire segmented region 100 , it is determined that the position of the lesion 42 is in the edge portion 40A of the endoscopic image 40 .

[0114] Here, the determination criterion is whether the edge portion 40A includes the entire segmented region 100. However, this is merely an example, and the determination criterion may also be whether a predetermined ratio (e.g., 80%) of the segmented region 100 is included in the edge portion 40A. Furthermore, the ratio may be a fixed value or a variable value that can be changed by a user or the like in accordance with instructions received by the receiving device 64 and / or imaging conditions.

[0115] The determination unit 82B then calculates the amount of change in the position of the lesion 42 between adjacent endoscopic images 40 in time series (hereinafter referred to as "lesion position change"). The determination unit 82B then determines whether the lesion position change is greater than or equal to a threshold. The threshold may be a fixed value or a variable value that can be changed by a user, for example, based on instructions received by the receiving device 64 and / or imaging conditions.

[0116] The determination unit 82B calculates the amount of change in the segmented region 100 (hereinafter also referred to as the "segmented region change amount") as the amount of change in the lesion position. Furthermore, it is determined whether the segmented region change amount is greater than a threshold value. The segmented region change amount is determined based on the degree of overlap between one segmented region 100 and another segmented region 100 obtained from adjacent endoscopic images 40 in time series. For example, the segmented region change amount can be determined based on the IoU (Intersection over Union) or simply based on the number of pixels in the area where one segmented region 100 overlaps with another segmented region 100.

[0117] Regardless of whether the lesion 42 is located within the edge portion 40A, if the segmented region change amount is greater than or equal to the threshold, the determination unit 82B determines not to measure the size of the lesion 42 (in other words, not to output the size of the lesion 42). Furthermore, if the lesion 42 is not located within the edge portion 40A and the segmented region change amount is less than the threshold for two consecutive frames, the determination unit 82B determines to measure the size of the lesion 42 (in other words, to output the size of the lesion 42).

[0118] Here, an example is given of a method of determining that the size of the lesion 42 is being measured based on the condition that the segmented area change is continuously less than the threshold for 2 frames. However, this is only an example. The size of the lesion 42 can also be determined based on the condition that the segmented area change is continuously less than the threshold for 3 or more frames, or the size of the lesion 42 can be determined based on the condition that the segmented area change is less than the threshold within a single frame.

[0119] As an example, Figure 7 As shown, when the position of the lesion 42 is at the edge portion 40A, the determination unit 82B determines not to perform the size measurement of the lesion 42 (in other words, not to output the size of the lesion 42) regardless of whether the change in the lesion position is above the threshold value. When the position of the lesion 42 is not at the edge portion 40A, the determination unit 82B determines to perform the size measurement of the lesion 42 (in other words, to output the size of the lesion 42) on the condition that the change in the lesion position is less than the threshold value.

[0120] In addition, for convenience of explanation, the result of the determination by the determination unit 82B as to whether or not to perform the size measurement of the lesion 42 is also referred to as a "determination result" hereinafter.

[0121] When the determination unit 82B determines that the size of the lesion 42 is to be measured, as an example, Figure 8 As shown, the measuring unit 82C measures the size 112 of the lesion 42 based on the endoscopic image 40. If the determining unit 82B determines that the size of the lesion 42 is not to be measured, the measuring unit 82C does not measure the size 112.

[0122] The measuring unit 82C acquires the endoscopic image 40 used in the determination by the determining unit 82B from the recognizing unit 82A, and derives the distance information 102 based on the acquired endoscopic image 40. The distance information 102 indicates the distance from the camera 52 to the intestinal wall 24 including the lesion 42 (refer to Figure 1 ) to the distance. The distance information 102 is derived for each pixel constituting the endoscopic image 40. Alternatively, the distance information 102 may be derived for each block larger than a pixel (e.g., a pixel group consisting of several to several hundred pixels).

[0123] The distance information 102 is derived using AI. In this embodiment, the distance derivation model 94 is used to derive the distance information 102.

[0124] The distance derivation model 94 is optimized by performing machine learning on the neural network using the second training data. The second training data is a data set including a plurality of data (ie, a plurality of frames of data) in which the second example question data and the second correct answer data are associated with each other.

[0125] The second example data is an image corresponding to the endoscopic image 40. The second correct answer data is correct answer data (i.e., annotation) for the second example data. Here, as an example of the second correct answer data, an annotation is used that specifies the distance corresponding to each pixel shown in the image used as the second example data.

[0126] The measurement unit 82C obtains the endoscopic image 40 used in the determination by the determination unit 82B from the recognition unit 82A and inputs the obtained endoscopic image 40 into the distance derivation model 94. Consequently, the distance derivation model 94 outputs distance information 102 for each pixel of the input endoscopic image 40. Specifically, in the measurement unit 82C, information indicating the distance from the position of the camera 52 (e.g., the position of the image sensor or objective lens mounted on the camera 52) to the intestinal wall 24 reflected in the endoscopic image 40 is output from the distance derivation model 94 as distance information 102 for each pixel of the endoscopic image 40. The position of the camera 52 is an example of an "observation position" according to the technique of the present invention.

[0127] The measuring unit 82C generates a distance image 104 based on the distance information 102 output from the distance derivation model 94. The distance image 104 is an image in which the distance information 102 is distributed in units of pixels included in the endoscopic image 40.

[0128] The measuring unit 82C refers to the position-specific information 98 obtained from the endoscopic image 40 input to the distance derivation model 94, and extracts distance information 102 corresponding to the position determined based on the position-specific information 98 from the distance image 104. Examples of the distance information 102 extracted from the distance image 104 include distance information 102 corresponding to a specific position (e.g., the center of gravity) of the lesion 42, or statistical values ​​(e.g., the median, average, or mode) of the distance information 102 for a plurality of pixels (e.g., all pixels) included in the lesion 42.

[0129] The measuring unit 82C extracts the number of pixels 106 from the endoscopic image 40. The number of pixels 106 is the number of pixels on the line segment 108 within the image area at the position determined according to the position-specific information 98 (i.e., the image area representing the lesion 42) in all the image areas of the endoscopic image 40 input to the distance derivation model 94. As an example of the line segment 108, the longest line segment parallel to the long side of the circumscribed rectangular frame 110 in the image area representing the lesion 42 can be cited. In addition, the line segment 108 is only an example, and the longest line segment parallel to the short side of the circumscribed rectangular frame 110 in the image area representing the lesion 42 can also be used instead of the line segment 108. In this embodiment, the number of pixels 106 is an example of the "number of pixels" involved in the technology of the present invention. Moreover, in this embodiment, the line segment 108 is an example of the "range of the measurement object within the observation object area" involved in the technology of the present invention.

[0130] The measuring unit 82C calculates a size 112 of the lesion 42 in real space based on the distance information 102 extracted from the distance image 104 and the number of pixels 106 extracted from the endoscopic image 40. The size 112 refers to, for example, the length of the lesion 42 in real space.

[0131] Dimension 112 is calculated using equation 114. Measuring unit 82C inputs distance information 102 extracted from distance image 104 and number of pixels 106 extracted from endoscopic image 40 into equation 114. Equation 114 uses distance information 102 and number of pixels 106 as independent variables and dimension 112 as a dependent variable. Equation 114 outputs dimension 112 corresponding to the input distance information 102 and number of pixels 106.

[0132] While the length of the lesion 42 in real space is exemplified here as dimension 112, the technology of the present invention is not limited thereto, and dimension 112 may also be the surface area or volume of the lesion 42 in real space. In this case, for example, a computational expression 114 is used that uses the number of pixels of all image regions representing the lesion 42 and the distance information 102 as independent variables, and uses the surface area or volume of the lesion 42 in real space as a dependent variable.

[0133] As an example, Figure 9 and Figure 10 As shown, the control unit 82D makes the display content displayed on the second screen 38 different according to the determination result. As an example, Figure 9 As shown, when the determination unit 82B determines that the size 112 is to be measured, the control unit 82D acquires the endoscopic image 40 used in the determination by the determination unit 82B from the camera 52 and displays the endoscopic image 40 acquired from the camera 52 on the first screen 36 .

[0134] The control unit 82D obtains from the measurement unit 82C the size 112 measured by the measurement unit 82C based on the endoscopic image 40 displayed on the first screen 36. The control unit 82D also obtains from the recognition unit 82A the segmented region 100 and the position identification information 98 corresponding to the endoscopic image 40 displayed on the first screen 36.

[0135] The control unit 82D uses the segmented region 100 acquired from the recognition unit 82A as the lesion image 44C (reference Figure 1 ) is displayed on the second screen 38. At this time, on the second screen 38, the segmented area 100 is displayed at a position determined based on the position-specific information 98 acquired by the control unit 82D from the recognition unit 82A. Furthermore, the control unit 82D displays the dimension 112 acquired from the measurement unit 82C on the second screen 38. Furthermore, the control unit 82D displays the dimension line 115 on the second screen 38 in a manner that allows identification of which part of the segmented area 100 the dimension 112 corresponds to. The dimension line 115 is created and displayed, for example, by the control unit 82D based on the position-specific information 98 acquired from the recognition unit 82A. The dimension line 115 can be created, for example, in the same manner as the line segment 108 (i.e., the same manner as when the circumscribed rectangular frame 110 is used).

[0136] On the other hand, when the determination unit 82B determines that the dimension 112 is not to be measured, as an example, Figure 10 As shown, the control unit 82D is Figure 9 The same procedure as in the example is used, but the endoscopic image 40 is displayed on the first screen 36 and the segmented area 100 is displayed on the second screen 38. Furthermore, the control unit 82D does not display the size 112 on the second screen 38, but displays the non-output specific information 116 on the second screen 38. The non-output specific information 116 is information that can be used to determine that the size 112 is not output (in other words, the size 112 is not measured) (here, as an example, it is information that can be used to determine that the determination unit 82B has determined that the size 112 is not measured). Figure 10In the example shown, the text "Unmeasurable" is displayed on the second screen 38. The text "Unmeasurable" is just an example, and may be a text such as "Not Outputtable" or any other information that can indicate that the dimension 112 will not be output (e.g., a mark or symbol).

[0137] In this embodiment, the non-output specific information 116 is an example of "non-output specific information" involved in the technology of the present invention. In addition, in this embodiment, the second screen 38 is an example of the "first screen," "second screen," and "third screen" involved in the technology of the present invention.

[0138] Next, refer to Figure 11 , the functions of the parts of the endoscope system 10 involved in the technology of the present invention will be described.

[0139] exist Figure 11 , an example of a medical assistance processing flow performed by the processor 82 is shown. Figure 11 The illustrated medical support processing flow is an example of the “image processing method” according to the technology of the present invention.

[0140] exist Figure 11 In the illustrated medical support process, first, in step ST10, the recognition unit 82A determines whether the camera 52 has captured one frame of imagery within the large intestine 22. If, in step ST10, the camera 52 has not captured one frame of imagery within the large intestine 22, the determination is negative, and the determination in step ST10 is repeated. If, in step ST10, the camera 52 has captured one frame of imagery within the large intestine 22, the determination is positive, and the medical support process proceeds to step ST12.

[0141] In step ST12, the recognition unit 82A and the control unit 82D acquire a single frame of the endoscopic image 40 (see FIG. 1 ) obtained by imaging the large intestine 22 with the camera 52. Figure 5 ) In addition, for the sake of convenience, the description here is based on the premise that the lesion 42 is reflected in the endoscopic image 40. After the processing of step ST12 is executed, the medical support processing moves to step ST14.

[0142] In step ST14, the control unit 82D displays the endoscopic image 40 acquired in step ST12 on the first screen 36 (see FIG. Figure 5 、 Figure 9 and Figure 10 ). After executing the process of step ST14, the medical support process moves to step ST16.

[0143] In step ST16, the recognition unit 82A recognizes the position of the lesion 42 in the endoscopic image 40 by performing recognition processing 96 using the endoscopic image 40 acquired in step ST12, and acquires position specifying information 98 (refer to FIG. Figure 5 ). After executing the process of step ST16, the medical support process moves to step ST18.

[0144] In step ST18, the determination unit 82B determines whether to measure the size 112 of the lesion 42 shown in the endoscopic image 40 acquired in step ST12 (see FIG. 1 ), based on the position identification information 98 acquired by the recognition unit 82A in step ST16. Figure 6 and Figure 7 If, in step ST18, it is determined that the size 112 of the lesion 42 shown in the endoscopic image 40 should be measured, the determination is affirmative, and the medical support process proceeds to step ST20. If, in step ST18, it is determined that the size 112 of the lesion 42 shown in the endoscopic image 40 should not be measured, the determination is negative, and the medical support process proceeds to step ST24.

[0145] In step ST20, the measuring unit 82C measures the size 112 (refer to the size 112 of the lesion 42 shown in the endoscopic image 40 acquired in step ST12) Figure 8 ). After executing the process of step ST20, the medical support process moves to step ST22.

[0146] In step ST22, the control unit 82D displays the size 112 measured by the measuring unit 82C in step ST20 on the second screen 38 (see FIG. Figure 9 ). After executing the process of step ST22, the medical support process moves to step ST26.

[0147] In step ST24, the control unit 82D displays the non-output specific information 116 on the second screen 38 (see Figure 10 ). After executing the process of step ST24, the medical support process moves to step ST26.

[0148] In step ST26, the control unit 82D determines whether a condition for terminating the medical assistance process is satisfied. An example of a condition for terminating the medical assistance process is a condition in which an instruction to terminate the medical assistance process is sent to the endoscope system 10 (e.g., a condition in which the receiving device 64 receives an instruction to terminate the medical assistance process).

[0149] In step ST26, if the condition for ending the medical support process is not met, the determination is negative, and the medical support process proceeds to step ST10. In step ST26, if the condition for ending the medical support process is met, the determination is positive, and the medical support process ends.

[0150] As described above, in the endoscope system 10 according to the present embodiment, based on the endoscopic image 40 in which the lesion 42 is reflected, the position of the lesion 42 in the endoscopic image 40 is recognized by the recognition unit 82A (refer to FIG. Figure 5 Here, when measuring the size 112 of the lesion 42 whose position in the endoscopic image 40 is identified by the recognition unit 82A, if the camera 52 shakes violently or the body moves violently, the endoscopic image 40 will be blurred, making it difficult to accurately measure the size 112 of the lesion 42 by the AI ​​method using the endoscopic image 40. Furthermore, if the edge portion 40A of the endoscopic image 40 is deformed due to the optical influence of the objective lens of the camera 52, it is difficult to accurately measure the size 112 of the lesion 42 by the AI ​​method using the endoscopic image 40. That is, when measuring the size 112 by a measurement method that does not take into account the blurring of the endoscopic image 40 and / or the deformation of the edge portion 40A (for example, a measurement method based on the distance derivation model 94 created without taking into account the blurring of the endoscopic image 40 and / or the deformation of the edge portion 40A), there is a possibility that an inaccurate size 112 will be measured.

[0151] Therefore, in the endoscope system 10 according to the present embodiment, the determination unit 82B determines whether to measure the size 112 of the lesion 42 based on the position of the lesion 42 in the endoscopic image 40 used by the recognition unit 82A (see FIG. Figure 6 and Figure 7 ). When the determination unit 82B determines that the size 112 is to be measured, the measurement unit 82C measures the size 112 of the lesion 42 shown in the endoscopic image 40 based on the endoscopic image 40 used by the recognition unit 82A and the determination unit 82B (refer to Figure 8 ).

[0152] Therefore, the doctor 16 can accurately grasp the size 112 of the lesion 42 shown in the endoscopic image 40. As a result, the doctor 16 can avoid performing medical treatment when it is not necessary or not performing medical treatment when it is necessary.

[0153] Furthermore, in the endoscope system 10 according to the present embodiment, the position of the lesion 42 in each endoscopic image 40 is recognized by the recognition unit 82A (refer to FIG. Figure 5 ). Furthermore, using the amount of change in the position of the lesion 42 between the endoscopic images 40 adjacent in time series (for example, the amount of change in the segmented area specified by IoU), it is determined whether to measure the size 112 of the lesion 42 (refer to Figure 6Therefore, even if the clarity of the endoscopic images 40 changes or body movement occurs between adjacent endoscopic images 40 in the time series, the doctor 16 can accurately grasp the size 112 of the lesion 42 reflected in the adjacent endoscopic images 40 in the time series.

[0154] Furthermore, in the endoscope system 10 according to the present embodiment, the position of the lesion 42 in the endoscopic image 40 is recognized by the recognition unit 82A in a segmented manner based on AI for each endoscopic image 40 (see FIG. Figure 5 ). Then, using the segmented region variation, the determination unit 82B determines whether to measure the size 112 of the lesion 42 (refer to Figure 6 ).

[0155] Furthermore, in the endoscope system 10 according to the present embodiment, whether or not to measure the size 112 of the lesion 42 is determined based on whether or not the position of the lesion 42 shown in the endoscopic image 40 is at the edge 40A of the endoscopic image 40 (see FIG. Figure 6 and Figure 7 ). Therefore, it is possible to suppress inaccurate measurement of the size 112 of the lesion 42 due to optical influences such as deformation of the edge portion 40A of the endoscopic image 40.

[0156] Furthermore, in the endoscope system 10 according to the present embodiment, when the size 112 of the lesion 42 is measured by the measuring unit 82C, the measured size 112 is displayed on the second screen 38 (see FIG. 1 ). Figure 9 ). Therefore, the doctor 16 can visually recognize the size 112 of the lesion 42 reflected in the endoscopic image 40.

[0157] Furthermore, in the endoscope system 10 according to the present embodiment, when it is determined that the size 112 of the lesion 42 is not to be measured, a non-output specific information 116 (see FIG. 1 ) is displayed on the second screen 38. Figure 10 The non-output specific information 116 is information that indicates that the size 112 will not be measured. Therefore, the doctor 16 can visually understand that the size 112 of the lesion 42 will not be measured.

[0158] Furthermore, in the above embodiment, an example of displaying the dimension 112 when measuring the dimension 112 is described, but the technology of the present invention is not limited to this. For example, when the determination unit 82B determines not to output the dimension 112, the dimension 112 may be measured but not displayed. However, when the determination unit 82B determines to output the dimension 112, the dimension 112 may be measured and displayed in the same manner as in the above embodiment.

[0159] In the above embodiment, each time the recognition unit 82A obtains the position specific information 98, the determination unit 82B makes the determination, but the technology of the present invention is not limited to this. Figure 12 As shown, the instructions issued (in Figure 12 In the example shown, determination unit 82B determines whether to measure size 112 of lesion 42 based on the position of lesion 42 within endoscopic image 40 (hereinafter referred to as "first frame FL1") selected by instruction 118 received by receiving device 64 and the position of lesion 42 within at least one endoscopic image 40 obtained earlier than first frame FL1 (hereinafter referred to as "second frame FL2") among a plurality of endoscopic images 40 in a time series. Here, first frame FL1 is an example of the "first frame" involved in the technology of the present invention, and second frame FL2 is an example of the "second frame" involved in the technology of the present invention.

[0160] exist Figure 12 In the example shown, a method is shown in which a plurality of position-specific information 98 corresponding to a plurality of second frames FL2 obtained earlier than the first frame FL1 is stored in the storage area 120. The storage area 120 is, for example, an area provided in the RAM 74. In the storage area 120, a plurality of position-specific information 98 corresponding to a specific number of frames (for example, several to several hundred frames) of the second frame FL2 is stored in a FIFO manner. Figure 12 In the example shown, the determination unit 82B determines whether to measure the size 112 of the lesion 42 based on the position identification information 98 selected as a representative one among the plurality of position identification information 98 stored in the storage area 120 and the position identification information 98 corresponding to the first frame FL1.

[0161] A first example of the positional identification information 98 selected as representative is the positional identification information 98 corresponding to the second frame FL2 that is adjacent to the first frame FL1 in time series. A second example of the positional identification information 98 selected as representative is a statistical value (e.g., an average, a median, or a mode) obtained from the plurality of positional identification information 98 stored in the storage area 120. A third example of the positional identification information 98 selected as representative is positional identification information 98 randomly selected from the plurality of positional identification information 98 stored in the storage area 120. A fourth example of the positional identification information 98 selected as representative is the positional identification information 98 that is located at the center of the plurality of positional identification information 98 stored in the storage area 120 in time series. A fifth example of the positional identification information 98 selected as representative is positional identification information 98 selected from the plurality of positional identification information 98 stored in the storage area 120 in accordance with an instruction received by the receiving device 64.

[0162] So, in Figure 12 In the illustrated example, whether or not to measure the size 112 of the lesion 42 is determined based on the position of the lesion 42 within the first frame FL1 selected in accordance with the instruction 118 received by the receiving device 64 among the multiple endoscopic images 40 in the time series, and the position of the lesion 42 within at least one second frame FL2 obtained earlier than the first frame FL1 among the multiple endoscopic images 40 in the time series. Therefore, it is possible to determine whether or not to measure the size 112 of the lesion 42 shown in the multiple endoscopic images 40 in the time series at a desired timing by the physician 16.

[0163] In the above embodiment, when the determination unit 82B determines not to measure the size 112 of the lesion 42, the example of displaying the non-output specific information 116 on the second screen 38 is given. However, the technology of the present invention is not limited to this. For example, when the determination unit 82B determines not to measure the size 112 of the lesion 42, the past measurement results of the measurement unit 82C (i.e., the size 112 previously measured by the measurement unit 82C) may be displayed on the second screen 38.

[0164] At this time, for example, Figure 13 As shown, in medical assistance treatment, instead of Figure 11 The process of step ST24 shown in FIG. 24 is replaced by the process of step ST24A.

[0165] In step ST24A, the control unit 82D displays the past results measured by the measuring unit 82C (i.e., the size 112 measured in the past by the measuring unit 82C) on the second screen 38. As a first example of the size 112 measured in the past by the measuring unit 82C, the size 112 measured last by the measuring unit 82C can be cited. As a second example of the size 112 measured in the past by the measuring unit 82C, the statistical value of the size 112 measured last by the measuring unit 82C can be cited (e.g., the median value, average value, mode value, maximum value, or minimum value of the size 112 of the lesion 42 that appeared in several to several hundred frames in the past). As a third example of the size 112 measured in the past by the measuring unit 82C, the size 112 measured in the previous endoscopic examination can be cited (e.g., the size 112 of the lesion 42 at the same position as the lesion 42 that appears in the endoscopic image 40 currently displayed on the first screen 36).

[0166] exist Figure 14 In the example shown, the size 112 measured in the past is displayed on the second screen 38. Figure 14 In the example shown, the size 112 displayed on the second screen 38 is in accordance with the current result measured by the measuring unit 82C (for example, Figure 9 The size 112 shown on the second screen 38 is displayed in a distinguishable manner. For example, the size 112 is displayed in Figure 9The second screen 38 shown is shown in bold, and the size 112 is shown in bold. Figure 14 The second screen 38 is shown as a thin line. The thin line display is only an example. The size 112 can be displayed as a semi-transparent color, a different font, or a different brightness than the current result measured by the measuring unit 82C. Figure 14 The size 112 shown on the second screen 38 shown in FIG. Figure 9 The size 112 displayed on the second screen 38 shown may be distinguishable.

[0167] So, in Figure 13 and Figure 14 In the example shown, when it is determined that the dimension 112 is not to be measured, the past results of the measurement by the measuring unit 82C are displayed on the second screen 38. Therefore, when it is determined that the dimension 112 is not to be measured, the doctor 16 can visually recognize the dimension 112 measured in the past.

[0168] And, in Figure 14 In the example shown, the past results measured by the measuring unit 82C (for example, Figure 14 The size 112 shown on the second screen 38 is in accordance with the current result measured by the measuring unit 82C (for example, Figure 9 The size 112 shown on the second screen 38 is displayed in a distinguishable manner on the second screen 38. Therefore, the doctor 16 can visually recognize the past results measured by the measuring unit 82C and the current results measured by the measuring unit 82C.

[0169] While the above embodiment illustrates an example of calculating the lesion position change based on the degree of overlap of segmented regions 100 between adjacent endoscopic images 40 in a time series, the present invention is not limited to this method. For example, the lesion position change can also be calculated based on the distance between the positions of the lesions 42 between adjacent endoscopic images 40 in a time series. In this case, for example, the distance between the center positions of the segmented regions 100 between adjacent endoscopic images 40 in a time series (i.e., the offset between the center positions) can be calculated as the lesion position change. In this case, the same effects as those of the above embodiment can be expected.

[0170] In the above embodiment, the position of the lesion 42 is identified by AI-based segmentation for each endoscopic image 40. However, the technology of the present invention is not limited to this. For example, the position of the lesion 42 may be identified by AI-based bounding box for each endoscopic image 40. In this case, as an example, Figure 15As shown, the determination unit 82B calculates the change amount of the bounding box 122 and determines whether to measure the size 112 of the lesion 42 based on the change amount of the bounding box 122 in the same manner as the above embodiment. In this case, the same effects as the above embodiment can be expected.

[0171] In the above embodiment, AI-based image recognition processing is exemplified as the recognition processing 96, but the technology of the present invention is not limited to this. The recognition unit 82A can also identify the position of the lesion 42 reflected in the endoscopic image 40 by performing non-AI-based image recognition processing (for example, template matching, etc.).

[0172] In the above embodiment, an example of determining whether to measure the size 112 is used based on the amount of change in the position of the lesion 42 between adjacent endoscopic images 40 in a time series. However, the technology of the present invention is not limited to this. It is also possible to determine whether to measure the size 112 based on the amount of change in the position of the lesion 42 between endoscopic images 40 spanning three or more frames in a time series. In this case, the amount of change in the position of the lesion 42 between endoscopic images 40 spanning three or more frames in a time series can be a statistical value such as the average, median, mode, or maximum value of the amount of change between the endoscopic images 40 spanning three or more frames in a time series. Furthermore, it is also possible to determine whether to measure the size 112 based on the amount of change in the position of the lesion 42 between multiple frames in a time series separated by one or more frames.

[0173] In the above embodiment, the size 112 is displayed on the second screen 38, but the technology of the present invention is not limited to this. For example, the size 112 may be displayed on the first screen 36. In this case, for example, the size 112 may be displayed near the lesion 42 in the endoscopic image 40 or outside the endoscopic image 40. Furthermore, the size 112 may be displayed on a display device different from the display device 14.

[0174] In the above embodiment, the display device 14 is exemplified as the output end of the dimension 112, but the technology of the present invention is not limited thereto, and the output end of the dimension 112 may also be a device other than the display device 14. As an example, Figure 16 As shown, as the output end of the size 112, an audio playback device 124, a printer 126 and / or an electronic medical record management device 128 can be cited.

[0175] The dimensions 112 can be output as audio by the audio playback device 124. Furthermore, the dimensions 112 can be printed as text on a medium (e.g., paper) by the printer 126. Furthermore, the dimensions 112 can be stored in the electronic medical record 130 managed by the electronic medical record management device 128 along with the display content of the first screen 36 and / or the second screen 38.

[0176] In the above embodiment, the example of displaying the dimension 112 when the determination unit 82B determines that the dimension 112 is to be measured (in other words, when the determination unit 82B determines that the dimension 112 is to be output) and not displaying the dimension 112 when the determination unit 82B determines that the dimension 112 is not to be measured (in other words, when the determination unit 82B determines that the dimension 112 is not to be output) is described. However, the technology of the present invention is not limited to this. For example, even when the determination unit 82B determines that the dimension 112 is not to be measured, the dimension 112 may be measured and the measured dimension 112 may be displayed.

[0177] In this case, the concept of not displaying the dimension 112 also includes the concept of reducing the display level of the dimension 112. For example, the concept of not displaying the dimension 112 also includes the concept of displaying the dimension 112 in a display method that prevents the user (e.g., the doctor 16) from visually perceiving the dimension 112. Examples of display methods in this case include reducing the font size of the dimension 112, thinning the dimension 112, dotting the dimension 112, making the dimension 112 blink, displaying the dimension 112 for an imperceptible display time, or making the dimension 112 transparent.

[0178] Furthermore, the concept of not displaying the size 112 also includes displaying the size 112 in a display manner that is visually perceived by a user or the like (e.g., the doctor 16) but in a manner that is different from the size 112. Figure 9 The concept of the second display mode displaying the size 112 is different from the first display mode shown. The first display mode refers to, for example, a display mode (for example, a display mode specified by display position, font type, font size, font color and / or brightness, etc.) that can be determined by the determination unit 82B to determine the measurement of the size 112 (in other words, the output of the size 112 is determined by the determination unit 82B). The second display mode refers to, for example, a display mode (for example, a display mode specified by display position, font type, font size, font color and / or brightness, etc.) that can be determined by the determination unit 82B to not measure the size 112 (in other words, the output of the size 112 is determined by the determination unit 82B). The first display mode and the second display mode are different display modes from each other, as long as the user or the like can determine the display mode of the determination result. In addition, it can be said that the same applies to various outputs such as the above-mentioned audio output, printing and storage.

[0179] In the above embodiment, the example of not outputting the specific information 116 and displaying it on the second screen 38 is described. However, the example of not outputting the specific information 116 is not limited to this. For example, the non-output specific information 116 may be output as audio by the audio playback device 124, recorded on a medium (e.g., paper) by the printer 126, or stored in a memory and / or the electronic medical record 130.

[0180] exist Figure 13 and Figure 14 In the example above, the past results measured by the measuring unit 82C are displayed on the second screen 38, but the technology of the present invention is not limited to this. For example, the past results measured by the measuring unit 82C can be output as audio via the audio playback device 124 or recorded on a medium (e.g., paper) via the printer 126.

[0181] In the above embodiment, the calculation of size 112 using equation 114 is described as an example. However, the technology of the present invention is not limited to this. Size 112 can also be measured by processing endoscopic image 40 using AI. In this case, for example, a learned model can be used that outputs size 112 of lesion 42 when endoscopic image 40 including lesion 42 is input. When creating the learned model, training data, in which lesions appearing in images used as example data are annotated with annotations indicating lesion size, can be used as correct answer data to perform deep learning on the neural network.

[0182] In the above embodiment, an example of a method for deriving distance information 102 using distance derivation model 94 is used for description, but the technology of the present invention is not limited to this. For example, other methods for deriving distance information 102 using AI include methods that combine segmentation and depth inference (for example, regression learning that assigns distance information 102 to the entire image (for example, all pixels constituting the image) or unsupervised learning that learns the distance of the entire image).

[0183] In the above embodiment, the distance from the camera 52 to the intestinal wall 24 is derived by AI, but the distance from the camera 52 to the intestinal wall 24 may be actually measured. Figure 2 ) A distance measuring sensor is set up, and the distance from the camera 52 to the intestinal wall 24 is measured by the distance measuring sensor.

[0184] In the above embodiment, an endoscopic image 40 is illustrated, but the technology of the present invention is not limited to this. The technology of the present invention is also applicable to medical images other than the endoscopic image 40 (for example, images obtained by medical imaging equipment other than the endoscope 12, such as radiographic images or ultrasonic images).

[0185] In the above embodiment, an example of measuring the size 112 of the lesion 42 shown in a moving image is given. However, this is only an example, and the technology of the present invention is also applicable to a frame-transferred image or a still image showing the lesion 42 .

[0186] In the above embodiment, an example is given of inputting distance information 102 extracted from distance image 104 into calculation formula 114. However, the technology of the present invention is not limited to this. For example, rather than generating distance image 104, distance information 102 corresponding to the position identified by position identification information 98 may be extracted from all distance information 102 output by distance derivation model 94, and this extracted distance information 102 may be input into calculation formula 114.

[0187] In the above embodiment, the medical support processing is performed by the processor 82 of the computer 78 included in the endoscope 12. However, the technology of the present invention is not limited to this. The device that performs the medical support processing may also be provided outside the endoscope 12. Examples of the device provided outside the endoscope 12 include at least one server and / or at least one personal computer that is communicatively connected to the endoscope 12. Furthermore, the medical support processing may be distributed and performed by multiple devices.

[0188] While the above embodiment illustrates an example in which the medical assistance program 90 is stored in the NVM 86, the present invention is not limited thereto. For example, the medical assistance program 90 may be stored in a non-transitory storage medium such as an SSD or USB memory stick that is readable by a portable computer. The medical assistance program 90 stored in the non-transitory storage medium is installed in the computer 78 of the endoscope 12. The processor 82 executes medical assistance processing in accordance with the medical assistance program 90.

[0189] Furthermore, the medical support program 90 may be stored in a storage device such as another computer or server connected to the endoscope 12 via a network, and the medical support program 90 may be downloaded and installed in the computer 78 in response to a request from the endoscope 12 .

[0190] In addition, it is not necessary to store all of the medical assistance program 90 in a storage device such as another computer or server device connected to the endoscope 12, or to store all of the medical assistance program 90 in the NVM 86. A portion of the medical assistance program 90 may be stored.

[0191] The various processors listed below can be used as hardware resources for performing medical assistance processing. Examples of processors include general-purpose processors (CPUs), which function as hardware resources for performing medical assistance processing by executing software (programs). Furthermore, examples of processors include processors (specialized circuits) with circuit structures specifically designed to perform specific processing, such as FPGAs, PLDs, and ASICs. Any processor also has built-in or connected memory, and any processor uses memory to perform medical assistance processing.

[0192] The hardware resource for performing medical assistance processing can be composed of one of these various processors, or a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Furthermore, the hardware resource for performing medical assistance processing can be a single processor.

[0193] As examples of systems composed of a single processor, one approach involves combining one or more CPUs and software to create a single processor, which functions as a hardware resource for executing medical assistance processing. Another approach involves using a processor, such as a SoC, that implements the functionality of an entire system, including multiple hardware resources for executing medical assistance processing, on a single IC chip. In this manner, medical assistance processing is implemented using one or more of the various processors described above as hardware resources.

[0194] Furthermore, as the hardware structure of these various processors, more specifically, circuits combining circuit elements such as semiconductor devices can be used. Furthermore, the above-described medical assistance processing is merely an example. Therefore, unnecessary steps may be deleted, new steps may be added, or the processing order may be changed without departing from the scope of the present invention.

[0195] The records and diagrams shown above are detailed descriptions of the parts involved in the technology of the present invention and are merely examples of the technology of the present invention. For example, the descriptions related to the above-mentioned structure, function, action and effect are descriptions related to an example of the structure, function, action and effect of the parts involved in the technology of the present invention. Therefore, without departing from the scope of the main purpose of the technology of the present invention, the records and diagrams shown above can be processed as follows: deleting unnecessary parts, adding new elements or replacing them. In addition, in order to avoid complexity and to make it easy to understand the parts involved in the technology of the present invention, in the records and diagrams shown above, descriptions related to technical common sense, etc. that do not require special explanation are omitted on the basis of being able to implement the technology of the present invention.

[0196] In this specification, "A and / or B" has the same meaning as "at least one of A and B." That is, "A and / or B" can mean only A, only B, or a combination of A and B. Furthermore, in this specification, when three or more items are linked using "and / or," the same considerations as for "A and / or B" apply.

[0197] All documents, patent applications, and technical standards described in this specification are incorporated herein by reference to the same extent as if each individual document, patent application, or technical standard were specifically and individually indicated to be incorporated by reference.

[0198] Regarding the above embodiment, the following supplementary notes are further disclosed.

[0199] (Note 1)

[0200] An image processing device includes a processor, wherein:

[0201] The above processor performs the following processing:

[0202] identifying a position of the observation target region within the medical image based on the medical image showing the observation target region;

[0203] determining whether to measure the size of the observation target area based on the position; and

[0204] When it is determined that the measurement is to be performed, the size is measured based on the medical image.

[0205] (Note 2)

[0206] The image processing device according to Supplementary Note 1, wherein:

[0207] The above medical images are multiple frames in time series.

[0208] The above processor performs the following processing:

[0209] identifying the location for each of the frames; and

[0210] Whether or not to perform the measurement is determined using the amount of change in the position between the plurality of frames.

[0211] (Note 3)

[0212] The image processing device according to Supplementary Note 2, wherein:

[0213] The amount of change in the position between the plurality of frames is specified based on the distance between the positions between the plurality of frames.

[0214] (Note 4)

[0215] The image processing device according to Supplementary Note 2, wherein:

[0216] The amount of change in the position between the plurality of frames is defined based on a degree of overlap of the observation target regions between the plurality of frames.

[0217] (Note 5)

[0218] The image processing device according to any one of Supplementary Notes 1 to 4, wherein:

[0219] The above medical images are multiple frames in time series.

[0220] The above processor performs the following processing:

[0221] Identifying the location using an AI-based bounding box approach for each of the frames; and

[0222] Whether or not to perform the above-mentioned measurement is determined using the amount of change in the bounding box.

[0223] (Note 6)

[0224] The image processing device according to any one of Supplementary Notes 1 to 4, wherein:

[0225] The above medical images are multiple frames in time series.

[0226] The above processor performs the following processing:

[0227] identifying the location in each of the frames using AI-based segmentation; and

[0228] Whether or not to perform the above-mentioned measurement is determined using the amount of change in the segmented area.

[0229] (Note 7)

[0230] The image processing device according to any one of Supplementary Notes 1 to 6, wherein:

[0231] The processor determines whether to perform the measurement based on whether the position is at an edge of the medical image.

[0232] (Note 8)

[0233] The image processing device according to any one of Supplementary Notes 1 to 7, wherein:

[0234] The above medical images are multiple frames in time series.

[0235] The processor determines whether to perform the measurement based on the position in a first frame selected according to the issued instruction among the plurality of frames and the position in at least one second frame among the plurality of frames obtained earlier than the first frame.

[0236] (Note 9)

[0237] The image processing device according to any one of Supplementary Notes 1 to 8, wherein:

[0238] The above-mentioned medical images are dynamic images.

[0239] (Note 10)

[0240] The image processing device according to any one of Supplementary Notes 1 to 9, wherein:

[0241] When it is determined that the measurement is to be performed, the processor outputs the size.

[0242] (Note 11)

[0243] The image processing device according to Supplementary Note 10, wherein:

[0244] The size output is achieved by displaying the size on the first screen.

[0245] (Note 12)

[0246] The image processing device according to any one of Supplementary Notes 1 to 11, wherein:

[0247] When it is determined that the measurement is not to be performed, the processor outputs a past result of the measurement.

[0248] (Note 13)

[0249] The image processing device according to Supplementary Note 12, wherein

[0250] The output of the above-mentioned past results is achieved by displaying the above-mentioned past results on the second screen.

[0251] (Note 14)

[0252] The image processing device according to Supplementary Note 13, wherein

[0253] The above processor performs the following processing:

[0254] When it is determined that the measurement is to be performed, the current result of the measurement is displayed on the second screen; and

[0255] When it is determined not to perform the measurement and when it is determined to perform the measurement, the past result and the current result are displayed on the second screen in a distinguishable manner.

[0256] (Note 15)

[0257] The image processing device according to any one of Supplementary Notes 1 to 14, wherein:

[0258] When it is determined that the measurement is not to be performed, the processor outputs measurement non-performance specifying information indicating that the measurement is not to be performed.

[0259] (Note 16)

[0260] The image processing device according to Supplementary Note 15, wherein

[0261] The output of the non-measurement specific information is achieved by displaying the non-measurement specific information on the third screen.

[0262] (Note 17)

[0263] The image processing device according to any one of Supplementary Notes 1 to 16, wherein:

[0264] The medical images are endoscopic images obtained by taking images with an endoscope.

[0265] (Note 18)

[0266] The image processing device according to any one of Supplementary Notes 1 to 17, wherein:

[0267] The above-mentioned observation target region is a lesion.

[0268] (Note 19)

[0269] An endoscope comprising:

[0270] The image processing device according to any one of Supplementary Notes 1 to 18; and

[0271] The module is inserted into a body including the observation target region to capture an image of the observation target region to acquire the medical image.

[0272] (Note 20)

[0273] An image processing method comprises the following steps:

[0274] identifying a position of the observation target region within the medical image based on the medical image showing the observation target region;

[0275] determining whether to measure the size of the observation target area based on the position; and

[0276] When it is determined that the measurement is to be performed, the size is measured based on the medical image.

[0277] (Note 21)

[0278] A program for causing a computer to execute processing comprising the steps of: identifying a position of an observation target region in a medical image showing the observation target region;

[0279] determining whether to measure the size of the observation target area based on the position; and

[0280] When it is determined that the measurement is to be performed, the size is measured based on the medical image.

Claims

1. An image processing device comprising a processor, wherein: The processor performs the following processing: identifying a position of the observation target region within the medical image based on the medical image in which the observation target region is reflected; determining whether to output the size of the observation target area according to the position; and When it is determined that the output is to be performed, the size is output.

2. The image processing apparatus according to claim 1, wherein: The medical image is a plurality of frames in time sequence, The processor performs the following processing: identifying the location in each of the plurality of frames; and Whether to perform the output is determined using the amount of change in the position between the plurality of frames.

3. The image processing apparatus according to claim 2, wherein: The amount of change of the position between the plurality of frames is specified according to the distance of the position between the plurality of frames.

4. The image processing apparatus according to claim 2, wherein: The amount of change in the position between the plurality of frames is specified according to a degree of overlap of the observation target areas between the plurality of frames.

5. The image processing apparatus according to claim 1, wherein: The processor measures the size by processing the medical image using AI. The image processing apparatus according to claim 1 , wherein: The processor performs the following processing: deriving a distance from an observation position to the observation target area by processing the medical image using AI; and The size is measured based on the distance and the number of pixels in a range to be measured within the observation target area.

7. The image processing apparatus according to claim 1, wherein: The processor performs the following processing: determining to perform the output when the position exists in a first area within the medical image; and When the position exists in a second region outside the first region in the medical image, it is determined that the output is not to be performed.

8. The image processing apparatus according to claim 1, wherein: The medical image is a plurality of frames in time sequence, The processor performs the following processing: identifying the location in each of the plurality of frames; and Whether to output is determined based on the amount of change in the position between the plurality of frames and whether the position exists in a first region in the medical image or a second region outside the first region in the medical image.

9. The image processing apparatus according to claim 1, wherein: The medical image is a plurality of frames in time sequence, The processor performs the following processing: identifying the location in each of the plurality of frames in an AI-based bounding box manner; and Whether to perform the output is determined using the amount of change in the bounding box.

10. The image processing apparatus according to claim 1, wherein: The medical image is a plurality of frames in time sequence, The processor performs the following processing: identifying the location in each of the plurality of frames in an AI-based segmented manner; and Whether to perform the output is determined using the amount of change in the segmented area.

11. The image processing apparatus according to claim 1, wherein: The processor determines whether to perform the output based on whether the position is at an edge of the medical image.

12. The image processing apparatus according to claim 1, wherein: The medical image is a plurality of frames in time sequence, The processor determines whether to perform the output based on the position in a first frame selected according to the issued instruction among the multiple frames and the position in at least one second frame obtained earlier than the first frame among the multiple frames.

13. The image processing apparatus according to claim 1, wherein: The medical image is a dynamic image.

14. The image processing apparatus according to claim 1, wherein: When it is determined that the output is to be performed, the processor outputs the size.

15. The image processing apparatus according to claim 14, wherein: The size output is achieved by displaying the size on the first screen.

16. The image processing apparatus according to claim 1, wherein: When it is determined that the output is not to be performed, the processor outputs a past result of measuring the size.

17. The image processing apparatus according to claim 16, wherein: The output of the past results is achieved by displaying the past results on the second screen.

18. The image processing apparatus according to claim 17, wherein: The processor performs the following processing: When it is determined that the output is to be performed, the current result of measuring the size is displayed on the second screen; and When it is determined that the output is not to be performed and when it is determined that the output is to be performed, the past result and the current result are displayed on the second screen in a distinguishable manner.

19. The image processing apparatus according to claim 1, wherein: When it is determined that the output is not to be performed, the processor outputs non-output specific information that can determine that the output is not to be performed.

20. The image processing apparatus according to claim 19, wherein: The output of the specific non-output information is achieved by displaying the specific non-output information on the third screen.

21. The image processing apparatus according to claim 1, wherein: The medical image is an endoscopic image obtained by taking an image with an endoscope.

22. The image processing apparatus according to claim 1, wherein: The observation target region is a lesion.

23. An endoscope comprising: The image processing device according to any one of claims 1 to 22; and The module is inserted into a body including the observation target area to capture the observation target area and acquire the medical image.

24. An image processing method comprising the following steps: identifying a position of the observation target region within the medical image based on the medical image in which the observation target region is reflected; determining whether to output the size of the observation target area based on the position; and When it is determined that the output is to be performed, the size is output.

25. A program for causing a computer to execute a process comprising the following steps: identifying a position of the observation target region within the medical image based on the medical image in which the observation target region is reflected; determining whether to output the size of the observation target area based on the position; and When it is determined that the output is to be performed, the size is output.

Images