Medical image processing device, endoscope system, medical image processing method, and medical image processing program

Abstract

The purpose of the present invention is to provide a medical image processing device, an endoscope system, a medical image processing method, and a medical image processing program with which it is possible to minimize unnecessary audio output while reducing the probability of a region-of-interest being missed. The medical image processing device according to an embodiment of the present invention is provided with a processor, the processor executing: an image acquisition process for acquiring a medical image in time series; a region-of-interest detection process for detecting a region-of-interest from the acquired medical image; a display control process for displaying the medical image on a display device; a first notification process for causing, if a region-of-interest is detected in the region-of-interest detection process, the display device to display the medical image and information relating to the detected region-of-interest in a superimposed manner; and a second notification process for outputting audio from an audio output device if a region-of-interest is detected in the region-of-interest detection process, the second notification process being executed after the first notification process.

Description

[Technical Field]

[0001] The present invention relates to a medical image processing apparatus, an endoscope system, a medical image processing method, and a medical image processing program, and more particularly to a technique for notifying the detection result of a region of interest. [Background technology]

[0002] It is known that a medical image processing device notifies a user of a detection result of a region of interest, which is used to support a user such as a doctor when observing or diagnosing a medical image. For example, Patent Document 1 describes detecting a region of interest using a region of interest detection unit selected from a plurality of region of interest detection units, and notifying (displaying) the result. Furthermore, Patent Document 2 describes notifying the detection result or the discrimination result by voice. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] WO2017 / 081976 publication [Patent Document 2] Japanese Patent Application Publication No. 2020-69300 Summary of the Invention [Problem to be solved by the invention]

[0004] When detecting regions of interest, such as lesions, from medical images, such as endoscopic images or ultrasound images, using AI (Artificial Intelligence) or the like and notifying the user of the detected region, it is necessary to immediately notify the detected region to prevent the lesion from being overlooked. However, automatic detection using AI can result in false positives, which can cause annoyance to the user. In particular, frequent erroneous audio notifications tend to be more annoying to users than visual notifications. However, with conventional technologies such as those described in Patent Documents 1 and 2 above, it has been difficult to suppress unnecessary audio output while reducing the possibility of overlooking a region of interest.

[0005] The present invention has been made in consideration of these circumstances, and aims to provide a medical image processing device, an endoscopic system, a medical image processing method, and a medical image processing program that can suppress unnecessary audio output while reducing the possibility of missing an area of ​​interest. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, a medical image processing device according to a first aspect of the present invention is a medical image processing device equipped with a processor, which executes an image acquisition process for acquiring time-series medical images, an area of ​​interest detection process for detecting an area of ​​interest from the acquired medical images, a display control process for displaying the medical images on a display device, a first notification process for, when an area of ​​interest is detected in the area of ​​interest detection process, superimposing information on the medical image and the detected area of ​​interest on the display device, and a second notification process for outputting audio from an audio output device when an area of ​​interest is detected in the area of ​​interest detection process, and the second notification process is executed after the first notification process.

[0007] In the medical image processing device according to the first aspect, the processor performs a first notification process (display on the display device) when a region of interest is detected. This has the effect of preventing the region of interest from being overlooked. From the viewpoint of preventing oversight, it is preferable that the processor performs the first notification process immediately (with as short a delay as possible) when a region of interest is detected, but inevitable delays associated with processing in the device and delays due to intermittent acquisition of medical images are acceptable.

[0008] In detecting an area of ​​interest, false positives tend to occur instantaneously and rarely occur continuously. Therefore, by suppressing audio output when an area of ​​interest is detected instantaneously, it is possible to reduce the degree to which the user feels annoyed by audio output caused by false positives. From this perspective, in the first embodiment, the processor executes the second notification process that outputs audio after the first notification process. That is, the processor does not output audio even if an area of ​​interest is detected during the period immediately after the first notification process (screen display), thereby suppressing audio output due to instantaneous false positives during this period.

[0009] The processor (medical image processing device) may set "how long to delay the second notification (delay time or waiting time)" according to a user's specification, or may set it independently. The user can specify the delay time by considering the balance between the degree of suppression of audio output due to false positives and the power of audio notification.

[0010] According to the medical image processing device of the first aspect, such first and second notification processes can reduce the possibility of overlooking the region of interest while suppressing unnecessary audio output (audio output due to momentary false positives).

[0011] In the first aspect and each of the following aspects, "acquiring time-series medical images" includes sequentially acquiring a plurality of medical images taken at a predetermined frame rate. The acquisition may or may not be in real time.

[0012] The medical image processing device according to the first aspect can be realized, for example, as a processor part of a medical image processing system, but is not limited to such an aspect. Note that the term "medical image" refers to an image obtained as a result of photographing, measuring, etc., of a living body such as a human body for the purpose of diagnosis, treatment, measurement, etc., and examples thereof include endoscopic images, ultrasound images, CT images (Computed Tomography), and MRI images (Magnetic Resonance Imaging). Medical images are also called medical images. Furthermore, in the first aspect and each of the following aspects, a "region of interest (ROI)" may be a region in a medical image that includes a lesion region or a suspected lesion region, an organ, a blood vessel, a treated region, a treatment tool, etc. A "region of interest" may also be called a "region of interest."

[0013] In the medical image processing device according to the second aspect, the processor executes the second notification process if a region of interest is detected even after a first period has elapsed since the region of interest was detected in the region of interest detection process, and does not execute the second notification process if a region of interest is not detected after the first period has elapsed. If a region of interest is detected even after the first period has elapsed since the region of interest was detected, it is considered that the detection is continuous and the possibility of an instantaneous false positive is low, so the second notification process (audio output) can be performed. Note that in the second aspect, the processor can set the value of the "first period" according to the purpose or target of the observation, or according to a user specification.

[0014] A medical image processing device according to a third aspect is the first or second aspect, wherein the processor superimposes information corresponding to the position of a region of interest in a medical image in a first notification process. The third aspect specifically defines the first notification process. The processor may display the information within the region of interest in the medical image, or may display the information in the periphery of the region of interest. The processor may also display the information outside the medical image display area on the display screen of the display device.

[0015] In a medical image processing device according to a fourth aspect, in any one of the first to third aspects, the processor executes a detection count calculation process that calculates the number of consecutive detections for a region of interest detected in the region of interest detection process, and executes a second notification process when the number of consecutive detections exceeds a predetermined number. When the number of consecutive detections exceeds the predetermined number, it is considered that "this is a continuous detection and the possibility of an instantaneous false positive is low," and therefore the second notification process can be executed as in the fourth aspect. The processor may set the "predetermined number" according to a user specification, or may set it independently of a user specification.

[0016] In a medical image processing device according to a fifth aspect, the processor further executes a feature amount storage process for storing features of the detected region of interest, and an identity determination process for determining identity between the first region of interest and the second region of interest by comparing the features of the first region of interest detected from a medical image captured at a first time with the features of the second region of interest detected from a second medical image captured at a second time, which is an earlier time than the first time. The detection count calculation process calculates the number of consecutive detections of the first region of interest based on the determination result of the identity determination process. As in the fifth aspect, calculating the number of consecutive detections while taking into account the identity of the regions of interest allows for more appropriate execution of the second notification process. Note that "storage" may refer to temporary or non-temporary recording. Furthermore, for the first and second regions of interest, "features" may be, for example, type, position, shape, size, and color, but are not limited to these examples.

[0017] A medical image processing device according to a sixth aspect is the fifth aspect, wherein, when the first region of interest and the second region of interest are determined to be identical in the identity determination process, the processor increments the number of consecutive detections recorded for the second region of interest and calculates the number of consecutive detections of the first region of interest as the number of consecutive detections of the second region of interest in the detection count calculation process. The sixth aspect defines a specific aspect of the calculation of the number of consecutive detections.

[0018] In a medical image processing device according to a seventh aspect, in the fifth or sixth aspect, the processor, in the identity determination process, determines identity by comparing, among the feature amounts retained in the feature amount retention process, feature amounts for a time up to a predetermined period before the first time with feature amounts of the first region of interest. In the seventh aspect, the "time up to a predetermined period before the first time" corresponds to the "second time" in the fifth aspect. This "predetermined period" can be set in consideration of problems such as increased calculation costs and reduced accuracy in identity determination.

[0019] The medical image processing device according to an eighth aspect is any one of the first to seventh aspects, wherein the processor, in the second notification process, causes the audio output device to output audio and then does not output audio for a predetermined period of time. The eighth aspect provides a period during which audio output is stopped, taking into consideration that a user may feel annoyed if audio is output frequently or continuously for a long period of time. The processor may set the "predetermined period" according to a user specification, or may set it independently of a user specification.

[0020] A medical image processing device according to a ninth aspect is any one of the first to eighth aspects, wherein the processor changes the mode of the first notification process in accordance with the audio output state in the second notification process. In the ninth aspect, the processor can increase the discriminability of the information superimposed in the first notification process when the second notification process is performed (i.e., when the region of interest is continuously detected and the possibility of an instantaneous false positive is low), and can notify of continuous detection by changing the mode of the first notification process in this way. Furthermore, the processor may change the mode of the first notification process simultaneously with the second notification process (audio output), or may change the timing before or after the second notification process.

[0021] A medical image processing device according to a tenth aspect is any one of the first to ninth aspects, wherein the processor superimposes and displays at least one of characters, figures, and symbols as information (information relating to the detected region of interest) in the first notification process. The tenth aspect defines specific aspects of the information to be superimposed on the medical image. The processor may superimpose and display information according to the feature amount of the region of interest.

[0022] To achieve the above-mentioned object, an endoscopic system according to an eleventh aspect of the present invention includes the medical image processing device according to any one of the first to tenth aspects, an endoscope to be inserted into a subject and having an imaging unit for capturing medical images, a display device, and an audio output device. Since the endoscopic system according to the eleventh aspect includes the medical image processing device according to any one of the first to tenth aspects, it is possible to suppress unnecessary audio output while reducing the possibility of overlooking a region of interest. The endoscopic system according to the eleventh aspect may also include a light source device. The light source device can use normal light (white light), special light (narrowband light, etc.), or a combination thereof as observation light. It is preferable that the light source device irradiates observation light with different wavelength bands depending on the organ, part, observation purpose, type of observation target, etc.

[0023] To achieve the above-mentioned object, a medical image processing method according to a twelfth aspect of the present invention is a medical image processing method executed by a medical image processing device including a processor, wherein the processor executes an image acquisition step of acquiring time-series medical images, a region of interest detection step of detecting a region of interest from the acquired medical images, a display control step of displaying the medical images on a display device, a first notification step of, if a region of interest is detected in the region of interest detection step, superimposing information on the medical image and the detected region of interest on the display device, and a second notification step of, if a region of interest is detected in the region of interest detection step, outputting sound from an audio output device, wherein the second notification step is executed after the first notification step. According to the twelfth aspect, as with the first aspect, unnecessary sound output can be suppressed while reducing the possibility of missing a region of interest. The medical image processing method according to the twelfth aspect may further include the same configuration as the second to tenth aspects.

[0024] To achieve the above-mentioned object, a medical image processing program according to a thirteenth aspect of the present invention is a medical image processing program that causes a medical image processing device having a processor to execute a medical image processing method. The medical image processing method includes an image acquisition step of acquiring time-series medical images, a region-of-interest detection step of detecting a region of interest from the acquired medical images, a display control step of displaying the medical images on a display device, a first notification step of superimposing information about the detected region of interest on the medical image on the display device when a region of interest is detected in the region-of-interest detection step, and a second notification step of outputting sound from an audio output device when a region of interest is detected in the region-of-interest detection step. The second notification step is performed after the first notification step. According to the thirteenth aspect, as with the first and twelfth aspects, it is possible to reduce the possibility of overlooking a region of interest while suppressing unnecessary sound output. Furthermore, the medical image processing program according to the thirteenth aspect may also be a program that further executes processing similar to the second to tenth aspects. Note that a non-transitory recording medium having computer-readable code recorded thereon for the program of these aspects can also be cited as an aspect of the present invention. [Effects of the Invention]

[0025] As described above, the medical image processing device, endoscope system, medical image processing method, and medical image processing program according to the present invention can suppress unnecessary audio output while reducing the possibility of missing an area of ​​interest. [Brief explanation of the drawings]

[0026] [Figure 1] FIG. 1 is an external view of an endoscope system according to a first embodiment. [Figure 2] FIG. 2 is a diagram showing the configuration of the main parts of the endoscope system. [Figure 3] FIG. 3 is a block diagram showing the functional configuration of the processor. [Figure 4] FIG. 4 is a diagram showing the configuration of the attention area detection unit. [Figure 5] FIG. 5 is a diagram showing an example of the layer configuration of the detector. [Figure 6] FIG. 6 is a diagram showing the convolution process by a filter. [Figure 7] FIG. 7 is a flowchart showing the procedure of the medical image processing method according to the first embodiment. [Figure 8] FIG. 8 is a diagram showing an example of a setting screen for processing conditions. [Figure 9] FIG. 9 is a diagram showing an example of superimposed display (first notification process). [Figure 10] FIG. 10 is a flowchart showing details of the notification by voice (second notification process). [Figure 11] FIG. 11 is a diagram showing a specific example (part 1) of audio output. [Figure 12] FIG. 12 is a diagram showing a specific example (part 2) of audio output. [Figure 13] FIG. 13 is a diagram showing a specific example (part 3) of audio output. [Figure 14] FIG. 14 is a diagram showing a specific example (part 4) of audio output. [Figure 15] FIG. 15 is a diagram showing a specific example (part 5) of audio output. [Figure 16]FIG. 16 is a diagram showing a specific example (part 6) of audio output. DETAILED DESCRIPTION OF THE INVENTION

[0027] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, with reference to the accompanying drawings, embodiments of a medical image processing apparatus, an endoscope system, a medical image processing method, and a medical image processing program according to the present invention will be described in detail.

[0028] First Embodiment <Configuration of endoscope system> Fig. 1 is an external view of an endoscopic system 10 (medical image processing device, endoscopic system) according to a first embodiment, and Fig. 2 is a block diagram showing the configuration of the main parts of the endoscopic system 10. As shown in Figs. 1 and 2, the endoscopic system 10 is composed of an endoscope 100 (endoscopic scope, imaging device), a medical image processing unit 200 (medical image processing device, processor), a light source device 300 (light source device), and a monitor 400 (display device). An external device (determination device) that determines the state of the endoscope 100 by using electromagnetic waves, ultrasound, or magnetism may be connected to the endoscopic system 10.

[0029] <Configuration of an endoscope> The endoscope 100 includes a handheld control unit 102 and an insertion section 104 connected to the handheld control unit 102. The surgeon (user) holds and operates the handheld control unit 102, and inserts the insertion section 104 into the body of a subject for observation. The handheld control unit 102 is also provided with an air / water supply button 141, a suction button 142, function buttons 143 to which various functions can be assigned, and an image capture button 144 for receiving image capture instructions (still images, moving images). The insertion section 104 is composed of, in order from the handheld control unit 102 side, a flexible section 112, a bending section 114, and a rigid tip section 116. That is, the bending section 114 is connected to the proximal end side of the rigid tip section 116, and the flexible section 112 is connected to the proximal end side of the bending section 114. The handheld control unit 102 is connected to the proximal end side of the insertion section 104. The user can bend the bending section 114 by operating the hand operation section 102, thereby changing the orientation of the tip rigid section 116 up, down, left, or right. The tip rigid section 116 is provided with an imaging optical system 130, an illumination section 123, a forceps port 126, etc. (see FIGS. 1 and 2).

[0030] During observation or treatment, white light (normal light) and / or narrowband light (special light: for example, one or more of red narrowband light, green narrowband light, blue narrowband light, and violet narrowband light) can be emitted from illumination lenses 123A and 123B of illumination unit 123 by operating operation unit 208 (see FIG. 2). Furthermore, by operating air / water supply button 141, cleaning water is discharged from a water supply nozzle (not shown) to clean photographing lens 132 (photographing lens, photographing unit) of photographing optical system 130 and illumination lenses 123A and 123B. A conduit (not shown) is connected to forceps port 126, which opens at tip rigid portion 116. A treatment tool (not shown) for tumor removal or the like is inserted into this conduit and can be moved forward and backward as appropriate to perform the necessary treatment on the subject.

[0031] 1 and 2, a photographing lens 132 (image pickup unit) is disposed on the distal end surface 116A of the distal end hard portion 116. Behind the photographing lens 132, there are a CMOS (Complementary Metal-Oxide Semiconductor) type image pickup element 134 (image pickup element, image pickup unit), a drive circuit 136, An AFE (Analog Front End) 138 (photographing unit) is provided, and these elements output an image signal. The image sensor 134 is a color image sensor and includes a plurality of pixels each composed of a plurality of light-receiving elements arranged in a matrix (two-dimensional array) in a specific pattern array (e.g., Bayer array, X-Trans (registered trademark) array, honeycomb array, etc.). Each pixel of the image sensor 134 includes a microlens, a red (R), green (G), or blue (B) color filter, and a photoelectric conversion unit (e.g., photodiode). The photographing optical system 130 can generate a color image from pixel signals of three colors (red, green, and blue), or can generate an image from pixel signals of any one or two colors selected from red, green, and blue. Note that, although the first embodiment describes a case in which the image sensor 134 is a CMOS-type image sensor, the image sensor 134 may also be a CCD (Charge Coupled Device) type. Each pixel of the image sensor 134 may further include a purple color filter corresponding to the purple light source 310V and / or an infrared filter corresponding to the infrared light source.

[0032] An optical image of the subject is formed on the light receiving surface (imaging surface) of the imaging element 134 by the photographing lens 132, converted into an electrical signal, and output to the medical image processing unit 200 via a signal cable (not shown), where it is converted into a video signal. As a result, an endoscopic image is displayed on the monitor 400 connected to the medical image processing unit 200.

[0033] Furthermore, illumination lenses 123A and 123B of the illumination unit 123 are provided adjacent to the photographing lens 132 on the distal end surface 116A of the distal end hard portion 116. An exit end of a light guide 170 (described later) is disposed behind the illumination lenses 123A and 123B. This light guide 170 is inserted through the insertion portion 104, the handheld operation unit 102, and the universal cable 106, and the entrance end of the light guide 170 is disposed inside the light guide connector 108.

[0034] The handheld operation unit 102 may include a scope information recording unit (not shown) that records individual information (individual information, scope information) of the endoscope 100. The individual information includes, for example, the type of the endoscope 100 (forward viewing, side viewing, etc.), the model, the individual identification number, and the characteristics of the optical system (field of view, distortion, etc.). The processor 210 (scope information acquisition unit, individual information acquisition unit) can acquire this individual information and use it in medical image processing. The scope information recording unit may be provided in the light guide connector 108.

[0035] In the endoscope system 10, the endoscope 100 having the above-described configuration is used to sequentially capture images of the subject at a determined frame rate (this can be done under the control of the imaging unit and image acquisition unit 220 (see FIG. 3)), thereby making it possible to sequentially acquire time-series medical images. The user performs observations by inserting and withdrawing the endoscope 100 (insertion unit 104) into and from the living body of the subject.

[0036] <Configuration of light source device> 2, light source device 300 is composed of an illumination light source 310, an aperture 330, a condenser lens 340, a light source control unit 350, etc., and causes observation light to enter light guide 170. Light source 310 is equipped with red light source 310R, green light source 310G, blue light source 310B, and purple light source 310V, which emit narrowband light of red, green, blue, and purple, respectively, and is capable of emitting narrowband light of red, green, blue, and purple. The illuminance of the observation light emitted by light source 310 is controlled by light source control unit 350, which can change (increase or decrease) the illuminance of the observation light or stop illumination as necessary.

[0037] The light source 310 can emit narrowband light of red, green, blue, and purple in any combination. For example, it can simultaneously emit narrowband light of red, green, blue, and purple to irradiate white light (normal light) as observation light, or it can emit narrowband light (special light) by emitting one or two of the colors. The light source 310 may further include an infrared light source that irradiates infrared light (an example of narrowband light). Alternatively, white light or narrowband light may be irradiated as observation light by using a light source that irradiates white light and a filter that transmits the white light and each narrowband light.

[0038] <Light source wavelength band> The light source 310 may be a light source that generates white light or light of multiple wavelength bands as white light, or may be a light source that generates light of a specific wavelength band narrower than the white wavelength band. The specific wavelength band may be the blue or green band of the visible range, or the red band of the visible range. If the specific wavelength band is the blue or green band of the visible range, it includes a wavelength band of 390 nm to 450 nm, or 530 nm to 550 nm, and Light in a specific wavelength band The light may have a peak wavelength within a wavelength range of 390 nm to 450 nm or 530 nm to 550 nm. When the specific wavelength range is the red range of the visible light, the specific wavelength range may include a wavelength range of 585 nm to 615 nm or 610 nm to 730 nm, and the light in the specific wavelength range may have a peak wavelength within the wavelength range of 585 nm to 615 nm or 610 nm to 730 nm.

[0039] The specific wavelength band includes a wavelength band in which the absorption coefficients of oxygenated hemoglobin and reduced hemoglobin are different, and Light in a specific wavelength band The peak wavelength may be in a wavelength range where oxygenated hemoglobin and reduced hemoglobin have different absorption coefficients. In this case, the specific wavelength range includes a wavelength range of 400±10 nm, 440±10 nm, 470±10 nm, or 600 nm or more and 750 nm or less, and Light in a specific wavelength bandThe peak wavelength may be in the wavelength band of 400±10 nm, 440±10 nm, 470±10 nm, or 600 nm or more and 750 nm or less.

[0040] In addition, the light generated by the light source 310 Wavelength band includes a wavelength band of 790 nm or more and 820 nm or less, or 905 nm or more and 970 nm or less, and The light emitted by the light source 310 is The peak wavelength may be in a wavelength band of 790 nm or more and 820 nm or less, or 905 nm or more and 970 nm or less.

[0041] Furthermore, the light source 310 may be equipped with a light source that emits excitation light having a peak wavelength of 390 nm or more and 470 nm or less. In this case, it is possible to acquire medical images (medical images, in vivo images) containing information on the fluorescence emitted by fluorescent substances in the subject (living body). When acquiring a fluorescent image, a dye agent for fluoroscopy (fluorescein, acridine orange, etc.) may be used.

[0042] The type of light source (laser light source, xenon light source, LED light source (LED: Light-Emitting Diode), etc.), wavelength, presence or absence of a filter, etc. of the light source 310 are preferably configured according to the type, part, organ, and purpose of observation of the subject, and it is also preferable to combine and / or switch the wavelength of the observation light during observation according to the type, part, organ, and purpose of observation of the subject. When switching wavelengths, the wavelength of the irradiated light may be switched, for example, by rotating a disk-shaped filter (rotary color filter) placed in front of the light source and equipped with a filter that transmits or blocks light of a specific wavelength.

[0043] Furthermore, the imaging element used in carrying out the present invention is not limited to a color imaging element such as the imaging element 134, in which a color filter is provided for each pixel, but may also be a monochrome imaging element. When a monochrome imaging element is used, imaging can be performed in a frame-sequential (color-sequential) manner by sequentially switching the wavelength of the observation light. For example, the wavelength of the emitted observation light may be sequentially switched between (purple, blue, green, red), or broadband light (white light) may be irradiated and the wavelength of the emitted observation light may be switched using a rotary color filter (red, green, blue, purple, etc.). Alternatively, one or more narrowband lights (green, blue, purple, etc.) may be irradiated and the wavelength of the emitted observation light may be switched using a rotary color filter (green, blue, purple, etc.). The narrowband light may be infrared light of two or more different wavelengths.

[0044] By connecting the light guide connector 108 (see Figures 1 and 2) to the light source device 300, the observation light emitted from the light source device 300 is transmitted to the illumination lenses 123A and 123B via the light guide 170, and is then irradiated onto the observation range from the illumination lenses 123A and 123B.

[0045] <Configuration of medical image processing unit> The configuration of the medical image processing unit 200 will be described with reference to FIG. 2. The medical image processing unit 200 receives an image signal output from the endoscope 100 via an image input controller 202, performs necessary image processing using a processor 210 (image acquisition unit 220: processor, computer, medical image processing device), and outputs the image via a video output unit 206. This results in an observed image (medical image) being displayed on a monitor 400 (display device). The communication control unit 205 controls communication with an in-hospital system (HIS: Hospital Information System), an in-hospital LAN (Local Area Network), and / or external systems and networks (not shown). The recording unit 207 (recording device) records images of the subject (endoscopic image, medical image, medical image), site information, information indicating detection results, and the like. The audio processing unit 209, under the control of the processor 210, can output messages (audio) related to the detection results and notification processing (second notification processing) from a speaker 209A (audio output device).

[0046] Furthermore, ROM 211 (Read Only Memory) is a non-volatile storage element (non-temporary recording medium) that stores computer-readable code of programs that cause the processor 210 to execute various image processing methods. RAM 212 (Random Access Memory) is a storage element for temporary storage during various processes, and can also be used as a buffer when acquiring images.

[0047] The user can issue instructions to execute medical image processing and specify the conditions required for execution via the operation unit 208, and the display control unit 232 (see Figure 3) can display on the monitor 400 the screen (see, for example, Figure 8) that appears when issuing these instructions, the detection results of the area of ​​interest, etc.

[0048] <Processor functions> Fig. 3 is a block diagram showing the functional configuration of the processor 210. The processor 210 includes an image acquisition unit 220 (image acquisition unit), an attention area detection unit 222 (attention area detection unit), a detection count calculation unit 226 (detection count calculation unit), a feature amount calculation unit 228 (feature amount calculation unit), an identity determination unit 230 (identity determination unit), a display control unit 232 (display control unit), a first notification unit 234 (first notification unit), a second notification unit 236 (second notification unit), a recording control unit 238 (recording control unit), and a communication control unit 240. As shown in Fig. 4, the attention area detection unit 222 includes a detector 223 and a switching control unit 224. 4 includes a pharynx detector 223A, an esophagus detector 223B, a stomach detector 223C, and a duodenum detector 223D. The switching control unit 224 may switch the detector that displays the detection results on the monitor 400 (display device) based on the analysis results of the endoscopic image (such as the region, organ, and line of sight) or may switch based on imaging information (information indicating the position and / or direction of the imaging device) acquired by the external device (determination device) described above. The processor 210 may operate multiple detectors and display the detection results of some of the detectors, or may operate only the detector that displays the detection results.

[0049] The processor 210, using the above-described functions, can calculate the features of a medical image, emphasize or reduce components in a specific frequency band, and emphasize or reduce the prominence of a specific target (e.g., a region of interest, blood vessels at a desired depth). The processor 210 may also include a special light image acquisition unit that acquires a special light image having information about a specific wavelength band based on a normal light image acquired by irradiating the image with white light or light in multiple wavelength bands as white light. In this case, the signal in the specific wavelength band can be obtained by calculation based on RGB (R: red, G: green, B: blue) or CMY (C: cyan, M: magenta, Y: yellow) color information contained in the normal light image. The processor 210 may also include a feature image generation unit that generates a feature image by calculation based on at least one of a normal light image acquired by irradiating the image with white light or light in multiple wavelength bands as white light and a special light image acquired by irradiating the image with light in a specific wavelength band, and acquire and display the feature image as a medical image.

[0050] Furthermore, the image acquisition unit 220 (processor) may acquire, as a medical image, an endoscopic image (medical image) captured with observation light of a wavelength band corresponding to the region indicated by the region information, and the display control unit 232 may cause the monitor 400 (display device) to display the recognition results for the medical image captured with observation light of that wavelength band. For example, an image captured with white light (normal light) for the stomach, and an image captured with special light (narrowband blue light) such as BLI (Blue Laser Imaging: registered trademark) for the esophagus may be used for detection (recognition). The image acquisition unit 220 may acquire, depending on the region, an image captured with special light such as LCI (Linked Color Imaging: registered trademark) and subjected to image processing (in the case of LCI, saturation differences and hue differences of colors close to the mucous membrane color are expanded).

[0051] Medical image processing using the above-mentioned functions will be described in detail later.

[0052] <Detector using a trained model> The above detector can be configured using a pre-trained model (a model trained using a set of images composed of images of a living body) configured by machine learning, such as a CNN (Convolutional Neural Network) or an SVM (Support Vector Machine). Hereinafter, the layer configuration when the detector 223 (the pharynx detector 223A to the duodenum detector 223D) is configured by a CNN will be described.

[0053] <Example of the layer configuration of CNN> FIG. 5 is a diagram showing an example of the layer configuration of the detector 223. In the example shown in the (a) part of FIG. 5, the detector 223 includes an input layer 250, an intermediate layer 252, and an output layer 254. The input layer 250 inputs an endoscopic image (medical image) acquired by the image acquisition unit 220 and outputs a feature amount. The intermediate layer 252 includes a convolutional layer 256 and a pooling layer 258, and inputs the feature amount output by the input layer 250 to calculate other feature amounts. These layers have a structure in which a plurality of "nodes" are connected by "edges" and hold a plurality of weight parameters. The values of the weight parameters change as learning progresses. The detector 223 may include a fully connected layer 260 as in the example shown in the (b) part of FIG. 5. The layer configuration of the detector 223 is not limited to the case where the convolutional layer 256 and the pooling layer 258 are repeated one by one, and any layer (for example, the convolutional layer 256) may be included continuously in a plurality. Also, a plurality of fully connected layers 260 may be included continuously.

[0054] <Processing in the intermediate layer> The intermediate layer 252 calculates features through convolution and pooling. The convolution performed in the convolution layer 256 is a process for obtaining a feature map through convolution using a filter, and is responsible for extracting features such as edge extraction from an image. The convolution using this filter generates one channel (one image) of a "feature map" for each filter. The size of the "feature map" is downscaled by the convolution, becoming smaller as convolution is performed in each layer. The pooling process performed in the pooling layer 258 reduces (or expands) the feature map output by the convolution to create a new feature map, and is responsible for providing robustness to the extracted features so that they are not affected by factors such as translation. The intermediate layer 252 can be composed of one or more layers that perform these processes.

[0055] FIG. 6 is a diagram showing the convolution process using a filter. In the first (1st) convolution layer of the intermediate layer 252, a convolution operation is performed between an image set consisting of multiple medical images (a training image set during training, and a recognition image set during recognition such as detection) and a filter F1. The image set is composed of N images (N channels) with an image size of H vertically and W horizontally. When a normal light image is input, the images that make up the image set are images of three channels: R (red), G (green), and B (blue). Since the image set has N channels (N images), for example, in the case of a filter of size 5 (5×5), the filter size of the filter F1 that is convolved with this image set is 5×5×N By performing a convolution operation using this filter F1, a one-channel (one-sheet) "feature map" is generated for each filter F1. If the filter F2 used in the second convolution layer is a filter of size 3 (3 × 3), for example, the filter size will be 3 × 3 × M.

[0056] Similar to the first convolutional layer, the second to nth convolutional layers use filters F2 to F nThe size of the "feature map" in the nth convolutional layer is smaller than the size of the "feature map" in the second convolutional layer because it has been downscaled by the previous convolutional layers or pooling layers.

[0057] Of the layers in the intermediate layer 252, low-level feature extraction (edge ​​extraction, etc.) is performed in the convolutional layers closest to the input side, while higher-level feature extraction (extraction of features related to the shape, structure, etc. of the object) is performed closer to the output side. When segmentation is performed for measurement purposes, upscaling is performed in the latter convolutional layers, and the final convolutional layer obtains a "feature map" of the same size as the input image set. On the other hand, when performing object detection, upscaling is not required since position information only needs to be output.

[0058] The intermediate layer 252 may include a layer that performs batch normalization in addition to the convolutional layer 256 and the pooling layer 258. Batch normalization is a process that normalizes the distribution of data in units of mini-batches when performing learning, and plays a role in speeding up learning, reducing dependency on initial values, and suppressing overlearning.

[0059] <Processing in the output layer> The output layer 254 is a layer that detects the position of the region of interest in the input medical image (normal light image, special light image) based on the feature values ​​output from the intermediate layer 252, and outputs the results. When performing segmentation, the output layer 254 grasps the position of the region of interest in the image at the pixel level using the "feature map" obtained from the intermediate layer 252. In other words, it can detect whether each pixel in the endoscopic image belongs to the region of interest and output the detection result. On the other hand, when performing object detection, no judgment at the pixel level is required, and the output layer 254 outputs the position information of the target object.

[0060] The output layer 254 may execute discrimination (classification) regarding the lesion and output the discrimination result. For example, the output layer 254 may classify the endoscopic image into three categories of "neoplastic", "non-neoplastic", and "others", and output, as the discrimination result, three scores corresponding to "neoplastic", "non-neoplastic", and "others" (the sum of the three scores is 100%), or may output the classification result when it can be clearly classified from the three scores. When outputting the discrimination result, the intermediate layer 252 or the output layer 254 may include a fully connected layer as the last one layer or a plurality of layers (refer to the (b) part of FIG. 5), or may not include it.

[0061] The output layer 254 may output the measurement result of the target region. When measurement is performed by the CNN, the target target region can be segmented as described above, for example, and then measured by the processor 210 or the like based on the result. Also, the measured value of the target target region can be directly output from the detector 223. When directly outputting the measured value, since the measured value itself is learned for the image, it becomes a regression problem of the measured value.

[0062] When using the CNN having the above-described configuration, in the learning process, it is preferable to calculate the loss (error) by comparing the result output by the output layer 254 with the correct answer of recognition for the image set, and perform the process (error backpropagation) of updating the weight parameters in the intermediate layer 252 from the output side layer toward the input side layer so that the loss becomes small.

[0063] <Recognition by a method other than CNN> The detector 223 may perform detection by a method other than CNN. For example, the target region can be detected based on the feature amount of the pixels of the acquired medical image. In this case, the detector 223 divides the detection target image into, for example, a plurality of rectangular regions, sets each divided rectangular region as a local region, calculates the feature amount (for example, hue) of the pixels in the local region for each local region of the detection target image, and determines the local region having a specific hue from among the local regions as the target region. Similarly, the detector 223 may perform classification and measurement based on the feature amount.

[0064] <Modifications of detector configuration> Each detector (pharynx detector 223A to duodenum detector 223D) constituting detector 223 may be composed of a plurality of detectors (for example, a normal light detector and a special light detector) corresponding to observation light of different wavelength bands. In this case, it is preferable that the normal light detector and the special light detector are trained models constructed by machine learning using normal light images and special light images, respectively.

[0065] 5 and 6, the configuration of the detector has been mainly described, but in the present invention, a classifier or a measuring instrument may be provided instead of or in addition to the detector. Also, the detector, classifier, or measuring instrument may be separated into one for normal light and one for special light.

[0066] <Functional implementation using various processors> The functions of the processor 210 described above can be realized using various processors and recording media. The various processors include, for example, a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) to realize various functions. The various processors also include a GPU (Graphics Processing Unit), which is a processor specialized for image processing, and a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacturing. When performing image processing as in the present invention, a configuration using a GPU is effective. Furthermore, dedicated electrical circuits, such as an ASIC (Application Specific Integrated Circuit), which are processors with a circuit configuration designed specifically to execute specific processing, are also included in the "various processors" described above.

[0067] The functions of each unit may be realized by a single processor, or by multiple processors of the same or different types (e.g., multiple FPGAs, a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). Also, multiple functions may be realized by a single processor. Examples of multiple functions configured by a single processor include: a first configuration, as typified by a computer, in which a single processor is configured by combining one or more CPUs and software, and this processor realizes multiple functions; a second configuration, as typified by a system-on-chip (SoC), in which a processor is used to realize the functions of the entire system on a single IC (Integrated Circuit) chip; and various functions are thus configured as hardware structures using one or more of the various processors described above. Furthermore, the hardware structures of these various processors are, more specifically, electrical circuits combining circuit elements such as semiconductor devices. These electrical circuits may be electrical circuits that realize the above-mentioned functions using logical operations such as logical sum, logical product, logical negation, exclusive OR, and combinations of these.

[0068] When the above-mentioned processor or electric circuit executes software (program), the code readable by the computer (for example, various processors and electric circuits constituting the processor 210, and / or a combination thereof) of the software to be executed is stored in a non-transitory recording medium such as a ROM 211 (Read Only Memory), and the computer references the software. The software stored in the non-transitory recording medium includes a medical image processing program for executing the medical image processing method according to the present invention and data used during execution (data used to set the display mode and notification mode, weight parameters used in the detector 223, etc.). The code may be recorded in a non-transitory recording medium such as various types of magneto-optical recording devices or semiconductor memories, instead of the ROM 211. When processing using the software, for example, a RAM 212 (RAM: Random Access Memory, memory) is used as a temporary storage area, and for example, an EEPROM (EEPROM) (not shown) is used. Electrically It is also possible to refer to data stored in an erasable and programmable read-only memory (ERASABLE). The recording unit 207 may be used as a "non-transitory recording medium."

[0069] <Information recorded in the recording unit> The recording unit 207 records endoscopic images (medical images), processed endoscopic images (medical images), imaging information (information indicating the imaging position and / or imaging direction of the endoscopic image), detection results, processing conditions (conditions for performing detection and notification), etc. Other information may also be recorded. The recording control unit 238 records these pieces of information in association with each other.

[0070] <Medical Image Processing> The medical image processing (execution of the medical image processing method and medical image processing program according to the present invention) in the endoscope system 10 configured as described above will now be described. FIG. 7 is a flowchart showing the procedure of the medical image processing method according to the first embodiment. The following describes the case where a region of interest is detected by the detector 223, but similar processing can also be performed when performing classification or measurement. Note that the order of the procedures described below may be changed as necessary.

[0071] <Initial settings> The processor 210 sets conditions necessary for executing the medical image processing method / program based on user operation via the operation unit 208 and / or preset processing conditions (e.g., default processing conditions) (step S100: initial setting step). For example, the processor 210 sets the detector to be operated, the conditions for switching or selecting the detector, and the display and notification mode of the detection results (display / hide setting, the characters, figures, symbols and their colors to be displayed, conditions for audio output, etc.). The processor 210 may operate all of the multiple detectors constituting the detector 223 (in this case, the detection results may be displayed for only some of the detectors), or may operate only some of the detectors and stop the detectors (detection processing) that do not display the detection results on the monitor 400 (display device). The user can set the processing conditions via a screen such as that shown in FIG. 8. In the example of FIG. 8, the user can set the processing conditions by turning radio buttons on / off or entering values ​​into numeric input fields via the operation unit 208. The processor 210 can set the processing conditions not only at the start of processing but also during the execution of the following steps.

[0072] <Acquiring endoscopic images> The image acquisition unit 220 (processor, image acquisition unit) acquires endoscopic images (medical images) captured inside the living body of the subject (step S110: image acquisition processing, image acquisition step). The image acquisition unit 220 can acquire time-series endoscopic images (medical images) in real time by sequentially capturing images of the inside of the living body of the subject at a predetermined frame rate using the imaging unit (photographing lens 132, image sensor 134, AFE 138, etc.) of the endoscope 100. The image acquisition unit 220 may also acquire endoscopic images that have already been captured and recorded in non-real time. For example, the image acquisition unit 220 may acquire endoscopic images recorded in the recording unit 207 or processed endoscopic images, or may acquire images from an external device or system via the communication control unit 205. The display control unit 232 (processor, display control unit) displays the acquired endoscopic images on the monitor 400 (display device) (step S120: display control processing, display control step).

[0073] <Detection of interest areas> The detector 223 (processor) detects a region of interest from an endoscopic image (medical image) using the detector 223 (step S130: region of interest detection process, region of interest detection step). The detector 223 can perform multiple detection processes by using multiple detectors from among the detectors constituting the detector 223. In detecting a region of interest, the detector 223 can grasp the position of the region of interest shown in the image at the pixel level using the above-mentioned "feature map" (i.e., detect whether each pixel in the endoscopic image belongs to the region of interest) and output the detection result. Examples of regions of interest (regions of interest) detected by the endoscope system 10 include polyps, cancer, colonic diverticula, inflammation, treatment scars (EMR scars (Endoscopic Mucosal Resection), ESD scars (Endoscopic Submucosal Dissection), clip locations, etc.), bleeding points, perforations, vascular atypia, and various treatment tools. In the case of an ultrasound device such as an ultrasound endoscope, organs and blood vessels may be detected as the region of interest.

[0074] The processor 210 may detect and notify the region of interest for all frames of the acquired endoscopic image, or may do so intermittently (at predetermined frame intervals).

[0075] Furthermore, it is preferable that the switching control unit 224 (processor) switches the detector whose detection results are to be displayed on the monitor 400 (display device) according to the organ or part to be observed, or the imaging information, etc. (switching process, switching step). If the detector to be switched to is in a stopped state, the switching control unit 224 starts the detection process by that detector. Furthermore, the switching control unit 224 may stop the operation (detection process) of a detector whose detection results are not to be displayed on the monitor 400 (display device). By switching the detector (recognizer) in this way, it is possible to provide the user with an appropriate diagnosis support function (detection results by the detector).

[0076] <Notification when an area of ​​interest is detected> When the detector 223 detects a region of interest (YES in step S140), the first notification unit 234 (processor) causes the monitor 400 (display device) to superimpose information about the detected region of interest on the endoscopic image (medical image) (step S150: first notification processing, first notification step). Furthermore, the second notification unit 236 (processor) causes the speaker 209A (audio output device) to output sound after the superimposed display (first notification processing) (step S160: second notification processing, second notification step). Details and specific aspects of the notification will be described later. The processor 210 repeats the processing of steps S110 to S160 until it determines that "processing should be terminated" due to the completion of acquisition of the endoscopic image or a user operation (YES in step S170).

[0077] <Specific aspects of superimposed display> FIG. 9 is a diagram showing an example of a superimposed display (first notification process). In the figure, an endoscopic image 502 is displayed on a screen 500 of the monitor 400. When a region of interest 504 is detected from the endoscopic image 502, the first notification unit 234 (processor) superimposes at least one of text, graphics, and symbols on the endoscopic image 502 as "information related to the region of interest." For example, as shown in part (a) of FIG. 9, the first notification unit 234 can superimpose an icon 506 (a flag-shaped graphic or symbol) outside the region of the endoscopic image 502. Furthermore, the first notification unit 234 may superimpose a bounding box 508 (graphical symbol) at the position of the region of interest 504 (within the region of the endoscopic image 502) as shown in part (b) of FIG. 9, or may superimpose an arrow 510 (graphical symbol) at a position away from the region of interest 504 as shown in part (c) of the same figure.

[0078] The first notification unit 234 may perform the superimposed display at a position independent of the position of the attention area 504, or may perform the superimposed display at a position corresponding to the position of the attention area 504. For example, in the example shown in part (d) of FIG. 9 , the first notification unit 234 colors an area 512 in the lower right part of the screen 500 in response to the attention area 504 appearing in the lower right part of the endoscopic image 502. When the first notification unit 234 performs the superimposed display at a position corresponding to the position of the attention area 504, it is preferable that when the position of the attention area in the endoscopic image, etc., changes, the position at which the information is superimposed is moved accordingly. Note that the first notification unit 234 may also change the color or brightness when performing the superimposed display.

[0079] <Details of audio notification> Even if some object, such as a region of interest, is detected consecutively, if the same object is not detected consecutively, there is a high possibility of a false positive, and therefore audio output is not considered necessary. Therefore, in the first aspect, the identity of the region of interest is determined as described below, and audio output is performed based on the determination result.

[0080] FIG. 10 is a flowchart showing details of the audio notification (second notification process, second notification step) in step S160. When the first notification process is performed in step S150, the feature amount calculation unit 228 (processor) calculates and stores feature amounts of the region of interest (step S200: feature amount calculation process / feature amount calculation step, feature amount storage process / feature amount storage step). The "feature amount" refers to, for example, type, position, size, shape, color, etc. The feature amount calculation unit 228 can calculate the feature amount based on analysis of the endoscopic image or the output of the detector 223. The feature amount calculation unit 228 may store the calculated feature amount in a temporary storage medium such as the RAM 212, or may store (record) the calculated feature amount in a non-temporary storage medium such as the recording unit 207. Here, "temporarily storing" includes, for example, sequentially deleting the feature amount when the process is completed, deleting the feature amount when the power is turned off, etc.

[0081] The identity determination unit 230 (processor) compares the calculated feature values ​​with the stored feature values ​​to determine the identity of the regions of interest (step S210: identity determination process, identity determination step). The identity determination can be performed based on the number of consecutive detections of the regions of interest. Specifically, the identity determination unit 230 determines the identity of the first region of interest and the second region of interest by comparing the feature values ​​of the region of interest (first region of interest) detected from the endoscopic image (medical image) captured at the first time with the feature values ​​(stored by the feature value calculation unit 228) of the region of interest (second region of interest) detected from the endoscopic image (second medical image) captured at the second time (a time before the first time). The second time can be a predetermined period before the first time, and the user can specify the value of this "predetermined period" via a processing condition setting screen such as that shown in FIG. 8.

[0082] The second notification unit 236 calculates the number of consecutive detections for the first region of interest according to the determination result of the identity determination process. Specifically, if the first region of interest and the second region of interest are identical (YES in step S220), the second notification unit 236 increases the number of consecutive detections for the second region of interest and calculates it as the number of consecutive detections for the first region of interest (step S230: detection number calculation process, detection number calculation step). On the other hand, if the first region of interest and the second region of interest are not identical (NO in step S220), the second notification unit 236 calculates the number of consecutive detections for the first region of interest as a new region of interest. detection The second notification unit 236 may store the calculated number of consecutive detections in a temporary recording medium such as the RAM 212, or may store (record) the calculated number of consecutive detections in a non-temporary recording medium such as the recording unit 207, as in the case of the feature amount described above. Here, "temporarily storing" includes modes such as sequentially deleting the number of consecutive detections when the processing is completed, deleting the number of consecutive detections when the power is turned off, and the like.

[0083] The identity determination unit 230 may determine the identity of the regions of interest based on tracking by the detector 223, overlapping positions of the regions of interest, or optical flow calculated from endoscopic images. "Optical flow" refers to the estimation and vectorization of the movement of parts of an image or the entire subject, based on corresponding points between images, etc.

[0084] Furthermore, the identity determination unit 230 and the second notification unit 236 (processor) may determine identity and calculate the number of consecutive detections for all frames of the endoscopic image, or may perform the determination intermittently. For example, if the frame rate of the endoscopic image is 30 fps (frames per second), the identity determination may be performed at 30 fps or at a rate less than 30 fps (for example, 10 fps). The identity determination can be adjusted to the frame rate of the detector 223.

[0085] When the number of consecutive detections calculated in this manner exceeds the threshold value (YES in step S250), second notification unit 236 outputs sound from speaker 209A (sound output device) (step S260: second notification process, second notification step).

[0086] <Example of audio output (part 1): When the number of consecutive detections exceeds the threshold> FIG. 11 shows a specific example (part 1) of audio output. t0 11, no attention area is detected, and at time t1, an attention area 504 is detected and a bounding box 508 is superimposed (first notification process). The same attention area 504 is also detected at times t2, t3, and t4, and a superimposed display of the bounding box 508 is initiated. In this situation, if the threshold for the number of consecutive detections is three, the number of consecutive detections exceeds the threshold (i.e., the same attention area 504 is detected even after a first period (= t2 - t1) has elapsed since the attention area 504 was detected at time t1), and therefore the second notification unit 236 outputs sound from the speaker 209A. Note that in FIG. 11, a speaker icon 520 indicates that sound will be output (the icon itself does not need to be displayed on the screen 500; the same applies to the following examples). In this way, the second notification unit 236 (processor) outputs sound after the superimposed display.

[0087] <Example of audio output (part 2): When the number of consecutive detections does not exceed the threshold> Fig. 12 is a diagram showing a specific example (part 2) of audio output. In the example shown in Fig. 12, the same region of interest 504 is detected from time t1 to t3, and a bounding box 508 is superimposed and displayed (first notification process). However, at time t4, the region of interest 504 is not detected (i.e., the same region of interest 504 is not detected after a first period has elapsed since the region of interest 504 was detected at time t1). Therefore, the number of consecutive detections (3 times) does not exceed the threshold value (3 times) (NO in step S250), and the second notification unit 236 does not output audio. In Fig. 12, an icon 522 marked with a cross indicates that audio will not be output.

[0088] As described above, false positives (detector 223 determining that a region of interest is not a region of interest) tend to occur momentarily and rarely occur continuously. Therefore, by not outputting audio when a region of interest is detected momentarily (times t1 to t3) as in the example of FIG. 12, it is possible to reduce the possibility that the user will be annoyed by audio output resulting from a false positive. On the other hand, because the region of interest is highlighted in the screen display (first notification process), this leads to a user's attention being drawn to the situation, and is expected to be effective in preventing lesions and the like from being overlooked.

[0089] <Example of audio output (part 3): Determining the identity of the area of ​​interest> Fig. 13 is a diagram showing a specific example (part 3) of audio output when determining the identity of attention regions. In the example of Fig. 13, attention regions (attention regions 504 and 507) are detected from time t1 to t4, and bounding boxes 508 and 509 are superimposed and displayed (first notification process). However, attention region 504 is detected three times (times t1 to t3) and attention region 507 is detected twice (times t3 and t4), so second notification unit 236 does not output audio at time t4. This makes it possible to suppress unnecessary audio output caused by momentary false positives.

[0090] <Example of voice output (part 4): Example of identity judgment period> When detecting a region of interest using AI such as the detector 223, false positives and false negatives (i.e., the AI ​​determines that a region of interest does not exist even though it does exist in the endoscopic image) are unavoidable. For example, as shown in the example of FIG. 14, a situation may occur in which, even if regions of interest (regions of interest 504A and 504B) are detected in the endoscopic image at times t1, t2, and t4, the region of interest (region of interest 504A) is not detected at time t3 due to a judgment error by the detector 223. In this case, a problem may arise in which the number of consecutive detections cannot be calculated appropriately, and sound is not output even though it should be. Therefore, the detection count calculation unit 226, the identity determination unit 230, and the second notification unit 236 (processor) determine the identity of the region of interest not only for the immediately preceding frame but also for regions of interest in earlier frames.

[0091] In the example of FIG. 14 , it is assumed that identity determination unit 230 compares the detection results (presence or absence of a region of interest, feature amounts) between region of interest 504B detected at time t4 and region of interest 504A detected at time t2 and determines that they are “identical.” In this case, detection count calculation unit 226 increases the number of consecutive detections of region of interest 504A. As a result, the number of consecutive detections at time t4 becomes four, exceeding the threshold value (three), so second notification unit 236 outputs a sound from speaker 209A at time t4 (indicated by icon 520 at time t4 in FIG. 14 ). This identity determination avoids the problem of inappropriate calculation of the number of consecutive detections due to false negatives. Note that, since calculating past frames to be compared too far back can increase calculation costs and reduce the accuracy of identity determination, it is preferable to limit the past frames to a time (up to a predetermined period before; time t2 (second time)) close to the current frame (in the example of FIG. 14 , time t4 (first time)).

[0092] 14, the detection count calculation unit 226 and the identity determination unit 230 may perform control such that the number of consecutive detections remains three, but the determination that the same region of interest is being continuously detected continues, rather than counting the number of consecutive detections as "four." In this way, if the number of consecutive detections is maintained without increasing it, when the same region of interest 504A is detected in the frame next to time t4, the number of consecutive detections becomes four, and a notification is made by audio output.

[0093] <Example of audio output (part 5): Example of restricting audio output> In the endoscope system 10, in the notification by audio output (second notification process), control may be performed so that audio output is not generated once the number of consecutive detections exceeds a predetermined number. For example, in the example of FIG. 15 , the number of consecutive detections reaches four at time t4, and audio is output. However, the second notification unit 236 (processor) does not output audio for a predetermined period (three frames until time t7) after time t5, when the number of consecutive detections reaches five. This avoids the problem of frequent audio output being annoying to the user. Note that in the example of FIG. 15 , the audio output mute is released at time t8 after the predetermined period has elapsed. However, if the same target (region of interest) is still detected after the mute is released, audio output may be suppressed again (in this case, audio output is also not performed after time t8). This avoids the problem of frequent audio output being annoying to the user while observing the same target.

[0094] <Example of audio output (part 6): Example of linking the overlay display mode with the audio output state> The endoscope system 10 may change the mode of the screen display (superimposed display: first notification process) depending on the audio output state in the second notification process. For example, in the example of FIG. 16, audio is output after time t4 when the number of consecutive detections reaches four and exceeds the threshold. However, the first notification unit 234 (processor) makes the border of the bounding box 511 superimposed on the attention area 504 thicker than the bounding box 508 from time t1 to t3. Changing the mode of the screen display depending on the audio output state may involve changing the color, size, or shape of the superimposed graphic or the like, or combining it with the superimposed display of other graphics or the like, as in the example of FIG. 9. Such a change in the mode of the screen display allows the user to intuitively understand that the endoscope system 10 is more confidently notifying the detection target. Note that the first notification unit 234 may change the mode of the screen display simultaneously with a change in the audio output state (start / stop of audio output, etc.) (time t4 in the example of FIG. 9) or at a time nearby before or after the change.

[0095] As described above, according to the first embodiment, it is possible to reduce the possibility of missing the region of interest while suppressing unnecessary audio output.

[0096] <Application to other medical images> In the first embodiment described above, a case has been described in which recognition is performed using an endoscopic image (an image obtained by an optical endoscope), which is one type of medical image (medical image). However, the medical image processing device, medical image processing method, and medical image processing program according to the present invention can also be used in an ultrasonic endoscope device (ultrasound endoscope system), an ultrasonic image diagnostic device, etc. Images obtained by The present invention can also be applied to cases where medical images other than endoscopic images are used.

[0097] (Addendum) In addition to the above-described embodiments and modifications, the following configurations are also included within the scope of the present invention.

[0098] (Appendix 1) The medical image analysis processing unit detects an area of ​​interest that is an area that requires attention based on the feature amount of pixels of the medical image, The medical image analysis result acquisition unit is a medical image processing device that acquires the analysis results of the medical image analysis processing unit.

[0099] (Appendix 2) The medical image analysis processing unit detects the presence or absence of a noteworthy object based on the feature amount of pixels of the medical image, The medical image analysis result acquisition unit is a medical image processing device that acquires the analysis results of the medical image analysis processing unit.

[0100] (Appendix 3) The medical image analysis result acquisition unit: Obtained from a recording device that records the results of medical image analysis; The analysis results are a region of interest, which is an area of ​​interest contained in a medical image, and / or the presence or absence of an object of interest.

[0101] (Appendix 4) A medical image processing apparatus in which the medical image is a normal light image obtained by irradiating white light or light of multiple wavelength bands as white light.

[0102] (Appendix 5) A medical image is an image obtained by irradiating light in a specific wavelength range. A medical imaging device, wherein the specific wavelength band is narrower than the white wavelength band.

[0103] (Appendix 6) A medical image processing device in which the specific wavelength band is the blue or green band of the visible range.

[0104] (Appendix 7) A medical image processing device, wherein the specific wavelength band includes a wavelength band of 390 nm to 450 nm or 530 nm to 550 nm, and the light of the specific wavelength band has a peak wavelength within the wavelength band of 390 nm to 450 nm or 530 nm to 550 nm.

[0105] (Appendix 8) The specific wavelength band is the red band of the visible range for medical imaging processing equipment.

[0106] (Appendix 9) A medical image processing device, wherein the specific wavelength band includes a wavelength band of 585 nm or more and 615 nm or less, or a wavelength band of 610 nm or more and 730 nm or less, and the light of the specific wavelength band has a peak wavelength within the wavelength band of 585 nm or more and 615 nm or less, or a wavelength band of 610 nm or more and 730 nm or less.

[0107] (Appendix 10) A medical image processing device, wherein the specific wavelength band includes a wavelength band in which oxygenated hemoglobin and reduced hemoglobin have different absorption coefficients, and the light in the specific wavelength band has a peak wavelength in the wavelength band in which oxygenated hemoglobin and reduced hemoglobin have different absorption coefficients.

[0108] (Appendix 11) A medical image processing device in which the specific wavelength band includes a wavelength band of 400±10 nm, 440±10 nm, 470±10 nm, or a wavelength band of 600 nm or more and 750 nm or less, and the light in the specific wavelength band has a peak wavelength in a wavelength band of 400±10 nm, 440±10 nm, 470±10 nm, or a wavelength band of 600 nm or more and 750 nm or less.

[0109] (Appendix 12) Medical images are in vivo images that show the inside of a living body. In vivo images are medical image processing devices that have information on the fluorescence emitted by fluorescent substances in the living body.

[0110] (Appendix 13) The fluorescence peaks at 390 nm A medical image processing device that obtains images by irradiating the inside of a living body with excitation light of 470 nm or more.

[0111] (Appendix 14) Medical images are in vivo images that show the inside of a living body. The specific wavelength band is a wavelength band of infrared light in a medical imaging device.

[0112] (Appendix 15) A medical image processing device in which the specific wavelength band includes a wavelength band of 790 nm or more and 820 nm or less or a wavelength band of 905 nm or more and 970 nm or less, and the light in the specific wavelength band has a peak wavelength in the wavelength band of 790 nm or more and 820 nm or less or a wavelength band of 905 nm or more and 970 nm or less.

[0113] (Appendix 16) the medical image acquisition unit includes a special light image acquisition unit that acquires a special light image having information of a specific wavelength band based on a normal light image obtained by irradiating white light or light of a plurality of wavelength bands as white light; Medical image processing equipment uses special light for medical images.

[0114] (Appendix 17) A medical image processing device that obtains signals in specific wavelength bands through calculations based on the RGB or CMY color information contained in normal light images.

[0115] (Appendix 18) a feature image generating unit that generates a feature image by calculation based on at least one of a normal light image obtained by irradiating light in a white band or light in a plurality of wavelength bands as white band light, and a special light image obtained by irradiating light in a specific wavelength band; Medical image processing device that processes medical images as feature images.

[0116] (Appendix 19) A medical image processing device according to any one of appendices 1 to 18; an endoscope that acquires an image by irradiating at least one of light in a white wavelength band and light in a specific wavelength band; An endoscope apparatus comprising:

[0117] (Appendix 20) A diagnostic support device comprising the medical image processing device according to any one of appendices 1 to 18.

[0118] (Appendix 21) A medical service support device comprising the medical image processing device according to any one of appendices 1 to 18.

[0119] Although the present invention has been described above with reference to the embodiment and other examples, the present invention is not limited to the above-described aspects, and various modifications are possible within the scope of the present invention. [Explanation of symbols]

[0120] 10 Endoscopy System 100 Endoscope 102 Handheld operation unit 104 Insertion section 106 Universal Cable 108 Light guide connector 112 Soft part 114 Curved section 116 Hard tip 116A Tip side end surface 123 Lighting Department 123A Lighting Lens 123B Lighting Lens 126 Forceps port 130 Photographing optical system 132 Camera Lens 134 Image sensor 136 Drive Circuit 141 Air and water supply button 142 Suction button 143 Function Button 144 Shooting button 170 Light Guide 200 Medical Image Processing Department 202 Image Input Controller 205 Communication control section 206 Video output section 207 Recording Department 208 Operation section 209 Audio Processing Unit 209A Speaker 210 processors 211 ROM 212 RAM 220 Image acquisition unit 222 Attention area detection unit 223 detector 223A Pharyngeal Detector 223B Esophageal detector 223C Gastric detector 223D Duodenal Detector 224 Switching control unit 226 Detection count calculation unit 228 Feature Calculation Unit 230 Identity determination section 232 Display control unit 234 First Notification Department 236 Second Notification Department 238 Recording control section 240 Communication Control Unit 250 input layers 252 Middle Class 254 output layer 256 convolutional layers 258 pooling layers 260 fully connected layer 300 Light source device 310 light source 310B Blue light source 310G green light source 310R red light source 310V purple light source 330 aperture 340 Condenser Lens 350 Light source control unit 400 monitors 500 screens 502 Endoscopic Images 504 Areas of Interest 504A Area of ​​Interest 504B Area of ​​Interest 506 Icons 507 Areas of Interest 508 Bounding Box 509 Bounding Box 510 Arrow 511 Bounding Box 512 areas 520 Icons 522 icons F 1 filter F 2. Filter S100~S260 Steps of medical image processing method

Claims

1. 1. A medical imaging device comprising a processor, The processor: an image acquisition process for acquiring time-series medical images; an attention area detection process for detecting an attention area from the acquired medical image; a display control process for displaying the medical image on a display device; When an area of ​​interest is detected by the area of ​​interest detection process, a first notification process is executed to superimpose information about the detected area of ​​interest on the medical image on the display device; While the first notification process is being performed, it is determined whether the detection is a continuous detection; When it is determined that the detection is the continuous detection, the first notification process and a second notification process of outputting a sound from an audio output device when an attention area is detected in the attention area detection process are executed; When it is determined that the detection is not the continuous detection, the medical image processing device does not execute the second notification process.

2. The processor: When the attention area is detected even after a first period has elapsed since the attention area was detected in the attention area detection process, the detection is determined to be a continuous detection, and the second notification process is executed; The medical image processing device according to claim 1 , wherein if the region of interest is not detected after the first period has elapsed, the detection is determined not to be a continuous detection, and the second notification process is not executed.

3. The medical image processing apparatus according to claim 1 or 2, wherein the processor, in the first notification process, causes the display device to superimpose the information corresponding to the position of the region of interest in the medical image.

4. The processor: executes a detection count calculation process for calculating a number of consecutive detections of the attention area detected in the attention area detection process; The medical image processing device according to claim 1 , wherein, when the number of consecutive detections exceeds a predetermined number, the detection is determined to be a continuous detection, and the second notification process is executed.

5. The processor: a feature amount storing process for storing the feature amount of the detected region of interest; an identity determination process for determining identity between a first region of interest detected from a medical image captured at a first time and a second region of interest detected from a second medical image captured at a second time that is earlier than the first time, by comparing the feature amount of the first region of interest detected from the medical image captured at a first time and the feature amount of the second region of interest detected from the second medical image that is stored and is captured at a second time; Further execute The medical image processing apparatus according to claim 4 , wherein the detection count calculation process calculates the continuous detection count for the first region of interest according to a determination result of the identity determination process.

6. The processor:

6. The medical image processing device according to claim 5, wherein, when the first region of interest and the second region of interest are determined to be identical in the identity determination process, the number of consecutive detections recorded for the second region of interest is increased in the detection count calculation process and then calculated as the number of consecutive detections of the first region of interest.

7. The processor:

7. A medical image processing device according to claim 5 or 6, wherein in the identity determination process, the identity is determined by comparing the feature amounts stored in the feature amount storage process for a time period up to a predetermined period before the first time with the feature amounts of the first region of interest.

8. The processor: The medical image processing apparatus according to claim 1 , wherein, in the second notification process, after the audio output device is caused to output audio, the audio output device is not caused to output audio for a predetermined period of time.

9. The processor: The medical image processing apparatus according to claim 1 , wherein, in the first notification process, a mode of the first notification process is changed depending on a voice output state in the second notification process.

10. The medical image processing apparatus according to claim 1 , wherein the processor causes the display device to superimpose and display at least one of a character, a graphic, and a symbol as the information in the first notification process.

11. A medical image processing device according to any one of claims 1 to 10; an endoscope to be inserted into a subject, the endoscope having an imaging unit that captures the medical image; the display device; the audio output device; An endoscope system comprising:

12. 1. A medical image processing method performed by a medical image processing device having a processor, comprising: The processor: an image acquisition step of acquiring time-series medical images; an attention area detection step of detecting an attention area from the acquired medical image; a display control step of displaying the medical image on a display device; a first notification step of, when a region of interest is detected in the region of interest detection step, causing the display device to superimpose information on the medical image and the detected region of interest; Run While the first notification step is being performed, it is determined whether the detection is a continuous detection; When it is determined that the detection is the continuous detection, the first notification step and a second notification step of outputting a sound from a sound output device when a region of interest is detected in the region of interest detection step are executed, The medical image processing method includes not executing the second notification step if it is determined that the detection is not the continuous detection.

13. A medical image processing program that causes a medical image processing device having a processor to execute a medical image processing method, The medical image processing method includes: an image acquisition step of acquiring time-series medical images; an attention area detection step of detecting an attention area from the acquired medical image; a display control step of displaying the medical image on a display device; a first notification step of, when a region of interest is detected in the region of interest detection step, causing the display device to superimpose information on the medical image and the detected region of interest; Including, The processor: While the first notification step is being performed, it is determined whether the detection is a continuous detection; When it is determined that the detection is the continuous detection, the first notification step and a second notification step of outputting a sound from a sound output device when a region of interest is detected in the region of interest detection step are executed, The medical image processing program does not execute the second notification step when it is determined that the detection is not the continuous detection.

14. A non-transitory computer-readable recording medium having the program according to claim 13 recorded thereon.

Images