Medical image processing device, endoscope system, medical image processing method, medical image processing program, and recording medium

Abstract

The purpose of the present invention is to provide a medical image processing device, an endoscope system, a medical image processing method, and a medical image processing program that are capable of appropriately switching a diagnosis assistance function. A medical image processing device according to an embodiment of the present invention is provided with a processor. The processor performs: an image acquisition process for acquiring time-series medical images; a plurality of detection processes for detecting a region of interest from the acquired medical images; a display control process for causing a display device to display at least one detection result among the detection results of the plurality of detection processes; a selection process for selecting a detection process from among the plurality of detection processes to display the detection result thereof on the display device; and a switch control process for controlling the permission and prohibition of switching of the target detection process of detection result display to the selected detection process on the basis of the detection status of the region of interest in the plurality of detection processes.

Description

[Technical Field]

[0001] The present invention relates to a medical image processing device, an endoscope system, and a medical image processing method. , medicine Medical image processing program , and recording medium The present invention relates to a technique for detecting a region of interest from a medical image. [Background technology]

[0002] It is known that, to assist users such as doctors when observing or diagnosing medical images, the results of detection of an area of ​​interest by a medical image processing device are displayed on a display device. For example, Patent Document 1 describes selecting an area of ​​interest detection unit corresponding to a position indicated by position information from among a plurality of area of ​​interest detection units, detecting an area of ​​interest using the selected area of ​​interest detection unit, and displaying the results. Patent Document 2 also describes a method of switching between detection results and discrimination results for display. It is stated that the [Prior art documents] [Patent documents]

[0003] [Patent Document 1] WO19 / 138773 publication [Patent Document 2] Japanese Patent Application Publication No. 2020-69300 Summary of the Invention [Problem to be solved by the invention]

[0004] In examinations using medical devices such as endoscopes and ultrasound devices, observations may be performed in a series of steps under multiple different conditions. For example, upper endoscopy, which observes different organs such as the pharynx, esophagus, stomach, and duodenum, and ultrasound diagnosis, which observes various organs from various fields of view, may both be performed as part of a series of examinations. Therefore, when providing users with diagnostic support functions for such examinations, the required support functions vary depending on the situation. Furthermore, even if the functions are similar, the optimal recognizer may differ depending on the situation. Therefore, it is preferable to always provide optimal support to users by automatically switching support functions depending on the situation.

[0005] However, with the conventional techniques such as those described in Patent Documents 1 and 2, it is difficult to appropriately switch between diagnostic support functions.

[0006] The present invention has been made in view of the above circumstances, and provides a medical image processing device, an endoscope system, and a medical image processing method that can appropriately switch between diagnostic support functions. , medicine Medical image processing program , and recording medium The purpose is to provide. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, a medical image processing device according to a first aspect of the present invention is a medical image processing device equipped with a processor, which executes an image acquisition process for acquiring time-series medical images, a plurality of detection processes for detecting areas of interest from the acquired medical images, a display control process for displaying at least one of the detection results from the plurality of detection processes on a display device, a selection process for selecting which of the plurality of detection processes the detection result from is to be displayed on the display device, and a switching control process for controlling whether or not to allow switching to the selected detection process of the detection process for displaying the detection results, depending on the detection state of the areas of interest in the plurality of detection processes.

[0008] The detection state of the region of interest is related to the transition of the observation situation (for example, a situation in which the organ or part of the observation target, the viewing direction, etc., is changing). Therefore, by controlling whether to permit or prohibit switching depending on the detection state of the region of interest, it is possible to display the detection result obtained by an appropriate detection process that matches the transition of the observation situation. As a result, according to the first aspect, it is possible to appropriately switch the diagnosis support function (a function that supports the user's diagnosis by displaying the detection result obtained by the appropriate detection process).

[0009] The "plurality of detection processes" in the first aspect may be, for example, multiple detection processes that differ in target organs or parts, observation directions, detection algorithms, image processing parameters, etc. These detection processes may be processes that use detectors configured by machine learning or trained models.

[0010] In the first aspect, the term "acquiring time-series medical images" includes sequentially acquiring a plurality of medical images taken at a predetermined frame rate. The acquisition may be in real time or not.

[0011] The medical image processing device according to the first aspect can be realized, for example, as a processor part of a medical image processing system, but is not limited to such an aspect. Note that "medical images" refer to images obtained as a result of photographing, measuring, etc., of a living body such as a human body for the purpose of diagnosis, treatment, measurement, etc., and examples thereof include endoscopic images, ultrasound images, CT images (CT: Computed Tomography), and MRI images (MRI: Magnetic Resonance Imaging). Medical images are also called medical images.

[0012] In the medical image processing device according to the second aspect, in the first aspect, the processor, in the switching control process, does not permit switching near the time when it is determined that a region of interest exists in the medical image in at least one of the plurality of detection processes. For example, the processor can perform control such that switching is not permitted until a predetermined time has elapsed since it was determined that "a region of interest exists in the medical image."

[0013] In the medical image processing device according to the third aspect, in the first aspect, the processor permits switching in the switching control process near the time when it is determined that no region of interest exists in the medical image in at least one of the plurality of detection processes. For example, the processor can perform control such that switching is permitted until a predetermined time has elapsed since it was determined that "no region of interest exists in the medical image."

[0014] In the medical image processing device according to the fourth aspect, in the first aspect, the processor stops the detection process that does not cause the display device to display the detection result in the switching control process, thereby reducing the processing load and power consumption.

[0015] A medical image processing device according to a fifth aspect is the fourth aspect, wherein the processor starts the detection process switched to a state in which the detection result is displayed in the switching control process.

[0016] In a medical image processing device according to a sixth aspect, in any one of the first to fifth aspects, the processor further executes a notification process for notifying a user of which detection process the detection results of which are displayed on the display device are from. According to the sixth aspect, the user can know which detection process the detection results of which are displayed are from. The processor can notify the user by displaying on the display device, outputting audio, etc.

[0017] In a medical image processing device according to a seventh aspect, in any one of the first to sixth aspects, the processor acquires information indicating the imaging position and / or imaging direction of the medical image from the medical image, and makes the selection based on the acquired information. In the seventh aspect, the processor may acquire the information using a detector configured by machine learning.

[0018] In a medical image processing device according to an eighth aspect, in any one of the first to seventh aspects, the processor acquires information indicating the imaging position and / or imaging direction of the medical image from a determination device that determines the state of the imaging device that captures the medical image, and makes a selection based on the acquired information. In the eighth aspect, the processor can determine the state of the imaging device based on information acquired from a determination device (external device) connected to the medical image processing device.

[0019] In a medical image processing device according to a ninth aspect, in any one of the first to eighth aspects, the processor changes the display mode of the detection results to be displayed on the display device in accordance with the detection process in the display control process. According to the ninth aspect, the user can easily understand which detection process the detection results are being displayed by.

[0020] A medical image processing device according to a tenth aspect is any one of the first to ninth aspects, wherein the processor notifies the user that the detector for displaying the detection result has been switched in the switching control process. The processor can notify the user by displaying on a display device, outputting audio, etc.

[0021] In order to achieve the above-mentioned object, an endoscopic system according to an eleventh aspect of the present invention comprises a medical image processing device according to any one of the first to tenth aspects, an endoscope that is inserted into a subject and has an imaging unit that sequentially captures medical images, and a display device. Because the endoscopic system according to the eleventh aspect comprises the medical image processing device according to any one of the first to tenth aspects, it is possible to appropriately switch between diagnostic support functions.

[0022] In order to achieve the above-mentioned object, a medical image processing method according to a twelfth aspect of the present invention is a medical image processing method executed by a medical image processing device equipped with a processor, and includes an image acquisition step of acquiring time-series medical images, a detection step of detecting an area of ​​interest from the acquired medical images using multiple detectors, a display control step of displaying the detection results for at least one of the detectors from the detection results of the detection step on a display device, a selection step of selecting which of the multiple detection processes the detection results of which will be displayed on the display device, and a switching control step of controlling whether or not to allow switching to the selected detection process of the detection process for displaying the detection results, depending on the detection state of the area of ​​interest in the multiple detection processes.

[0023] According to the twelfth aspect, similarly to the first aspect, it is possible to appropriately switch between diagnostic support functions. Note that the medical image processing method according to the twelfth aspect may further include the same configuration as the second to tenth aspects.

[0024] In order to achieve the above-mentioned object, a medical image processing program according to a thirteenth aspect of the present invention is a medical image processing program that causes a medical image processing device having a processor to execute a medical image processing method, and the medical image processing method includes an image acquisition process that acquires time-series medical images, a detection process that detects areas of interest from the acquired medical images using multiple detectors, a display control process that displays the detection results for at least one of the detectors from the detection results of the detection process on a display device, a selection process that selects which of the multiple detection processes the detection results of which are to be displayed on the display device, and a switching control process that controls whether or not to switch the detection process for which the detection results are to be displayed to the selected detection process depending on the detection state of the areas of interest in the multiple detection processes.

[0025] According to the thirteenth aspect, similarly to the first and twelfth aspects described above, it is possible to appropriately switch between diagnostic support functions. Note that the medical image processing program according to the thirteenth aspect may be a program that further executes processing similar to that of the second to tenth aspects. Furthermore, a non-transitory recording medium on which computer-readable code of the programs of these aspects is recorded can also be cited as an aspect of the present invention. [Effects of the Invention]

[0026] As described above, the medical image processing device, endoscope system, medical image processing method, and medical image processing program according to the present invention can appropriately switch between diagnostic support functions. [Brief explanation of the drawings]

[0027] [Figure 1] FIG. 1 is an external view of an endoscope system according to a first embodiment. [Figure 2] FIG. 2 is a diagram showing the configuration of the main parts of the endoscope system. [Figure 3] FIG. 3 is a block diagram showing the functional configuration of the processor. [Figure 4] FIG. 4 is a diagram showing an example of the layer configuration of the detector. [Figure 5] FIG. 5 is a diagram showing the convolution operation in the detector. [Figure 6] FIG. 6 is a diagram showing a modified example of the configuration of the detector. [Figure 7] FIG. 7 is a diagram showing an example in which a recognizer is composed of a detector and a classifier. [Figure 8] FIG. 8 is a flowchart showing the procedure of the medical image processing method according to the first embodiment. [Figure 9] FIG. 9 is a diagram showing an example of a setting screen for processing conditions. [Figure 10] FIG. 10 is a diagram showing an example of switching control in the vicinity of the timing at which the region of interest is detected. [Figure 11] FIG. 11 is a diagram showing an example of display and notification of the detection result of the region of interest. [Figure 12] FIG. 12 is a diagram showing another example of display and notification of the detection result of the region of interest. [Figure 13] FIG. 13 is a diagram showing an example of notification in a state where the attention area has not been detected. [Figure 14] FIG. 14 is a diagram showing yet another example of display and notification of the detection result of the region of interest. [Figure 15] FIG. 15 is a diagram showing how the position and / or shape of an endoscope is measured using an endoscope shape measuring device. DETAILED DESCRIPTION OF THE INVENTION

[0028] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, with reference to the accompanying drawings, embodiments of a medical image processing device, an endoscope system, a medical image processing method, and a medical image processing program according to the present invention will be described in detail.

[0029] First Embodiment <Configuration of endoscope system> Fig. 1 is an external view of an endoscopic system 10 (medical image processing device, endoscopic system) according to a first embodiment, and Fig. 2 is a block diagram showing the configuration of the main parts of the endoscopic system 10. As shown in Figs. 1 and 2, the endoscopic system 10 is composed of an endoscope 100 (endoscopic scope, imaging device), a medical image processing unit 200 (medical image processing device, processor), a light source device 300 (light source device), and a monitor 400 (display device). An external device (determination device) that determines the state of the endoscope 100 using electromagnetic waves, ultrasound, or magnetism may be connected to the endoscopic system 10 (see the example of Fig. 15 described later).

[0030] <Configuration of an endoscope> The endoscope 100 includes a handheld control unit 102 and an insertion section 104 connected to the handheld control unit 102. The surgeon (user) holds and operates the handheld control unit 102, and inserts the insertion section 104 into the body of a subject for observation. The handheld control unit 102 is also provided with an air / water supply button 141, a suction button 142, function buttons 143 to which various functions can be assigned, and an image capture button 144 for receiving image capture instructions (still images, moving images). The insertion section 104 is composed of, in order from the handheld control unit 102 side, a flexible section 112, a bending section 114, and a rigid tip section 116. That is, the bending section 114 is connected to the proximal end side of the rigid tip section 116, and the flexible section 112 is connected to the proximal end side of the bending section 114. The handheld control unit 102 is connected to the proximal end side of the insertion section 104. The user can bend the bending section 114 by operating the hand operation section 102, thereby changing the orientation of the tip rigid section 116 up, down, left, or right. The tip rigid section 116 is provided with an imaging optical system 130, an illumination section 123, a forceps port 126, etc. (see FIGS. 1 and 2).

[0031] During observation or treatment, white light (normal light) and / or narrowband light (special light: for example, one or more of red narrowband light, green narrowband light, blue narrowband light, and violet narrowband light) can be emitted from illumination lenses 123A and 123B of illumination unit 123 by operating operation unit 208 (see FIG. 2). Furthermore, by operating air / water supply button 141, cleaning water is discharged from a water supply nozzle (not shown) to clean photographing lens 132 (photographing lens, imaging unit) of photographing optical system 130 and illumination lenses 123A and 123B. A conduit (not shown) is connected to forceps port 126, which opens at tip rigid portion 116. A treatment tool (not shown) for tumor removal or the like is inserted into this conduit and can be moved forward and backward as needed to perform the necessary treatment on the subject.

[0032] As shown in FIGS. 1 and 2, a photographing lens 132 (image capturing unit) is disposed on the distal end surface 116A of the distal end rigid portion 116. A CMOS (Complementary Metal-Oxide Semiconductor) type image capturing element 134 (image capturing element, image capturing unit), a driver circuit 136, and an AFE 138 (AFE: Analog Front End, image capturing unit) are disposed behind the photographing lens 132, and these elements output an image signal. The image capturing element 134 is a color image capturing element and includes a plurality of pixels each composed of a plurality of light receiving elements arranged in a matrix (two-dimensional array) in a specific pattern array (Bayer array, X-Trans (registered trademark) array, honeycomb array, etc.). Each pixel of the image capturing element 134 includes a microlens, a red (R), green (G), or blue (B) color filter, and a photoelectric conversion unit (e.g., a photodiode). The photographing optical system 130 can generate a color image from pixel signals of three colors, red, green, and blue, or can generate an image from pixel signals of any one or two colors of red, green, and blue. Although the first embodiment describes a case where the image sensor 134 is a CMOS-type image sensor, the image sensor 134 may also be a CCD (Charge Coupled Device) type. Each pixel of the image sensor 134 may further include a purple color filter corresponding to a purple light source 310V and / or an infrared filter corresponding to an infrared light source.

[0033] An optical image of the subject is formed on the light receiving surface (imaging surface) of the imaging element 134 by the photographing lens 132, converted into an electrical signal, and output to the medical image processing unit 200 via a signal cable (not shown), where it is converted into a video signal. As a result, an endoscopic image is displayed on the monitor 400 connected to the medical image processing unit 200.

[0034] Furthermore, illumination lenses 123A and 123B of the illumination unit 123 are provided adjacent to the photographing lens 132 on the distal end surface 116A of the distal end hard portion 116. An exit end of a light guide 170 (described later) is disposed behind the illumination lenses 123A and 123B. This light guide 170 is inserted through the insertion portion 104, the handheld operation unit 102, and the universal cable 106, and the entrance end of the light guide 170 is disposed inside the light guide connector 108.

[0035] The handheld operation unit 102 may include a scope information recording unit (not shown) that records individual information (individual information, scope information) of the endoscope 100. The individual information includes, for example, the type of the endoscope 100 (forward viewing, side viewing, etc.), the model, the individual identification number, and the characteristics of the optical system (field of view, distortion, etc.). The processor 210 (scope information acquisition unit, individual information acquisition unit) can acquire this individual information and use it in medical image processing. The scope information recording unit may be provided in the light guide connector 108.

[0036] In the endoscope system 10, the endoscope 100 having the above-described configuration is used to sequentially capture images of the subject at a determined frame rate (this can be done under the control of the imaging unit and image acquisition unit 220 (see FIG. 3)), thereby making it possible to sequentially acquire time-series medical images. The user performs observations by inserting and withdrawing the endoscope 100 (insertion unit 104) into and from the living body of the subject.

[0037] <Configuration of light source device> 2, light source device 300 is composed of an illumination light source 310, an aperture 330, a condenser lens 340, a light source control unit 350, etc., and causes observation light to enter light guide 170. Light source 310 is equipped with red light source 310R, green light source 310G, blue light source 310B, and purple light source 310V, which emit narrowband light of red, green, blue, and purple, respectively, and is capable of emitting narrowband light of red, green, blue, and purple. The illuminance of the observation light emitted by light source 310 is controlled by light source control unit 350, which can change (increase or decrease) the illuminance of the observation light or stop illumination as necessary.

[0038] The light source 310 can emit narrowband light of red, green, blue, and purple in any combination. For example, it can simultaneously emit narrowband light of red, green, blue, and purple to irradiate white light (normal light) as observation light, or it can emit narrowband light (special light) by emitting one or two of the colors. The light source 310 may further include an infrared light source that irradiates infrared light (an example of narrowband light). Alternatively, white light or narrowband light may be irradiated as observation light by using a light source that irradiates white light and a filter that transmits the white light and each narrowband light.

[0039] <Light source wavelength band> The light source 310 may be a light source that generates white light or light of multiple wavelength bands as white light, or a light source that generates light of a specific wavelength band narrower than the white wavelength band. The specific wavelength band may be the blue or green band in the visible range, or the red band in the visible range. When the specific wavelength band is the blue or green band in the visible range, A specific wavelength band is It includes a wavelength band of 390 nm or more and 450 nm or less, or 530 nm or more and 550 nm or less, and Light in a specific wavelength band The peak wavelength may be within a wavelength band of 390 nm or more and 450 nm or less, or 530 nm or more and 550 nm or less. In addition, if the specific wavelength band is the red band of the visible range, A specific wavelength band is The wavelength band may include 585 nm or more and 615 nm or less, or 610 nm or more and 730 nm or less, and the light in the specific wavelength band may have a peak wavelength within the wavelength band of 585 nm or more and 615 nm or less, or 610 nm or more and 730 nm or less.

[0040] The specific wavelength band may include a wavelength band in which the absorption coefficients of oxygenated hemoglobin and reduced hemoglobin differ, and the light in the specific wavelength band may have a peak wavelength in the wavelength band in which the absorption coefficients of oxygenated hemoglobin and reduced hemoglobin differ. In this case, the specific wavelength band may include a wavelength band of 400±10 nm, 440±10 nm, 470±10 nm, or a wavelength band of 600 nm to 750 nm, and the light in the specific wavelength band may have a peak wavelength in a wavelength band of 400±10 nm, 440±10 nm, 470±10 nm, or a wavelength band of 600 nm to 750 nm.

[0041] Furthermore, the wavelength band of the light generated by light source 310 may include a wavelength band of 790 nm or more and 820 nm or less, or a wavelength band of 905 nm or more and 970 nm or less, and the light generated by light source 310 may have a peak wavelength in the wavelength band of 790 nm or more and 820 nm or less, or 905 nm or more and 970 nm or less.

[0042] Furthermore, the light source 310 may be equipped with a light source that emits excitation light having a peak wavelength of 390 nm or more and 470 nm or less. In this case, it is possible to acquire medical images (medical images, in vivo images) containing information on the fluorescence emitted by fluorescent substances in the subject (living body). When acquiring a fluorescent image, a dye agent for fluoroscopy (fluorescein, acridine orange, etc.) may be used.

[0043] The type of light source (laser light source, xenon light source, LED light source (LED: Light-Emitting Diode), etc.), wavelength, presence or absence of a filter, etc. of the light source 310 are preferably configured according to the type, part, organ, and purpose of observation of the subject, and it is also preferable to combine and / or switch the wavelength of the observation light during observation according to the type, part, organ, and purpose of observation of the subject. When switching wavelengths, the wavelength of the irradiated light may be switched, for example, by rotating a disk-shaped filter (rotary color filter) placed in front of the light source and equipped with a filter that transmits or blocks light of a specific wavelength.

[0044] Furthermore, the imaging element used in carrying out the present invention is not limited to a color imaging element such as the imaging element 134, in which a color filter is provided for each pixel, but may also be a monochrome imaging element. When a monochrome imaging element is used, imaging can be performed in a frame sequential (color sequential) manner by sequentially switching the wavelength of the observation light. For example, the wavelength of the emitted observation light may be sequentially switched between (purple, blue, green, red), or broadband light (white light) may be irradiated and the wavelength of the emitted observation light may be switched using a rotary color filter (red, green, blue, purple, etc.). Alternatively, one or more narrowband lights (green, blue, purple, etc.) may be irradiated and the wavelength of the emitted observation light may be switched using a rotary color filter (green, blue, purple, etc.). The narrowband light may be infrared light of two or more different wavelengths.

[0045] By connecting the light guide connector 108 (see Figures 1 and 2) to the light source device 300, the observation light emitted from the light source device 300 is transmitted to the illumination lenses 123A and 123B via the light guide 170, and is then irradiated onto the observation range from the illumination lenses 123A and 123B.

[0046] <Configuration of medical image processing unit> The configuration of the medical image processing unit 200 will be described with reference to FIG. 2. The medical image processing unit 200 receives an image signal output from the endoscope 100 via an image input controller 202, performs necessary image processing in a processor 210 (image acquisition unit 220: processor, computer, medical image processing device), and outputs the image via a video output unit 206. This results in an observation image (medical image) being displayed on a monitor 400 (display device). The communication control unit 205 controls communication with an in-hospital system (HIS: Hospital Information System), an in-hospital LAN (Local Area Network), and / or external systems and networks (not shown). The recording unit 207 (recording device) records images of the subject (endoscopic image, medical image, medical image), processing conditions, imaging information, information indicating the detection results of the region of interest, and the like. The audio processing unit 209 can output messages (audio) related to the detection results and notification processing from a speaker 209A under the control of the processor 210.

[0047] Furthermore, ROM 211 (Read Only Memory) is a non-volatile storage element (non-temporary recording medium) that stores computer-readable code of programs that cause the processor 210 to execute various image processing methods. RAM 212 (Random Access Memory) is a storage element for temporary storage during various processes, and can also be used as a buffer when acquiring images.

[0048] The user can issue instructions to execute medical image processing and specify the conditions required for execution via the operation unit 208 (see Figure 9), and the display control unit 230 (see Figure 3) can display the screen when issuing these instructions, the detection results of the area of ​​interest, etc. on the monitor 400.

[0049] <Processor functions> FIG. 3 is a block diagram showing the functional configuration of the processor 210. The processor 210 includes an image acquisition unit 220 (image acquisition unit), an imaging information acquisition unit 222 (imaging information acquisition unit), a detector 224 (multiple detectors), a selection unit 226 (selection unit), a switching control unit 228 (switching control unit), a display control unit 230 (display control unit), a notification processing unit 232 (notification processing unit), and a recording control unit 234 (recording control unit). The detector 224 detects a region of interest from a medical image, and in the embodiment of FIG. 3, includes multiple detectors (pharynx detector 224A, esophagus detector 224B, stomach detector 224C, and duodenum detector 224D) corresponding to different organs and regions. These detectors can be configured, for example, by a hierarchical neural network such as a convolutional neural network (CNN), as will be described later with reference to FIGS. 4 and 5.

[0050] The processor 210 (image acquisition unit 220, etc.) can use the above-described functions to calculate features of a medical image, emphasize or reduce components in specific frequency bands, and emphasize or reduce the prominence of specific targets (regions of interest, blood vessels at a desired depth, etc.). The processor 210 may also include a special light image acquisition unit that acquires a special light image having information about a specific wavelength band based on a normal light image acquired by irradiating the image with white light or light in multiple wavelength bands as white light. In this case, the signal in the specific wavelength band can be obtained by calculation based on RGB (R: red, G: green, B: blue) or CMY (C: cyan, M: magenta, Y: yellow) color information contained in the normal light image. The processor 210 may also include a feature image generation unit that generates a feature image by calculation based on at least one of a normal light image acquired by irradiating the image with white light or light in multiple wavelength bands as white light, and a special light image acquired by irradiating the image with light in a specific wavelength band, and acquire and display the feature image as a medical image.

[0051] In addition, the image acquisition unit 220 (processor) acquires an endoscopic image (medical image) taken with observation light in a wavelength band corresponding to the organ or part to be observed as a medical image, and the display control unit 230 may display the recognition result for the medical image taken with the observation light in that wavelength band on the monitor 400 (display device). For example, in the case of the stomach, an image taken with white light (normal light) can be used for detection (recognition), and in the case of the esophagus, an image taken with special light (blue narrow-band light) such as BLI (Blue Laser Imaging: registered trademark) can be used. The image acquisition unit 220 may acquire an image taken with special light such as LCI (Linked Color Imaging: registered trademark) and subjected to image processing (in the case of LCI, the chroma difference and color phase difference of colors close to the mucosal color are expanded) according to the part.

[0052] Regarding the medical image processing using the functions described above, the details will be described later.

[0053] <Detector using a trained model> The detector described above can be configured using a trained model (a model trained using a set of images composed of images of a living body) configured by machine learning, such as a CNN (Convolutional Neural Network) or an SVM (Support Vector Machine). Hereinafter, the layer configuration in the case of configuring the detector 224 (pharynx detector 224A to duodenum detector 224D) by a CNN will be described.

[0054] <Example of the layer configuration of a CNN> FIG. 4 is a diagram illustrating an example of the layer configuration of the detector 224. In the example shown in part (a) of FIG. 4, the detector 224 includes an input layer 250, an intermediate layer 252, and an output layer 254. The input layer 250 receives an endoscopic image (medical image) acquired by the image acquisition unit 220 and outputs features. The intermediate layer 252 includes a convolutional layer 256 and a pooling layer 258, and receives the features output by the input layer 250 and calculates other features. These layers have a structure in which multiple "nodes" are connected by "edges," and hold multiple weight parameters. The values ​​of the weight parameters change as learning progresses. The detector 224 may also include a fully connected layer 260, as shown in part (b) of FIG. 4. The layer configuration of the detector 224 is not limited to a case in which the convolutional layer 256 and the pooling layer 258 are repeated one by one, but may include multiple consecutive layers (e.g., convolutional layers 256). Furthermore, multiple consecutive fully connected layers 260 may also be included.

[0055] <Processing in the middle layer> The intermediate layer 252 calculates features through convolution and pooling. The convolution performed in the convolution layer 256 is a process for obtaining a feature map through convolution using a filter, and is responsible for extracting features such as edge extraction from an image. The convolution using this filter generates one channel (one image) of a "feature map" for each filter. The size of the "feature map" is downscaled by the convolution, becoming smaller as convolution is performed in each layer. The pooling process performed in the pooling layer 258 reduces (or expands) the feature map output by the convolution to create a new feature map, and is responsible for providing robustness to the extracted features so that they are not affected by factors such as translation. The intermediate layer 252 can be composed of one or more layers that perform these processes.

[0056] FIG. 5 illustrates the convolution operation in the detector 224 shown in FIG. 4. In the first convolutional layer of the hidden layer 252, a convolution operation is performed between a filter F1 and an image set consisting of multiple medical images (a training image set during training, and a recognition image set during recognition such as detection). The image set consists of N images (N channels) with an image size of H vertically and W horizontally. When a normal light image is input, the images constituting the image set are images with three channels: R (red), G (green), and B (blue). The filter F1 convolved with this image set has N channels (N images). For example, a filter of size 5 (5 × 5) has a filter size of 5 × 5 × N. The convolution operation using this filter F1 generates a one-channel (one-image) "feature map" for each filter F1. The filter F2 used in the second convolutional layer has a filter size of 3 × 3 × M, for example, a filter of size 3 (3 × 3).

[0057] Similar to the first convolutional layer, the second to nth convolutional layers use filters F2 to F n The size of the "feature map" in the nth convolutional layer is smaller than the size of the "feature map" in the second convolutional layer because it has been downscaled by the previous convolutional layers or pooling layers.

[0058] Of the layers in the intermediate layer 252, low-level feature extraction (edge ​​extraction, etc.) is performed in the convolutional layers closest to the input side, while higher-level feature extraction (extraction of features related to the shape, structure, etc. of the object) is performed closer to the output side. When segmentation is performed for measurement purposes, upscaling is performed in the latter convolutional layers, and the final convolutional layer obtains a "feature map" of the same size as the input image set. On the other hand, when performing object detection, upscaling is not required since position information only needs to be output.

[0059] The intermediate layer 252 may include a layer that performs batch normalization in addition to the convolutional layer 256 and the pooling layer 258. Batch normalization is a process that normalizes the distribution of data in units of mini-batches when performing learning, and plays a role in speeding up learning, reducing dependency on initial values, and suppressing overlearning.

[0060] <Processing in the output layer> The output layer 254 is a layer that detects the position of the region of interest in the input medical image (normal light image, special light image) based on the feature values ​​output from the intermediate layer 252, and outputs the results. When performing segmentation, the output layer 254 grasps the position of the region of interest in the image at the pixel level using the "feature map" obtained from the intermediate layer 252. In other words, it can detect whether each pixel in the endoscopic image belongs to the region of interest and output the detection result. On the other hand, when performing object detection, no judgment at the pixel level is required, and the output layer 254 outputs the position information of the target object.

[0061] The output layer 254 may perform a classification of the lesion and output the classification result. 254 may classify endoscopic images into three categories: "neoplastic," "non-neoplastic," and "other," and output the discrimination results as three scores corresponding to "neoplastic," "non-neoplastic," and "other" (the sum of the three scores is 100%), or may output the classification results if a clear classification is possible from the three scores. Note that when the discrimination results are output, the intermediate layer 252 or the output layer 254 may or may not include a fully connected layer as the last layer or layers (see part (b) of Figure 4).

[0062] The output layer 254 may output the measurement result of the target area. When measurement is performed by a CNN, the target area of interest can be segmented as described above, for example, and then measured by the processor 210 or the like based on the result. Also, the measured value of the target area of interest can be directly output from the detector 224. When directly outputting the measured value, since the measured value itself is learned for the image, it becomes a regression problem of the measured value.

[0063] When using a CNN having the above-described configuration, in the process of learning, it is preferable to calculate a loss (error) by comparing the result output by the output layer 254 with the correct answer of recognition for the image set, and perform a process (error backpropagation) of updating the weight parameters in the intermediate layer 252 from the output side layer toward the input side layer so that the loss becomes small.

[0064] <Recognition by Methods Other than CNN> The detector 224 may perform detection by a method other than CNN. For example, the target area can be detected based on the feature amount of the pixels of the acquired medical image. In this case, the detector 224 divides the detection target image into, for example, a plurality of rectangular areas, sets each of the divided rectangular areas as a local area, calculates the feature amount (for example, hue) of the pixels in the local area for each local area of the detection target image, and determines a local area having a specific hue from among the local areas as the target area. Similarly, the detector 224 may perform classification or measurement based on the feature amount.

[0065] <Modified Example of Detector Configuration> Each detector (the pharynx detector 224A to the duodenum detector 224D) constituting the detector 224 may be composed of a plurality of detectors corresponding to observation lights in different wavelength bands. FIG. 6 is a diagram showing a modified example of the detector configuration (an example in which the gastric detector 224C includes a normal light detector 224C1 and a special light detector 224C2). The normal light detector 224C1 and the special light detector 224C2 are preferably learned models configured by machine learning using a normal light image and a special light image, respectively.

[0066] 4 to 6 have mainly described the configuration of the detectors, but in the present invention, classifiers and measuring instruments may be provided instead of or in addition to the detectors. For example, as shown in FIG. 7, the pharynx recognizer 224E may be composed of a pharynx detector 224E1 and a pharynx classifier 224E2, the esophagus recognizer 224F may be composed of an esophagus detector 224F1 and an esophagus classifier 224F2, the stomach recognizer 224G may be composed of a stomach detector 224G1 and a stomach classifier 224G2, and the duodenum recognizer 224H may be composed of a duodenum detector 224H1 and a duodenum classifier 224H2. These classifiers (discriminators) and measuring instruments may also have the same layer structure as the detectors described above. Furthermore, the detectors, classifiers, or measuring instruments may be separated into those for normal light and those for special light, as in the example of FIG. 6.

[0067] <Functional implementation using various processors> The functions of the processor 210 described above can be realized using various processors and recording media. The various processors include, for example, a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) to realize various functions. The various processors also include a GPU (Graphics Processing Unit), which is a processor specialized for image processing, and a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacturing. When performing image processing as in the present invention, a configuration using a GPU is effective. Furthermore, dedicated electrical circuits, such as an ASIC (Application Specific Integrated Circuit), which are processors with a circuit configuration designed specifically to execute specific processing, are also included in the "various processors" described above.

[0068] The functions of each unit may be realized by a single processor, or by multiple processors of the same or different types (e.g., multiple FPGAs, a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). Also, multiple functions may be realized by a single processor. Examples of multiple functions configured by a single processor include: a first configuration, as typified by a computer, in which a single processor is configured by combining one or more CPUs and software, and this processor realizes multiple functions; a second configuration, as typified by a system-on-chip (SoC), in which a processor is used to realize the functions of the entire system on a single IC (Integrated Circuit) chip; and various functions are thus configured as hardware structures using one or more of the various processors described above. Furthermore, the hardware structures of these various processors are, more specifically, electrical circuits combining circuit elements such as semiconductor devices. These electrical circuits may realize the above-mentioned functions using logical operations such as logical sum, logical product, logical negation, exclusive OR, and combinations of these.

[0069] When the above-mentioned processor or electric circuit executes software (program), the code readable by the computer (for example, various processors and electric circuits constituting the processor 210, and / or a combination thereof) of the software to be executed is stored in a non-transitory recording medium such as a ROM 211 (Read Only Memory), and the computer references the software. The software stored in the non-transitory recording medium includes a medical image processing program for executing the medical image processing method according to the present invention and data used during execution (data used to set the display mode and notification mode, weight parameters used in the detector 224, etc.). The code may be recorded in a non-transitory recording medium such as various types of magneto-optical recording devices or semiconductor memories, instead of the ROM 211. When processing using the software, for example, a RAM 212 (RAM: Random Access Memory) is used as a temporary storage area, and for example, an EEPROM (EEPROM) (not shown) is used as a temporary storage area. Electrically It is also possible to refer to data stored in an erasable and programmable read-only memory (ERASABLE). The recording unit 207 may be used as a "non-transitory recording medium."

[0070] <Information recorded in the recording unit> The recording unit 207 records endoscopic images (medical images), processed endoscopic images (medical images), imaging information (information indicating the imaging position and / or imaging direction of the endoscopic image), detection results, processing conditions (conditions for performing detection and notification), etc. Other information may also be recorded. The recording control unit 234 records these pieces of information in association with each other.

[0071] <Medical Image Processing> The medical image processing (execution of the medical image processing method and medical image processing program according to the present invention) in the endoscope system 10 configured as described above will now be described. FIG. 8 is a flowchart showing the procedure of medical image processing according to the first embodiment. The following describes the case where a region of interest is detected by the detector 224, but similar processing can also be performed when performing classification or measurement. Note that the procedure described below is an example, and the order may be changed as necessary.

[0072] <Initial settings> The processor 210 sets conditions necessary for executing the medical image processing method / program based on user operation via the operation unit 208 and / or preset processing conditions (e.g., default processing conditions) (step S100: initial setting step). For example, the processor 210 specifies the detectors to be operated and sets the display and notification mode of the detection results (display / non-display setting, characters, figures, symbols, and their colors to be displayed, etc.). The processor 210 may operate all of the multiple detectors constituting the detector 224 (in this case, the detection results may be displayed for only some of the detectors), or may operate only some of the detectors and stop the detectors (detection processing) that do not display the detection results on the monitor 400 (display device). The user can set the processing conditions by turning radio buttons on / off or entering numerical values ​​via the operation unit 208 on a screen such as that shown in FIG. 9 (which does not show the settings of all processing conditions). The processor 210 can set the processing conditions not only at the start of processing but also during the execution of the following steps.

[0073] <Acquiring endoscopic images> The image acquisition unit 220 (processor, image acquisition unit) acquires endoscopic images (medical images) captured inside the living body of the subject (step S110: image acquisition processing, image acquisition step). The image acquisition unit 220 can acquire time-series endoscopic images in real time by sequentially capturing images of the inside of the living body of the subject at a predetermined frame rate using the imaging unit (photographing lens 132, image sensor 134, AFE 138, etc.) of the endoscope 100. The image acquisition unit 220 may also acquire endoscopic images that have already been captured and recorded in non-real time. For example, the image acquisition unit 220 may acquire endoscopic images recorded in the recording unit 207 or processed endoscopic images, or may acquire images from an external device or system via the communication control unit 205. The display control unit 230 (processor, display control unit) displays the acquired endoscopic images on the monitor 400 (step S112: display control processing, display control step).

[0074] <Acquisition of imaging information> The imaging information acquisition unit 222 (processor) acquires imaging information (information indicating the imaging position and / or imaging direction of the endoscopic image) (step S114: imaging information acquisition processing, imaging information acquisition step). The imaging position may be specified in organ units (pharynx, esophagus, stomach, duodenum, etc.), or may be specified by further dividing one organ into smaller parts (in the case of the stomach, the cardia, fundus, body, pylorus, pylorus, etc.). The imaging direction may be specified by "from which position to which position is being observed (for example, observing the body of the stomach from the cardia)" or by the direction of the field of view (for example, in the case of an ultrasound endoscope, the pancreatic long axis direction or a uniaxial direction).

[0075] The imaging information acquisition unit 222 can acquire imaging information by analyzing an endoscopic image, or by using information input by a user or information from an external device (determination device) that determines the state of the endoscope 100 (imaging device). The imaging information acquisition unit 222 can determine which method to use to acquire imaging information based on the conditions set in step S100. When analyzing an endoscopic image, the imaging information acquisition unit 222 may perform the analysis using feature quantities such as the color of the subject, or may use a trained model for analysis (CNN, SVM, etc.). When analyzing by user input, the imaging information acquisition unit 222 can use information input via the operation unit 208. Note that the above-mentioned "external device" can be a device (determination device) that observes the imaging position and / or imaging direction of the endoscope 100 using electromagnetic waves, ultrasound, radiation, etc. (see the example in FIG. 15 , which will be described later).

[0076] <Detection of interest areas> The detector 224 (processor) detects a region of interest from an endoscopic image (medical image) using multiple detectors constituting the detector 224 (step S120: detection process, detection step). The processor 210 can perform multiple detection processes by using multiple detectors constituting the detector 224. In detecting a region of interest, the detector 224 can grasp the position of the region of interest shown in the image at the pixel level using the above-mentioned "feature map" (i.e., detect whether each pixel of the endoscopic image belongs to the region of interest) and output the detection result. Examples of regions of interest (regions of interest) detected by the endoscope system 10 include polyps, cancer, colonic diverticula, inflammation, treatment scars (EMR scars (Endoscopic Mucosal Resection), ESD scars (Endoscopic Submucosal Dissection), clip locations, etc.), bleeding points, perforations, vascular atypia, and various treatment tools. In the case of an ultrasound device such as an ultrasound endoscope, organs and blood vessels may be detected as the region of interest.

[0077] <Selection of detection process (detector) to display detection results> The selection unit 226 (processor) selects which of the multiple detectors (detection processes) constituting the detector 224 should display the detection results of the detector on the monitor 400 (display device) (step S130: selection process, selection step). The selection unit 226 may select a detector based on the imaging information described above, or may select a detector based on the wavelength band of the observation light (e.g., normal light or special light; see the configuration of FIG. 6) or the purpose of the observation. Furthermore, if the processor 210 has multiple types of recognizers (e.g., detector, classifier, measuring instrument, etc.; see the configuration of FIG. 7), it may select a specific type of recognizer. By switching the detectors (recognizers) in this way, it is possible to provide the user with an appropriate diagnostic support function (detection results by the detectors).

[0078] The selection unit 226 selects at least one detector (or multiple detectors) for displaying the detection results. The selection unit 226 may initially select a specific detector (for example, the duodenum detector 224D).

[0079] <Control to allow or disallow switching> As described above, when switching detectors (recognizers) based on imaging information, etc., there is a risk that switching will occur at an inappropriate time. For example, when switching by recognizing an organ from a medical image, an error in organ recognition may result in erroneous switching. Therefore, in the present invention, in order to reduce such risks, switching control (whether to permit or prohibit switching) is performed taking into consideration not only imaging information, etc., but also the detection status of the region of interest.

[0080] Specifically, the switching control unit 228 (processor) controls whether or not to allow switching of the detector (detection process) to be used to display the detection results, depending on the detection state of the area of ​​interest in the multiple detectors (detection processes) that make up the detector 224 (switching control process, switching control step).

[0081] First, the switching control unit 228 determines whether or not a region of interest has been detected by at least one detector (detection process) (step S140: switching control process, switching control step). If the determination is affirmative, i.e., if a region of interest has been detected by at least one detector (detection process), the switching control unit 228 prohibits switching of the detector (detection process) that displays the detection result (step S150: switching control process, switching control step). If the determination is negative, i.e., if a region of interest has not been detected, the switching control unit 228 permits switching (step S160: switching control process, switching control step). If switching is prohibited, the switching control unit 228 can select a detector that was selected immediately before (a detector that was selected up until a predetermined time ago, a detector that was selected up until a predetermined number of frames ago, a detector that was selected by initial setting, etc.).

[0082] In the above-described switching control, the switching control unit 228 does not permit switching of the detector that displays the detection results on the display device while at least one detector (detection process) is detecting a region of interest. In other words, it determines that "if a region of interest is present in an endoscopic image, it is unlikely that a transition in the observation status has occurred (and therefore there is little need to switch the detector)." In fact, when detecting a lesion or the like from an endoscopic image, it is unlikely that a lesion is present in the image while the observation target organ is transitioning. Similarly, when detecting an organ from an ultrasound image, the organ is not properly depicted during a transition in the observation status, so the detector is unlikely to determine that a "region of interest exists." Furthermore, when detecting a lesion, such as cancer, as a region of interest, the organ often appears unnatural in the situation in which the lesion is detected, increasing the risk of errors in organ recognition processing. This is due to the fact that, when a detector (recognizer) is trained using machine learning, it is primarily trained using images of normal organs.

[0083] Under such circumstances, the present invention controls whether or not to allow switching depending on the detection status of the area of ​​interest, as described above, thereby reducing the risk of errors occurring in the recognition process and enabling appropriate switching of diagnostic support functions.

[0084] If the detector to be switched to is in a stopped state, the switching control unit 228 (processor) starts the detection process by that detector. Also, the switching control unit 228 can reduce the load on the memory, etc. by stopping the operation (detection process) of the detector that does not display the detection results on the monitor 400 (display device) (it may be possible to keep the detector operating instead of stopping it, if necessary).

[0085] The switching control unit 228 (processor) may control whether to permit or prohibit switching not only at the time when a region of interest is detected but also around that time. Specifically, the switching control unit 228 can prohibit switching around the time when at least one of the multiple detectors (detection processes) determines that a region of interest exists in the endoscopic image (medical image), and can permit switching around the time when at least one of the multiple detectors determines that a region of interest does not exist in the endoscopic image. FIG. 10 shows an example of such switching control. Part (a) of FIG. 10 shows a mode in which switching is not permitted during a period Δt1 until time t2 if it is determined at time t1 that a region of interest exists in the endoscopic image. Part (b) of FIG. 10 shows a mode in which switching is permitted during a period Δt2 until time t4 if it is determined at time t3 that a region of interest does not exist in the endoscopic image.

[0086] <Display of detection results and notification of switching> The display control unit 230 (processor) displays at least one detection result from the multiple detectors (detection processes) on the monitor 400 (display device) based on the result of the above-mentioned switching control (step S170: display control process, display control step). The notification processing unit 232 (processor) executes notification process to notify the user which detector (detection process) is displaying the detection result (step S175: notification process, notification step). The notification processing unit 232 may notify the user that the detector for displaying the detection result has been switched at the time of switching.

[0087] Processor 210 repeats the above-described process until it determines that "processing is to be ended" upon completion of acquisition of endoscopic images or a user operation (step S180).

[0088] <Examples of display and notification> FIG. 11 is a diagram showing an example of display and notification of the detection result of the region of interest (screen 600 of the monitor 400 (display device)). In part (a) of FIG. 11, the display control unit 230 and the notification processing unit 232 (processor) display an endoscopic image 603 in an image display area 602 on the left side of the screen. Also, a region of interest 610 has been detected from the endoscopic image 603, and the display control unit 230 and the notification processing unit 232 superimpose and display a bounding box 620 on the region of interest 610. Also, in a notification area 604 on the right side of the screen, it is highlighted (displayed in an identifiable manner) to indicate that the bounding box 620 is a detection result by the stomach detector 224C (displaying "STOMACH") (an example of notification processing). In the example shown in part (a) of Figure 11, only the detector displaying the detection results (stomach detector 224C) is operating, and this fact (a list of operating detectors) is notified (displayed) in the notification area 604 (notification process), and the detectors not displaying the detection results (pharynx detector 224A, esophagus detector 224B, duodenum detector 224D) are not displayed.

[0089] 11(b), all detectors are operating and this fact is notified (displayed in a list) in the notification area 604 (notification process). Furthermore, in the notification area 604, the stomach detector 224C, whose detection results are displayed in the image display area 602, is highlighted, and the remaining detectors are grayed out (notification process).

[0090] 11, each detector is identified by text and the line type of the bounding box, but in addition to or instead of the text and line type, the color and brightness of the bounding box may be changed depending on the detector. That is, the display control unit 230 and the notification processing unit 232 (processor 210) can change the display mode of the detection results displayed on the monitor 400 (display device) depending on the detector (the same applies to the modes shown in FIGS. 12 and 13). Such displays in the image display area 602 and the notification area 604 allow the user to easily understand which detector's detection results are being displayed.

[0091] 12 is a diagram showing another example of display and notification of the detection results of the region of interest. In the example shown in the figure, the detector before switching (duodenum detector 224D; displayed as DUODENUM) is displayed grayed out in the notification area 604, and the detector after switching (gastric detector 224C) is highlighted. This display also allows the user to easily understand which detector's detection results are being displayed and the detector switch. Note that the display control unit 230 and the notification processing unit 232 may erase the display of the grayed-out detector after a predetermined time has elapsed since the switch (displaying only at and around the time when the switch occurred).

[0092] 13 is a diagram showing an example of notification when the region of interest has not been detected (all detectors are operating and the detection result of the stomach detector 224C is displayed). In the notification region 604, the stomach detector 224C is highlighted and the remaining detectors are grayed out.

[0093] Fig. 14 is a diagram showing yet another example of display and notification of detection results of a region of interest. In the example of Fig. 14, the detection results of each detector are displayed using the initial letter of the organ. In part (a) of Fig. 14, the display control unit 230 and the notification processing unit 232 (processor 210) notify that the detection result is from the duodenum detector 224D by superimposing a symbol 630 (the initial letter "D" of "DUODENUM") on the region of interest 610 in the image display area 602. Furthermore, the display control unit 230 and the notification processing unit 232 use a right-pointing triangle symbol 640 in the notification area 604 to distinguish and display the duodenum detector 224D, which is the target of the detection results, with the name of the detector (here, "DUODENUM") highlighted and the names of the remaining detectors grayed out. On the other hand, part (b) of FIG. 14 displays the detection results when switched to the stomach detector 224C, and by superimposing a symbol 650 (the initial "S" of "STOMACH") on the attention area 610, it is indicated that the detection results are from the stomach detector 224C. Also, in this part, a right-pointing triangle symbol 640 is used to identify the stomach detector 224C for which the detection results are to be displayed, and the name of the detector is highlighted while the names of the remaining detectors are grayed out. Even with this aspect, the user can easily understand which detector's detection results are being displayed and that the detector has been switched.

[0094] Regarding the notification of "which detector (detection process) will display the detection results," information about the detector that will display the detection results may be directly notified, as in the examples of Figures 11 to 14, or the notification may be indirectly notified by presenting information such as the judgment results of the observation situation.

[0095] As described above, according to the first embodiment, the diagnostic support function can be appropriately switched, and the user can easily understand which detector's detection results are being displayed and how to switch detectors.

[0096] <Acquisition of imaging information using external devices> As described above in the first embodiment, information on the imaging position and / or imaging direction of the imaging device can be obtained by an external device (determination device) that determines the state of the endoscope 100 (imaging device). In the example shown in Fig. 15, multiple magnetic generators 140 (magnets, coils, etc.) that generate magnetism are provided on the endoscope 100, and the endoscope shape measuring device 500 (determination device) detects the magnetism emitted by the magnetic generators 140 with a magnetic antenna 510, and the processor 520 calculates the position and / or shape of the endoscope 100 based on the detection results. The imaging information acquisition unit 222 (processor) of the endoscope system 10 acquires the calculated position and / or shape information and uses it for selection processing.

[0097] <Application to other medical images> In the first embodiment and the modified example described above, the case where recognition is performed using an endoscopic image (optical endoscope), which is one type of medical image (medical image), has been described. However, the medical image processing device, medical image processing method, and medical image processing program according to the present invention can also be used in an ultrasonic endoscope device (ultrasound endoscope system), an ultrasonic image diagnostic device, etc. Images obtained by The present invention can also be applied to cases where medical images other than endoscopic images are used.

[0098] (Addendum) In addition to the above-described embodiments and modifications, the following configurations are also included within the scope of the present invention.

[0099] (Appendix 1) The medical image analysis processor detects an area of ​​interest based on pixel features of the medical image, The medical image analysis result acquisition unit (processor) is a medical image processing device that acquires the analysis results of the medical image analysis processing unit.

[0100] (Appendix 2) The medical image analysis processing unit detects the presence or absence of a noteworthy object based on the feature amount of pixels of the medical image, The medical image analysis result acquisition unit is a medical image processing device that acquires the analysis results of the medical image analysis processing unit.

[0101] (Appendix 3) The medical image analysis result acquisition unit: Obtained from a recording device that records the results of medical image analysis; The analysis results are a region of interest, which is an area of ​​interest contained in a medical image, and / or the presence or absence of an object of interest.

[0102] (Appendix 4) A medical image processing apparatus in which the medical image is a normal light image obtained by irradiating white light or light of multiple wavelength bands as white light.

[0103] (Appendix 5) A medical image is an image obtained by irradiating light in a specific wavelength range. A medical imaging device, wherein the specific wavelength band is narrower than the white wavelength band.

[0104] (Appendix 6) A medical image processing device in which the specific wavelength band is the blue or green band of the visible range.

[0105] (Appendix 7) A medical image processing device, wherein the specific wavelength band includes a wavelength band of 390 nm to 450 nm or 530 nm to 550 nm, and the light of the specific wavelength band has a peak wavelength within the wavelength band of 390 nm to 450 nm or 530 nm to 550 nm.

[0106] (Appendix 8) The specific wavelength band is the red band of the visible range for medical imaging processing equipment.

[0107] (Appendix 9) A medical image processing device, wherein the specific wavelength band includes a wavelength band of 585 nm or more and 615 nm or less, or a wavelength band of 610 nm or more and 730 nm or less, and the light of the specific wavelength band has a peak wavelength within the wavelength band of 585 nm or more and 615 nm or less, or a wavelength band of 610 nm or more and 730 nm or less.

[0108] (Appendix 10) A medical image processing device, wherein the specific wavelength band includes a wavelength band in which oxygenated hemoglobin and reduced hemoglobin have different absorption coefficients, and the light in the specific wavelength band has a peak wavelength in the wavelength band in which oxygenated hemoglobin and reduced hemoglobin have different absorption coefficients.

[0109] (Appendix 11) A medical image processing device in which the specific wavelength band includes a wavelength band of 400±10 nm, 440±10 nm, 470±10 nm, or a wavelength band of 600 nm or more and 750 nm or less, and the light in the specific wavelength band has a peak wavelength in a wavelength band of 400±10 nm, 440±10 nm, 470±10 nm, or a wavelength band of 600 nm or more and 750 nm or less.

[0110] (Appendix 12) Medical images are in vivo images that show the inside of a living body. In vivo images are medical image processing devices that have information on the fluorescence emitted by fluorescent substances in the living body.

[0111] (Appendix 13) The fluorescence peaks at 390 nm A medical image processing device that obtains images by irradiating the inside of a living body with excitation light of 470 nm or more.

[0112] (Appendix 14) Medical images are in vivo images that show the inside of a living body. The specific wavelength band is a wavelength band of infrared light in a medical imaging device.

[0113] (Appendix 15) A medical image processing device in which the specific wavelength band includes a wavelength band of 790 nm or more and 820 nm or less or a wavelength band of 905 nm or more and 970 nm or less, and the light in the specific wavelength band has a peak wavelength in the wavelength band of 790 nm or more and 820 nm or less or a wavelength band of 905 nm or more and 970 nm or less.

[0114] (Appendix 16) The medical image acquisition unit (image acquisition unit, processor) includes a special light image acquisition unit (processor) that acquires a special light image having information of a specific wavelength band based on a normal light image obtained by irradiating white light or light of a plurality of wavelength bands as white light, Medical image processing equipment uses special light for medical images.

[0115] (Appendix 17) A medical image processing device that obtains signals in specific wavelength bands through calculations based on the RGB or CMY color information contained in normal light images.

[0116] (Appendix 18) a feature image generating unit (processor) that generates a feature image by calculation based on at least one of a normal light image obtained by irradiating light in a white band or light in a plurality of wavelength bands as white band light, and a special light image obtained by irradiating light in a specific wavelength band; Medical image processing device that processes medical images as feature images.

[0117] (Appendix 19) A medical image processing device according to any one of appendices 1 to 18; an endoscope (endoscopy scope) that acquires an image by irradiating light in a white wavelength band or at least one of light in a specific wavelength band; An endoscope device (endoscope system) comprising:

[0118] (Appendix 20) A diagnostic support device comprising the medical image processing device according to any one of appendices 1 to 18.

[0119] (Appendix 21) A medical service support device comprising the medical image processing device according to any one of appendices 1 to 18.

[0120] Although the embodiment and other examples of the present invention have been described above, the present invention is not limited to the above-described aspects, and various modifications are possible within the scope of the spirit of the present invention. [Explanation of symbols]

[0121] 10 Endoscopy System 100 Endoscope 102 Handheld operation unit 104 Insertion section 106 Universal Cable 108 Light guide connector 112 Soft part 114 Curved section 116 Hard tip 116A Tip side end surface 123 Lighting Department 123A Lighting Lens 123B Lighting Lens 126 Forceps port 130 Photographing optical system 132 Camera Lens 134 Image sensor 136 Drive Circuit 140 Magnetic Generator 141 Air and water supply button 142 Suction button 143 Function Button 144 Shooting button 170 Light Guide 200 Medical Image Processing Department 202 Image Input Controller 205 Communication control section 206 Video output section 207 Recording Department 208 Operation section 209 Audio Processing Unit 209A Speaker 210 processors 211 ROM 212 RAM 220 Image acquisition unit 222 Imaging information acquisition unit 224 detector 224A Pharyngeal Detector 224B Esophageal detector 224C Gastric detector 224C1 Normal light detector 224C2 Special light detector 224D Duodenal Detector 224E Pharyngeal Recognition Device 224E1 Pharyngeal detector 224E2 Pharyngeal Classifier 224F Esophageal recognition device 224F1 Esophageal detector 224F2 Esophageal Classifier 224G Stomach recognition device 224G1 Gastric Detector 224G2 Gastric Classifier 224H Duodenum recognition device 224H1 Duodenal detector 224H2 Duodenal Classifier 226 Selection Section 228 Switching control section 230 Display control unit 232 Notification processing section 234 Recording control section 250 input layers 252 Middle Class 254 output layer 256 convolutional layers 258 pooling layers 260 fully connected layer 300 Light source device 310 light source 310B Blue light source 310G green light source 310R red light source 310V purple light source 330 aperture 340 Condenser Lens 350 Light source control unit 400 monitors 500 Endoscope shape measuring device 510 Magnetic Antenna 520 processor sa 6 00 screen 602 Image display area 603 Endoscopic Images 604 Notification Area 610 Areas of Interest 620 Bounding Box 630 symbol 640 symbols 650 symbol F 1 filter F 2. Filter S100~S180 Each step of the medical image processing method Δt1 period Δt2 period

Claims

1. 1. A medical imaging device comprising a processor, The processor: an image acquisition process for acquiring time-series medical images; a plurality of detection processes for detecting regions of interest from the acquired medical images; a display control process for displaying at least one of the detection results obtained by the plurality of detection processes on a display device; a selection process for selecting a detection result from one of the plurality of detection processes to be displayed on the display device; a switching control process that determines whether an attention area has been detected in at least one of the plurality of detection processes, and if the determination is affirmative, prohibits switching of the detection process whose detection results are to be displayed to the selected detection process, and permits the switching if the determination is negative, thereby controlling whether the switching is permitted or not; A medical imaging device that performs

2. The processor: The medical image processing device according to claim 1 , wherein the switching control process does not permit the switching near a time when it is determined that a region of interest exists in the medical image in at least one of the plurality of detection processes.

3. The processor: The medical image processing apparatus according to claim 1 , wherein the switching control process permits the switching near a time when it is determined that no region of interest exists in the medical image in at least one of the plurality of detection processes.

4. The processor: The medical image processing apparatus according to claim 1 , wherein the switching control process stops a detection process that does not cause the display device to display the detection result.

5. The processor: The medical image processing apparatus according to claim 4 , wherein the switching control process starts the detection process switched to a state in which the detection result is displayed.

6. The processor: The medical image processing apparatus according to claim 1 , further comprising a notification process for notifying a user of which detection process the detection results of which are displayed on the display device are from.

7. The processor: The medical image processing apparatus according to claim 1 , wherein information indicating an imaging position and / or an imaging direction of the medical image is acquired from the medical image, and the selection is made based on the acquired information.

8. The processor:

8. A medical image processing device according to claim 1, wherein information indicating the imaging position and / or imaging direction of the medical image is obtained from a determination device that determines the state of the imaging device that captures the medical image, and the selection is made based on the obtained information.

9. The processor: The medical image processing apparatus according to claim 1 , wherein the display control process changes a display mode of the detection result displayed on the display device in accordance with the detection process.

10. The processor: The medical image processing apparatus according to claim 1 , wherein, when the detection process for displaying the detection result is switched in the switching control process, a notification is given that the switching has been performed.

11. A medical image processing device according to any one of claims 1 to 10; an endoscope to be inserted into a subject, the endoscope having an imaging unit that sequentially captures the medical images; the display device; An endoscope system comprising:

12. 1. A medical image processing method performed by a medical image processing device having a processor, comprising: an image acquisition step of acquiring time-series medical images; a detecting step of detecting a region of interest from the acquired medical image using a plurality of detectors; a display control step of displaying, on a display device, a detection result of at least one of the detection results of the plurality of detectors; a selection step of selecting which of the plurality of detectors the detection result of which is to be displayed on the display device; a switching control step of controlling whether or not to switch the detector to be used to display the detection results to the selected detector, depending on the detection states of the region of interest in the plurality of detectors; a switching control step of determining whether a region of interest has been detected by at least one of the plurality of detectors, and disallowing switching of the detector to be used to display the detection result to the selected detector if the determination is affirmative, and allowing the switching if the determination is negative, thereby controlling whether or not to allow the switching; A medical image processing method comprising:

13. A medical image processing program that causes a medical image processing device having a processor to execute a medical image processing method, The medical image processing method includes: an image acquisition step of acquiring time-series medical images; a detecting step of detecting a region of interest from the acquired medical image using a plurality of detectors; a display control step of displaying, on a display device, a detection result of at least one of the detection results of the plurality of detectors; a selection step of selecting which of the plurality of detectors the detection result of which is to be displayed on the display device; a switching control step of determining whether or not a region of interest has been detected by at least one of the plurality of detectors, and disallowing switching of the detector to be used to display the detection result to the selected detector if the determination is affirmative, and allowing the switching if the determination is negative, thereby controlling whether or not to allow the switching; Medical imaging programs, including:

14. A non-transitory computer-readable recording medium having the program according to claim 13 recorded thereon.

Images