Image processing device and medical task support device

Abstract

To provide an image processing device that prevents misunderstanding between a result of performing recognition processing and a result of not performing the recognition processing, and to provide a medical task support device.SOLUTION: At least one processor inputs an endoscopic image into a learned model that executes recognition processing on a portion having specific features in the endoscopic image, acquires a result of the recognition processing from the learned model, sets the recognition processing to be valid or invalid on the basis of an operation of the subject, outputs, to a monitor, information indicating that the recognition processing is valid and the result of the recognition processing when the recognition processing is set to be valid, and outputs, to the monitor, the information indicating that the recognition processing is invalid when the recognition processing is set to be invalid.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an image processing apparatus and a medical operation support apparatus that perform recognition processing of a subject using an image captured by an endoscope.

Background Art

[0002] In the medical field, an endoscope system including a light source device, an endoscope, and a processor device has become widespread. The light source device generates illumination light. The endoscope captures an image of a subject using an image sensor. The processor device performs generation of an image and other image processing.

[0003] The endoscope system may have additional functions in addition to simply capturing an image of a subject for observation. For example, an endoscope system having an "input mode" for operating the position and orientation of a treatment instrument with a touch panel is known (Patent Document 1). In the endoscope system of this Patent Document 1, the state of the input mode is displayed on the screen. Also, an endoscope system having a function of displaying an image indicating the curved state of the distal end of the endoscope and a character string notifying that calibration has been completed, etc. on the screen to notify system abnormalities, etc. is known (Patent Document 2). In addition, an endoscope system that displays the on or off state of a foot switch on the screen (Patent Document 3), an endoscope system that displays a mark indicating that air is being supplied on the screen (Patent Document 4), and an endoscope system that displays the on or off state of recording on the screen (Patent Document 5) are known.

[0004] Also, an endoscope system that detects a treatment instrument and uses the result is known. For example, an endoscope system that determines the correction intensity of an image using the detection result of a treatment instrument is known (Patent Document 6). An endoscope system that stops the autofocus function when a treatment instrument is detected is known (Patent Document 7).

[0005] In addition, in recent years, an endoscope system that supports diagnosis by calculating biological function information using an image of a subject has also become known (Patent Document 8).

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Patent Document 5

Patent Document 6

Patent Document 7

Patent Document 8

Summary of the Invention

Problems to be Solved by the Invention

[0007] When obtaining information for assisting diagnosis (hereinafter referred to as diagnostic support information) by recognizing a subject or a part of a subject having specific features using medical images such as images taken with an endoscope (hereinafter referred to as endoscope images), it is necessary to use medical images for which recognition processing can function by taking pictures under specific conditions. This is because if a medical image for which recognition processing cannot function is used, even if the result of the recognition processing is obtained, the result of the recognition processing may be inaccurate.

[0008] However, usually, it is not always possible to always obtain a medical image on which the recognition process can function. For this reason, as a result of inaccurate recognition processing, or diagnostic support information calculated or the like using the result of inaccurate recognition processing may instead hinder the observation or diagnosis of the subject. For example, in an endoscope system, when a treatment tool is captured in an endoscope image, the accuracy of the recognition process may decrease.

[0009] In order not to provide the results of inaccurate recognition processing as described above, etc., it is conceivable not to perform the recognition process when a medical image on which the recognition process can function cannot be obtained. However, if simply not performing the recognition process, a user such as a doctor may misrecognize the result of the recognition process or the result of not performing the recognition process. For example, when using a device that performs a recognition process for detecting candidates for lesions, if simply not performing the recognition process, a doctor or the like using this may misrecognize that no candidates for lesions were detected as a result of performing the recognition process. That is, a doctor or the like may not be able to determine whether the result of the recognition process is not displayed because the recognition process was not performed, or because there was no object to be recognized as a result of performing the recognition process.

[0010] Also, when starting a surgery or the like on a subject, or when starting a detailed observation by spraying a coloring agent, etc., in order not to interfere with the operation of the operator, the automatic recognition process or the display of the result of the recognition process may be disabled. However, also in this case, as in the above, a doctor or the like may misrecognize the result of performing the recognition process or the result of not performing the recognition process. In addition, there is also a problem that the concentration of a doctor or the like may decrease when the recognition function is automatically disabled even though the doctor or the like has not explicitly disabled the recognition function.

[0011] An object of the present invention is to provide an image processing apparatus and a medical service support apparatus that at least prevent misrecognition of the result of performing a recognition process and the result of not performing the recognition process.

Means for Solving the Problem

[0012] The image processing apparatus of the present invention includes at least one processor. The at least one processor acquires an endoscopic image captured by an endoscope, inputs the endoscopic image into a learned model that executes recognition processing on a part having specific features in the endoscopic image, obtains the result of the recognition processing from the learned model, and sets the recognition processing to be effective or ineffective based on an operation on the subject. The result of the recognition processing is information indicating a lesion included in the endoscopic image, or information indicating the type or degree of progression of the lesion. When the recognition processing is set to be effective, information indicating that the recognition processing is effective and the result of the recognition processing are output to the monitor. When the recognition processing is set to be ineffective, information indicating that the recognition processing is ineffective is output to the monitor.

[0013] It is preferable that the invalidation of the recognition processing means not executing the recognition processing or not notifying the result of the recognition processing. The notification preferably indicates that the recognition processing using artificial intelligence is invalid.

[0014] As a result of the recognition processing, it is preferable to output information indicating a lesion to the monitor. The operation on the subject is preferably the use of a treatment instrument, the spraying of a liquid, or the cleaning of the subject. The operation on the subject is preferably detected from the endoscopic image.

[0015] As a result of the recognition processing, it is preferable to output information indicating the type or degree of progression of the lesion to the monitor. The operation on the subject is preferably the use of a treatment instrument, the spraying of a liquid, or the cleaning of the subject. The operation on the subject is preferably detected from the endoscopic image.

[0016] The operation on the subject is preferably the use of a treatment instrument, the spraying of a liquid, or the cleaning of the subject. The recognition process is preferably set to be invalid when a treatment instrument is detected. The treatment instrument is preferably detected from the endoscopic image. It is preferable to detect a plurality of types of treatment instruments from the endoscopic image and set the recognition process to be invalid when a specific type of treatment instrument is detected. The recognition process is preferably set to be valid when no treatment instrument is detected.

[0017] The medical operation support device of the present invention includes the image processing device of the present invention described above, an endoscope that acquires an image of a subject, and a processor device that controls the endoscope to generate an endoscopic image, and the image processing device and the processor device are connected via a network.

Advantages of the Invention

[0018] According to the present invention, it is possible to provide an endoscope system that prevents misrecognition of the result of performing the recognition process and the result of not performing the recognition process.

Brief Description of the Drawings

[0019]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Mode for Carrying Out the Invention

[0020] [First Embodiment] As shown in FIG. 1, an endoscope system 10 (endoscope device) includes an endoscope 12, a light source device 14, a processor device 16, a monitor 18, and a console 19. The endoscope 12 captures an image of a subject. The light source device 14 generates illumination light. The processor device 16 performs system control and image processing of the endoscope system 10 and the like. The monitor 18 is a display unit that displays images and the like from the endoscope 12. The console 19 is an input device that performs setting input and the like to the processor device 16 and the like.

[0021] The endoscope 12 has an insertion portion 12a to be inserted into a subject, an operation portion 12b provided at the proximal end portion of the insertion portion 12a, a bending portion 12c provided at the distal end side of the insertion portion 12a, and a distal end portion 12d. By operating the angle knob 12e of the operation portion 12b, the bending portion 12c bends. As a result, the distal end portion 12d faces in a desired direction. In addition to the angle knob 12e, the operation portion 12b is provided with a treatment tool insertion port 12f, a zoom operation portion 13a, and a water supply button 13b. The treatment tool insertion port 12f is an entrance for inserting treatment tools such as biopsy forceps, a snare, or an electrosurgical knife. The treatment tool inserted into the treatment tool insertion port 12f protrudes from the distal end portion 12d. By operating the zoom operation portion 13a, the subject can be enlarged or reduced for imaging. Also, by operating the water supply button 13b, a liquid such as water stored in the water supply tank 17 is ejected from the distal end portion 12d.

[0022] As shown in FIG. 2, the light source device 14 includes a light source unit 20 that emits illumination light and a light source control unit 22 that controls the operation of the light source unit 20.

[0023] The light source unit 20 emits illumination light for illuminating the subject. The emission of the illumination light includes the emission of light such as excitation light used for emitting the illumination light. The light source unit 20 includes, for example, a light source such as a laser diode (hereinafter referred to as LD), an LED (Light Emitting Diode), a xenon lamp, or a halogen lamp, and emits at least white illumination light or excitation light used for emitting white illumination light. White includes a so-called pseudo-white that is substantially equivalent to white in imaging a subject using the endoscope 12. The light source unit 20 includes, as necessary, a phosphor that emits light upon receiving irradiation of excitation light, or an optical filter that adjusts the wavelength band, spectral spectrum, or light amount of the illumination light or excitation light. In addition, the light source unit 20 can emit light having a specific wavelength band necessary for imaging an image used for calculating biological information such as the oxygen saturation of hemoglobin contained in the subject.

[0024] In this embodiment, the light source unit 20 includes four-color LEDs: a V-LED 20a, a B-LED 20b, a G-LED 20c, and an R-LED 20d. The V-LED 20a emits purple light VL with a central wavelength of 405 nm and a wavelength band of 380 to 420 nm. The B-LED 20b emits blue light BL with a central wavelength of 460 nm and a wavelength band of 420 to 500 nm. The G-LED 20c emits green light GL with a wavelength band ranging from 480 to 600 nm. The R-LED 20d emits red light RL with a central wavelength of 620 to 630 nm and a wavelength band ranging from 600 to 650 nm. Note that the central wavelengths of the V-LED 20a and the B-LED 20b have a width of approximately ±20 nm, preferably approximately ±5 nm to approximately ±10 nm.

[0025] The light source control unit 22 controls the timing of turning on, turning off, or shielding each light source constituting the light source unit 20, as well as the light emission amount and the like. As a result, the light source unit 20 can emit a plurality of types of illumination light having different spectral spectra. In this embodiment, the light source control unit 22 adjusts the spectral spectrum of the illumination light by inputting independent control signals to the lighting, extinguishing, light emission amount during lighting, insertion and extraction of the optical filter, etc. of each of the LEDs 20a to 20d. Thereby, the light source unit 20 emits white light. Further, the light source unit 20 can emit illumination light composed of at least narrow-band purple light. "Narrow band" means that it is substantially a single wavelength band in relation to the characteristics of the subject and / or the spectral characteristics of the color filter included in the image sensor 48. For example, based on the central wavelength, when the wavelength band is, for example, about ±20 nm or less (preferably about ±10 nm or less), this light is narrow band.

[0026] An illumination optical system 30a and a photographing optical system 30b are provided at the distal end portion 12d of the endoscope 12. The illumination optical system 30a includes an illumination lens 45, and illumination light is emitted toward the subject through this illumination lens 45.

[0027] The imaging optical system 30b includes an objective lens 46, a zoom lens 47, and an image sensor 48. The image sensor 48 captures an object using reflected light of illumination light returning from the object (including, in addition to the reflected light, scattered light, fluorescence emitted by the object, or fluorescence caused by a drug administered to the object, etc.) via the objective lens 46 and the zoom lens 47. The zoom lens 47 moves by operating the zoom operation unit 13a to magnify or reduce the object image.

[0028] The image sensor 48 has, for each pixel, one color filter out of a plurality of color filters of different colors. In the present embodiment, the image sensor 48 is a color sensor having a color filter of the primary color system. Specifically, the image sensor 48 includes an R pixel having a red color filter (R filter), a G pixel having a green color filter (G filter), and a B pixel having a blue color filter (B filter).

[0029] Note that as the image sensor 48, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor can be used. Also, although the image sensor 48 of the present embodiment is a color sensor of the primary color system, a color sensor of the complementary color system can also be used. The complementary color sensor has, for example, a cyan pixel provided with a cyan color filter, a magenta pixel provided with a magenta color filter, a yellow pixel provided with a yellow color filter, and a green pixel provided with a green color filter. When a complementary color sensor is used, the image obtained from the pixels of each color can be converted into an image similar to the image obtained by a color sensor of the primary color system by performing complementary color-primary color conversion. The same applies when the sensor of the primary color system or the complementary color system has one or more types of pixels having characteristics other than those described above, such as a W pixel (a white pixel that receives light in almost the entire wavelength band). Also, although the image sensor 48 of the present embodiment is a color sensor, a monochrome sensor without a color filter may be used.

[0030] The processor device 16 includes a control unit 52, an image acquisition unit 54, an image processing unit 61, a notification unit 65, and a display control unit 66 (see FIG. 2).

[0031] The control unit 52 performs overall control of the endoscope system 10 such as synchronous control of the irradiation timing of illumination light and the imaging timing. When various settings are input using the console 19 or the like, the control unit 52 inputs the settings to each part of the endoscope system 10 such as the light source control unit 22, the image sensor 48, or the image processing unit 61.

[0032] The image acquisition unit 54 acquires an image of the subject. More specifically, the image acquisition unit 54 acquires an image of the subject using pixels of each color from the image sensor 48, that is, a RAW image. Also, the RAW image is an image before performing demosaicing processing. As long as it is an image before performing demosaicing processing, an image obtained by performing any processing such as noise reduction processing on the image acquired from the image sensor 48 is also included in the RAW image.

[0033] The image acquisition unit 54 includes a DSP (Digital Signal Processor) 56, a noise reduction unit 58, and a conversion unit 59 in order to perform various processes on the acquired RAW image as necessary to generate an endoscope image.

[0034] The DSP 56 includes, for example, an offset processing unit, a defect correction processing unit, a demosaicing processing unit, a complement processing unit, a linear matrix processing unit, and a YC conversion processing unit, etc. (none of which are shown). The DSP 56 performs various processes on the RAW image or an image generated using the RAW image.

[0035] The offset processing unit performs offset processing on the RAW image. The offset processing is a process of reducing the dark current component from the RAW image and setting an accurate zero level. The offset processing may be referred to as clamp processing. The defect correction processing unit performs defect correction processing on the RAW image. The defect correction processing is a process of correcting or generating the pixel value of the RAW pixel corresponding to the defective pixel of the image sensor 48 when the image sensor 48 has defective pixels (defective pixels) due to the manufacturing process or aging. The demosaicing processing unit performs demosaicing processing on the RAW images of each color corresponding to each color filter. The demosaicing processing is a process of generating the missing pixel values by interpolation in the RAW image due to the arrangement of the color filters. The linear matrix processing unit performs linear matrix processing on the endoscopic image generated by allocating one or more RAW images to the RGB color channels. The linear matrix processing is a process of enhancing the color reproducibility of the endoscopic image. The YC conversion processing unit converts the endoscopic image generated by allocating one or more RAW images to the RGB color channels into an endoscopic image having a luminance channel Y, a color difference channel Cb, and a color difference channel Cr.

[0036] The noise reduction unit 58 performs noise reduction processing on the endoscopic image having the luminance channel Y, the color difference channel Cb, and the color difference channel Cr, for example, using the moving average method or the median filter method. The conversion unit 59 reconverts the luminance channel Y, the color difference channel Cb, and the color difference channel Cr after the noise reduction processing into an endoscopic image having the BGR color channels again.

[0037] The image processing unit 61 performs necessary image processing on the endoscopic image output by the image acquisition unit 54. Also, the image processing unit 61 performs recognition processing for recognizing a subject or a part of the subject having specific features using the endoscopic image. Specifically, as shown in FIG. 3, the image processing unit 61 includes an image generation unit 71, a recognition unit 72, a discrimination unit 73, a setting unit 74, and the like.

[0038] The image generation unit 71 acquires an endoscopic image from the image acquisition unit 54 and generates an endoscopic image for display on the monitor 18 and the like. For example, the image generation unit 71 acquires from the image acquisition unit 54 a B image obtained by photographing a subject using B pixels, a G image obtained by photographing the subject using G pixels, and an R image obtained by photographing the subject using R pixels, and generates an endoscopic image for display using all or part of these images.

[0039] Also, when generating an endoscopic image for display, the image generation unit 71 performs necessary image processing on the endoscopic image acquired from the image acquisition unit 54 or an image generated using the endoscopic image acquired from the image acquisition unit 54. The image processing performed by the image generation unit 71 is, for example, an enhancement process for enhancing a subject or a part of the subject. Enhancement means making it possible to obtain information on a specific part by distinguishing it from other tissues or structures, etc. For example, processes such as surrounding a part having a specific feature with a frame, showing the contour, or relatively changing the color or brightness with respect to other parts (e.g., normal mucous membranes, etc.) are enhancement processes. "Making it possible to obtain information" includes making it possible to recognize the position, shape, color or brightness, and / or size (range), etc. of a specific part, as well as making it possible to obtain biological function information (e.g., oxygen saturation or blood vessel density, etc.) about a specific part.

[0040] The recognition unit 72 performs a recognition process of recognizing a subject using an image obtained by photographing the subject. "Recognizing a subject" means detecting the presence or absence of a part of a subject having specific features (including the case of detecting the entire subject), discriminating the type or degree of progress of a part of a subject having specific features (including discriminating about the entire subject), and / or obtaining (calculating, etc.) biological function information about part or all of the subject. A part of a subject having specific features is, for example, a lesion, a candidate for a lesion, or a treatment mark such as a treatment scar (hereinafter referred to as a lesion, etc.). That is, the recognition unit 72 can recognize the presence or absence of a lesion, etc. by the recognition process. Further, the recognition unit 72 can recognize the type or degree of progress of a lesion, etc. by the recognition process. The recognition of the type of a lesion, etc. means, for example, when the lesion, etc. is a polyp, discriminating the type such as adenoma, hyperplastic polyp, or cancer. The recognition of the degree of progress of a lesion, etc. means, for example, if the lesion, etc. is cancer, discriminating the cancer stage, or otherwise, discriminating the NICE (NBI (Narrow Band Imaging) International Colorectal Endoscopic Classification) classification or the JNET (The Japan NBI Expert Team) classification, etc. In the present embodiment, the image obtained by photographing the subject is an endoscopic image obtained from the image acquisition unit 54 or an endoscopic image generated by the image generation unit 71. Further, in the present embodiment, the recognition unit 72 detects a lesion, etc. by the recognition process.

[0041] The recognition unit 72 is, for example, an artificial intelligence (AI) with a learning function. Specifically, the recognition unit 72 is an AI that has learned using machine learning algorithms such as a neural network (NN), a convolutional neural network (CNN), adaboost, or a random forest. Also, since the recognition unit 72 has learned to perform recognition processing using specific images, even if it can perform recognition processing using other images, the accuracy of the recognition processing results may be low. In the present embodiment, the recognition unit 72 is an AI that has learned to detect lesions and the like using endoscopic images. "Having a learning function" means being able to learn and includes having already learned. Note that instead of being configured with an AI, the recognition unit 72 can be configured to calculate feature amounts from images and perform detection and the like using the calculated feature amounts.

[0042] In the present embodiment, the recognition unit 72 includes a detection unit 76 and a discrimination unit 77. The detection unit 76 is an AI that has learned to detect lesions and the like using endoscopic images. The discrimination unit 77 is an AI that has learned to discriminate the degree of progression of lesions and the like using endoscopic images.

[0043] The recognition unit 72 inputs the result of the recognition process to the image generation unit 71 or the display control unit 66 according to the content of the recognition process. When the recognition unit 72 inputs the result of the recognition process to the image generation unit 71, the image generation unit 71 generates an endoscopic image for display reflecting the result of the recognition process. When the recognition unit 72 inputs the result of the recognition process to the display control unit 66, the display control unit 66 displays the result of the recognition process on the screen of the monitor 18 together with the endoscopic image obtained from the image generation unit 71. The recognition unit 72 inputs the result of the recognition process to the image generation unit 71, for example, when changing the color or the like of the endoscopic image for display using the value of the biological function information which is the result of the recognition process. The recognition unit 72 inputs the result of the recognition process to the display control unit 66, for example, when indicating the position or the like of a lesion or the like which is the result of the recognition process by means of the display of a frame superimposed on the endoscopic image. In the present embodiment, the recognition unit 72 inputs information such as the position of the detected lesion or the like which is the result of the recognition process to the image generation unit 71. Then, the image generation unit 71 generates an endoscopic image for display in which the part with a lesion or the like is subjected to an emphasis process.

[0044] The discrimination unit 73 discriminates an operation on the subject. An operation on the subject refers to an act of exerting some (any) action on the subject. For example, taking a biological tissue used for a biopsy or a treatment-related procedure such as an endoscopic mucosal resection (EMR), spraying a liquid on the subject, or cleaning the subject are operations on the subject. Spraying a liquid on the subject means administering a liquid medicine to the subject or spraying water. Spraying water means pouring water on part or all of the subject. In addition, administering a medicine includes spraying a coloring agent such as indigo carmine on the subject, as well as intravenous injection of a fluorescent medicine such as indocyanine green (ICG) into the subject. Cleaning the subject means washing away residues, residual liquids, or substances that interfere with observation such as blood adhering to the subject with water or air, etc., and exposing the subject to be observed to the endoscope 12. Note that even if no action actually affects the subject as a result, an act that can exert an action on the subject is included in the "operation on the subject". For example, the act of ejecting air (air) to clean the subject corresponds to an operation on the subject even if the subject does not become clean as a result. Also, the "operation on the subject" includes a series of related acts before and after exerting an action on the subject. For example, when performing a biopsy, the act of protruding forceps from the endoscope 12 is a series of acts related to the biopsy even before tissue collection and is included in the operation on the subject.

[0045] The determination of an operation on a subject means detecting that an operation is performed on the subject or detecting that an operation has been performed on the subject. The determination of an operation is performed, for example, by detecting the use of a treatment instrument, detecting a treatment mark (including detection of a surgical mark, detection of bleeding caused by a treatment, detection of a tissue or structure whose color or the like has changed due to spraying of a coloring agent, etc.), or detecting an operation of a water supply button 13b or the like. The discrimination unit 73 can detect the presence or absence of a treatment instrument or the type (shape, etc.) of the treatment instrument using an endoscopic image, setting information, or the like. The discrimination unit 73 can perform the detection of the above treatment instrument or the like (including spraying of water) by analyzing the endoscopic image. Further, the discrimination unit 73 can detect an operation of a water supply button 13b or the like by acquiring a signal or setting information from the control unit 52.

[0046] The determination of an operation on a subject includes determining the presence or absence of an operation on the subject and determining the type of the operation on the subject. The determination of the presence or absence of an operation includes detecting the "operation on the subject" to be detected regardless of the type of the operation. Further, the determination of the type of the operation on the subject means distinguishing the content of the operation (for example, whether it is a biopsy, drug administration, etc.) and determining the operation on the subject. In the present embodiment, the discrimination unit 73 determines the presence or absence of an operation on the subject.

[0047] The setting unit 74 sets the recognition unit 72 to be valid or invalid using the discrimination result of the discrimination unit 73. The discrimination result of the discrimination unit 73 is the presence or absence of an operation on the subject and / or the type of operation on the subject. "Setting the recognition unit 72 to be valid" means making the recognition process performed by the recognition unit 72 valid, setting the recognition unit 72 to execute the recognition process, setting to notify the result of the recognition process, or setting the recognition unit 72 to output the result of the recognition process. "Setting the recognition unit 72 to be invalid" means making the recognition process performed by the recognition unit 72 invalid, setting the recognition unit 72 not to execute the recognition process, or setting not to notify the result of the recognition process, or setting not to output the result of the recognition process. That is, "setting the recognition unit 72 to be valid or invalid" means making the recognition process performed by the recognition unit 72 valid or invalid, or making the notification (or output of the recognition process result) of the recognition process result valid or invalid. "Notification of the result of the recognition process" means making the result of the recognition process in a state that can be recognized by a doctor or the like. "Output of the result of the recognition process" means that the recognition unit 72 delivers the result of the recognition process to another processing unit such as the image generation unit 71. In the present embodiment, the setting unit 74 sets the recognition process itself performed by the recognition unit 72 to be valid or invalid.

[0048] When the discrimination unit 73 discriminates the presence or absence of an operation on the subject, the setting unit 74 sets the recognition unit 72 to be invalid when there is an operation on the subject, and sets the recognition unit 72 to be valid when there is no operation on the subject. This is because when there is an operation on the subject, the accuracy of the recognition process may decrease, or it is a situation where a doctor or the like needs to concentrate on the operation.

[0049] When the discrimination unit 73 discriminates the type of operation on the subject, the setting unit 74 invalidates the recognition unit 72 when there is a specific operation on the subject. The specific operation is an operation that reduces the accuracy of the recognition process performed by the recognition unit 72 or an operation that requires a doctor or the like to concentrate on the operation. Therefore, when the discrimination unit 73 discriminates the type of operation on the subject, depending on the type of operation, the setting unit 74 can set the recognition unit 72 to be valid. For this reason, appropriate diagnostic support can be continued according to the situation.

[0050] In addition, when the recognition unit 72 performs a plurality of types of recognition processes, the setting unit 74 can set each recognition process to be valid or invalid. For example, when the setting unit 74 sets the recognition unit 72 to be valid or invalid, it can set the detection unit 76 to be valid or invalid and can set the discrimination unit 77 to be valid or invalid regardless of the valid or invalid setting of the detection unit 76. Of course, the setting unit 74 can set both the detection unit 76 and the discrimination unit 77 together to make the entire recognition unit 72 valid or invalid. In the present embodiment, the setting unit 74 sets the entire recognition unit 72, that is, the detection unit 76 and the discrimination unit 77, to be valid or invalid together.

[0051] The notification unit 65 obtains the discrimination result from the discrimination unit 73 (or obtains the information related to the setting of the recognition unit 72 from the setting unit 74), and notifies the valid or invalid state of the recognition unit 72. "Notify" regarding the valid or invalid state of the recognition unit 72 means to make the state such that a user such as a doctor can know the valid or invalid state of the recognition unit 72. For example, the notification unit 65 can notify the valid or invalid state of the recognition unit 72 by displaying a message (character string), characters, graphics, and / or symbols (including the display of marks, icons, or indicators, etc.) on the screen of the monitor 18, not displaying these, or changing the display of these. In addition, the notification unit 65 can notify the valid or invalid state of the recognition unit 72 by turning on, off, or flashing a lamp, sound (including voice), vibration of a member having a vibration function, or changing these. Of course, the notification unit 65 can notify the valid or invalid state of the recognition unit 72 by combining a character string or the like with the lighting of a lamp or the like. In the present embodiment, the notification unit 65 displays a display indicating the valid or invalid state of the recognition unit 72 on the screen of the monitor 18.

[0052] The display control unit 66 converts the endoscopic image output by the image processing unit 61 into a format suitable for display and outputs it to the monitor 18. Thereby, the monitor 18 displays the endoscopic image. Also, in the present embodiment, the notification unit 65 inputs information indicating the valid or invalid state of the recognition unit 72. Therefore, the display control unit 66 displays that information on the screen of the monitor 18.

[0053] The endoscopic system 10 configured as described above operates as follows regarding the notification of the valid or invalid state of the recognition unit 72. As shown in FIG. 4, when photographing a subject using the endoscope 12 (step S111), the image acquisition unit 54 acquires an endoscopic image. Then, the image generation unit 71 performs necessary image processing to generate an endoscopic image for display (step S112).

[0054] Thereafter, or in parallel with the endoscopic image for display, the discrimination unit 73 discriminates an operation on the subject, for example, using the endoscopic image for display (step S113). When there is no operation on the subject (step S113: YES), the setting unit 74 enables the recognition unit 72 (step S114). For this reason, the recognition unit 72 executes recognition processing on, for example, the endoscopic image for display (step S115). In the present embodiment, the recognition unit 72 includes a detection unit 76 and a discrimination unit 77, and the setting unit 74 collectively enables or disables them. Therefore, when there is no operation on the subject, the recognition unit 72 executes both detection and discrimination recognition processing of lesions and the like. Thereafter, since the recognition unit 72 outputs the recognition result to the image generation unit 71, the image generation unit 71 uses the recognition result to generate, for example, an endoscopic image for display with lesions and the like emphasized, and outputs it to the display control unit 66. Thereby, the monitor 18 displays the endoscopic image on the screen. Further, the notification unit 65 notifies the enabled state of the recognition unit 72 (step S116). Here, since the setting unit 74 enables the recognition unit 72, the notification unit 65 displays a character string or the like indicating that the recognition unit 72 is enabled on the screen of the monitor 18. For example, as shown in FIG. 5, the notification unit 65 displays a display 122 of "AI: ON" on the screen of the monitor 18 that displays the endoscopic image 121. Thereby, the state in which the recognition unit 72 is enabled is notified. Note that the endoscopic image 121 is an image without an operation on the subject and without lesions or the like.

[0055] On the other hand, when the discrimination unit 73 detects an operation on the subject (step S113: NO), the setting unit 74 disables the recognition unit 72 (step S117). For this reason, the recognition unit 72 does not execute either detection or discrimination, and the image generation unit 71 outputs the generated endoscopic image for display to the display control unit 66. Thereby, the monitor 18 displays the endoscopic image on the screen. Further, the notification unit 65 notifies the disabled state of the recognition unit 72 (step S118). For example, as shown in FIG. 6, the notification unit 65 displays a display 127 of "AI: OFF" on the screen of the monitor 18 that displays the endoscopic image 126. Thereby, the state in which the recognition unit 72 is disabled is notified. Note that the endoscopic image 126 has a biopsy forceps 128. For this reason, the recognition unit 72 is disabled.

[0056] As described above, in the endoscope system 10, the determination unit 73 determines an operation on the subject, and the setting unit 74 sets the recognition unit 72 to be enabled or disabled using the determination result of the determination unit 73. Therefore, since the endoscope system 10 enables or disables the recognition unit 72 according to the observation situation, there is an advantage that a user such as a doctor does not have to manually set the recognition unit 72 to be enabled or disabled during the observation. However, if only the enabling or disabling of the recognition unit 72 is automatically set, a doctor or the like may misrecognize the result of the recognition process or the result of not performing the recognition process. For example, if there is a preconception that the recognition unit 72 is enabled, and the recognition unit 72 is automatically disabled and the result of the recognition process is not displayed, even if there is a lesion or the like, the doctor or the like may misrecognize that there is no lesion or the like as a result of the recognition unit 72 performing the recognition process. Therefore, in the endoscope system 10, the notification unit 65 notifies the enabled or disabled state of the recognition unit 72. For this reason, by looking at the display 122 of "AI: ON" or the display 127 of "AI: OFF", it is clear whether the recognition unit 72 is performing the recognition process, so the above-mentioned misrecognition can be prevented. As a result, misdiagnosis can be prevented.

[0057] [Second Embodiment] When the setting unit 74 makes an effective or ineffective setting for each recognition process, the notification unit 65 can notify the effective or ineffective state for each recognition process. For example, when the recognition unit 72 includes a detection unit 76 and a discrimination unit 77, and the setting unit 74 sets the detection unit 76 and the discrimination unit 77 to be effective or ineffective respectively, as shown in FIG. 7, the notification unit 65 shows the effective or ineffective state of the detection unit 76 and the discrimination unit 77 in the display column 222 indicating the effective or ineffective state of the recognition unit 72, respectively. A display 223 indicating the effective or ineffective state of the detection unit 76 and a display 224 indicating the effective or ineffective state of the discrimination unit 77 can be displayed respectively, and these states can be notified. For example, the display 223 of "detection: ON" indicates that the detection unit 76 is in an effective state, and the display of "detection: OFF" (not shown) indicates that the detection unit 76 is in an ineffective state. Similarly, the display 224 of "discrimination: ON" indicates that the discrimination unit 77 is in an effective state, and the display of "discrimination: OFF" (not shown) indicates that the discrimination unit 77 is in an ineffective state.

[0058] As described above, when the setting unit 74 makes an effective or ineffective setting for each recognition process, it is possible to enable the recognition process appropriate for the observation situation and disable other recognition processes. Therefore, the endoscope system 10 can provide optimal diagnostic support. Moreover, since the notification unit 65 notifies the effective or ineffective state for each recognition process, a doctor or the like can accurately grasp the effectiveness or ineffectiveness of each recognition process. Therefore, misrecognition of the result of the recognition process or the result of not performing the recognition process can be prevented, and as a result, misdiagnosis can be prevented.

[0059] In the above-described second embodiment, an example in which both the detection unit 76 and the discrimination unit 77 are effective is shown. However, as shown in FIG. 8, depending on the discrimination situation of the discrimination unit 73, it may be appropriate to make only the detection unit 76 effective, it may be appropriate to make only the discrimination unit 77 effective, or it may be appropriate to make both the detection unit 76 and the discrimination unit 77 ineffective.

[0060] "Discrimination situation 1" is a situation where it is appropriate to enable both the detection unit 76 and the discrimination unit 77. For example, when there is no operation on the subject, or when the subject is washed using water or the like. These are situations where there is a high possibility that detection and discrimination of lesions or the like are required.

[0061] "Discrimination situation 2" is a situation where it is appropriate to enable the detection unit 76 and disable the discrimination unit 77. For example, when performing a biopsy or surgery, etc., and setting the bleeding from the subject as the discrimination target (detection target) of the discrimination unit 73, that is, a situation where bleeding detection is necessary for confirmation of hemostasis or the like, but discrimination is not required because a surgery or the like has already been performed. The discrimination unit 73 can discriminate the above situation, for example, according to the type of operation on the subject.

[0062] "Discrimination situation 3" is a situation where it is appropriate to disable the detection unit 76 and enable the discrimination unit 77. For example, when lesions or the like have already been discovered, there is no need to detect lesions or the like again, and in the diagnosis of the discovered lesions or the like, when more detailed diagnostic support is desired from the endoscope system 10. Specifically, it is a case where a coloring agent is sprayed on the subject and the degree of progression of lesions or the like is to be discriminated. The discrimination unit 73 can discriminate the above situation, for example, according to the type of operation on the subject.

[0063] "Discrimination situation 4" is a situation where it is appropriate to disable both the detection unit 76 and the discrimination unit 77. For example, when detection and discrimination of lesions or the like have been completed, and a biopsy or surgery or the like is to be performed where it is necessary to concentrate on the operation. The discrimination unit 73 can discriminate the above situation according to the type of operation. When the discrimination unit 73 detects bleeding from the subject, for example, it becomes discrimination situation 3.

[0064] In the above-described second embodiment, the validity or invalidity of the detection unit 76 and the discrimination unit 77 is represented by "ON" or "OFF", but the validity or invalidity of the detection unit 76 and the discrimination unit 77 can be represented by other displays. For example, instead of the "OFF" display, a display such as "not supported" may be used. Further, even when the detection unit 76 or the discrimination unit 77 is in an effective state, if the accuracy (certainty or reliability) of the recognition process is low, a display such as "Unsupported" can be made instead of the "ON" display. For example, in discrimination situation 3, when the discrimination unit 77 does not support the discrimination of a subject to which a coloring agent has been sprayed, the accuracy of discrimination decreases, so a display of "Detection: OFF" and "Discrimination: Unsupported" can be made. These changes in the display mode and the like can be similarly made for the first embodiment.

[0065] [Third Embodiment] In the above-described first and second embodiments, the notification unit 65 notifies the valid or invalid state of the recognition unit 72, but the notification unit 65 can further notify the presence or absence of an operation discriminated by the discrimination unit 73 or the type of operation. That is, the notification unit 65 can notify the discrimination result of the discrimination unit 73 in addition to the valid or invalid state of the recognition unit 72.

[0066] For example, when the discrimination unit 73 detects an operation on the subject, as shown in FIG. 9, in addition to notifying the valid or invalid state of the recognition unit 72 such as the display 127 of "AI: OFF", the notification unit 65 can notify that an operation on the subject has been detected by a display 301 such as "Actions" ("operation"). Note that the endoscope image 310 is an endoscope image in which a coloring agent 311 has been sprayed on the subject.

[0067] As described above, when the notification unit 65 notifies the presence or absence of an operation on the subject, a doctor or the like can recognize the cause of the validity or invalidity of the recognition unit 72. As a result, it is possible to more reliably prevent misrecognition of the result of the recognition process or the result of not performing the recognition process, and to prevent misdiagnosis.

[0068] In addition, the notification unit 65 can notify the type of operation determined by the determination unit 73. For example, as shown in FIG. 10, the notification unit 65 lists "Biopsy" representing biopsy, "Chromo" representing the spraying of a coloring agent, and "Resection" representing endoscopic resection in the operation type display column 313, and attaches a check mark ("レ") to the operation type determined by the determination unit 73 (in FIG. 10, "Chromo"), etc., so that the operation type determined by the determination unit 73 can be notified.

[0069] As described above, when the determination unit 73 determines the type of operation, if the notification unit 65 notifies the type of operation determined by the determination unit 73, a doctor or the like can recognize in detail the cause of the validity or invalidity of the recognition unit 72. As a result, it is possible to more reliably prevent misrecognition of the result of the recognition process or the result of not performing the recognition process, and to prevent misdiagnosis.

[0070] Note that the notification of the presence or absence of an operation on the subject or the type of operation in the third embodiment can also be applied when, as in the second embodiment, the validity or invalidity is set and notified for each recognition process. In this case, the display column 222 (see FIG. 7) indicating the valid or invalid state of the recognition unit 72 and the display 301 of "Actions" or the operation type display column 313 may be displayed on the monitor 18.

[0071] In addition, in the third embodiment, the presence or absence of an operation on the subject or the type of operation can be notified by a display other than characters or character strings. For example, as shown in FIG. 11, an icon 331 indicating biopsy, an icon 332 representing the spraying of a coloring agent, and an icon 333 representing endoscopic resection are listed in the operation type display column 313, and the icon corresponding to the operation type determined by the determination unit 73 is colored, etc., so that the operation type determined by the determination unit 73 can be notified. Note that in FIG. 11, since there is a biopsy forceps 128 in the endoscopic image 321, the icon 331 indicating biopsy is colored.

[0072] As described above, when indicating the type of operation with an icon, as shown in FIG. 12, only the icon corresponding to the type of operation determined by the determination unit 73 can be displayed in the operation type display column 313. The same applies to the case where the type of operation on the subject is displayed as a character string in the operation type display column 313 (see FIG. 10), and only a character string or the like indicating the type of operation determined by the determination unit 73 can be displayed in the operation type display column 313.

[0073] In the first embodiment, the second embodiment, and the third embodiment, the setting unit 74 sets the recognition process performed by the recognition unit 72 to be valid or invalid. Instead, the setting unit 74 can set the validity or invalidity of the recognition unit 72 by setting the notification of the result of the recognition process to be valid or invalid. In this case, when setting the notification of the result of the recognition process to be invalid, the setting unit 74 stops the output of the recognition result to, for example, the image generation unit 71 or the like while keeping the detection unit 76 and the discrimination unit 77 set to be valid, so that a doctor or the like cannot know the recognition result. As a result, the validity or invalidity of the recognition unit 72 can be set more simply and quickly than when setting the validity or invalidity of the recognition process itself.

[0074] In the first embodiment, the second embodiment, and the third embodiment, the notification unit 65 notifies the valid or invalid state of the recognition unit 72 by the display on the screen of the monitor 18. When notifying the valid or invalid state of the recognition unit 72 in a manner other than the display on the screen of the monitor 18, as shown in FIG. 13, the processor device 16 can include a notification device 171. The notification device 171 is a speaker that emits sound or voice, an indicator composed of a light-emitting element such as an LED, or a vibration element such as a motor or a piezoelectric element. In addition, the notification unit 65 can use an element or the like included in the endoscope system 10 as the notification device 171. Further, a notification device 171 may be provided outside the processor device 16, that is, in the endoscope 12 or the light source device 14.

[0075] Note that, as shown in FIG. 14, the recognition unit 72, the discrimination unit 73, the setting unit 74, and / or the notification unit 65 can be provided, for example, in a medical image processing device 901 that communicates with the processor device 16 and cooperates with the endoscope system 10. Further, as shown in FIG. 15, the recognition unit 72, the discrimination unit 73, the setting unit 74, and / or the notification unit 65 can be provided, for example, directly from the endoscope system 10 (including those without the notification unit 65, etc.) or indirectly from the PACS (Picture Archiving and Communication Systems) 910 in a diagnostic support device 911 that acquires the RAW image captured by the endoscope 12. Further, as shown in FIG. 16, the recognition unit 72, the discrimination unit 73, the setting unit 74, and / or the notification unit 65 can be provided in a medical business support device 930 that is connected via a network 926 to various inspection devices such as a first inspection device 921, a second inspection device 922,..., a K-th inspection device 923, etc., including the endoscope system 10.

[0076] That is, the present invention includes a medical image processing device and its operation method, comprising an image acquisition unit that acquires an image of a subject, a recognition unit that performs recognition processing to recognize the subject using the image, a discrimination unit that discriminates an operation on the subject, a setting unit that sets the recognition unit to be valid or invalid using the discrimination result of the discrimination unit, and a notification unit that notifies the valid or invalid state of the recognition unit. Further, the present invention includes a diagnostic support device and its operation method, comprising an image acquisition unit that acquires an image of a subject, a recognition unit that performs recognition processing to recognize the subject using the image, a discrimination unit that discriminates an operation on the subject, a setting unit that sets the recognition unit to be valid or invalid using the discrimination result of the discrimination unit, and a notification unit that notifies the valid or invalid state of the recognition unit. Similarly, the present invention includes a medical business support device and its operation method, comprising an image acquisition unit that acquires an image of a subject, a recognition unit that performs recognition processing to recognize the subject using the image, a discrimination unit that discriminates an operation on the subject, a setting unit that sets the recognition unit to be valid or invalid using the discrimination result of the discrimination unit, and a notification unit that notifies the valid or invalid state of the recognition unit.

[0077] The present invention also includes a method of operating an endoscope system, which comprises steps of: an image acquisition unit acquiring an image of a subject; a recognition unit performing a recognition process of recognizing the subject using the image; a setting unit having a discrimination unit for discriminating an operation on the subject, and setting the recognition unit to be enabled or disabled using the discrimination result of the discrimination unit; and an alerting unit alerting the enabled or disabled state of the recognition unit. The present invention further includes a processor device comprising an image acquisition unit for acquiring an image of a subject, a recognition unit for performing a recognition process of recognizing the subject using the image, a discrimination unit for discriminating an operation on the subject, a setting unit for setting the recognition unit to be enabled or disabled using the discrimination result of the discrimination unit, and an alerting unit for alerting the enabled or disabled state of the recognition unit, and a method of operating the same.

[0078] Note that a capsule endoscope can be used as the endoscope 12. In this case, the light source device 14 and a part of the processor device 16 can be mounted on the capsule endoscope.

[0079] In the above embodiment, the hardware structure of a processing unit that executes various processes such as the recognition unit 72, the discrimination unit 73, the setting unit 74, or the alerting unit 65 is various processors as described below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (program) and functions as various processing units, a GPU (Graphical Processing Unit), a programmable logic device (PLD) such as an FPGA (Field Programmable Gate Array), which is a processor whose circuit configuration can be changed after manufacture, and an application-specific electric circuit, which is a processor having a circuit configuration designed specifically for executing various processes.

[0080] One processing unit may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same or different types (for example, a plurality of FPGAs, a combination of a CPU and an FPGA, or a combination of a CPU and a GPU, etc.). Further, a plurality of processing units may be composed of one processor. As an example of configuring a plurality of processing units with one processor, firstly, as represented by a computer such as a client or a server, one processor is configured by a combination of one or more CPUs and software, and this processor functions as a plurality of processing units. Secondly, as represented by a System On Chip (SoC), there is a form in which a processor that realizes the functions of the entire system including a plurality of processing units with one IC (Integrated Circuit) chip is used. Thus, various processing units are configured by using one or more of the above various processors as a hardware structure.

[0081] Furthermore, the hardware structure of these various processors is more specifically an electrical circuit (circuitry) in the form of a combination of circuit elements such as semiconductor elements.

[0082] In addition, the present invention can be used not only in an endoscope system for acquiring an endoscope image, a processor device, and other related devices, but also in a system or device for acquiring a medical image (including a video) other than an endoscope image. For example, the present invention can be applied to an ultrasonic inspection device, an X-ray imaging device (including a CT (Computed Tomography) inspection device and a mammography device, etc.), an MRI (magnetic resonance imaging) device, etc.

Explanation of Reference Numerals

[0083] 10 Endoscope system 12 Endoscope 12a Insertion part 12b Operation part 12c Bending part 12d Tip part 12e Angle knob Insertion port for treatment instrument at 12f Zoom operation unit at 13a Water supply button at 13b Light source device at 14 Processor device at 16 Water supply tank at 17 Monitor at 18 Console at 19 Light source unit at 20 Light source control unit at 22 Illumination optical system at 30a Imaging optical system at 30b Illumination lens at 45 Objective lens at 46 Zoom lens at 47 Image sensor at 48 Control unit at 52 Image acquisition unit at 54 DSP at 56 Noise reduction unit at 58 Conversion unit at 59 Image processing unit at 61 Notification unit at 65 Display control unit at 66 Image generation unit at 71 Recognition unit at 72 Discrimination unit at 73 Setting unit at 74 Detection unit at 76 Identification unit at 77 Endoscopic images 121, 126, 310, 321 Display 122, 127, 223, 224, 301 Forceps at 128 Notification device at 171 Display column at 222 Dye agent at 311 Type display column at 313 Icons 331, 332, 333 Medical image processing device at 901 PACS at 910 Diagnosis support device at 911 First inspection device at 921 Second inspection device at 922 Kth inspection device at 923 Network at 926 930 Medical service support device Steps S111 to S118 of the operation

Claims

1. Comprising at least one processor, wherein the at least one processor acquires an endoscopic image of a subject taken by an endoscope, inputs the endoscopic image into a trained model that performs recognition processing on a part having specific features in the endoscopic image, acquires the result of the recognition processing from the trained model, sets the recognition processing to be valid or invalid based on an operation on the subject, wherein the result of the recognition processing is information indicating a lesion included in the endoscopic image, or information indicating the type or degree of progression of the lesion, when the recognition processing is set to be valid, outputs information indicating that the recognition processing is valid and the result of the recognition processing to a monitor, and when the recognition processing is set to be invalid, outputs information indicating that the recognition processing is invalid to the monitor, an image processing apparatus.

2. The image processing apparatus according to claim 1, wherein the invalidation of the recognition processing is not performing the recognition processing or not notifying the result of the recognition processing.

3. The image processing apparatus according to claim 2, wherein the notification indicates that the recognition processing using artificial intelligence is invalid.

4. The image processing apparatus according to any one of claims 1 to 3, wherein information indicating the lesion is output to the monitor as the result of the recognition processing.

5. The image processing apparatus according to claim 4, wherein the operation on the subject is the use of a treatment instrument, the spraying of a liquid, or the cleaning of the subject.

6. The image processing apparatus according to claim 4 or 5, wherein the operation on the subject is detected from the endoscopic image.

7. The image processing apparatus according to any one of claims 1 to 3, wherein information indicating the type or degree of progression of the lesion is output to the monitor as the result of the recognition processing.

8. The image processing apparatus according to claim 7, wherein the operation on the subject is the use of a treatment instrument, the spraying of a liquid, or the cleaning of the subject.

9. The image processing apparatus according to claim 7 or 8, wherein the operation on the subject is detected from the endoscopic image.

10. The image processing apparatus according to claim 1, wherein the operation on the subject is the use of a treatment instrument, the spraying of a liquid, or the cleaning of the subject.

11. The image processing apparatus according to any one of claims 1 to 10, wherein the recognition processing is set to be invalid when a treatment instrument is detected.

12. The treatment instrument is the image processing apparatus according to claim 10, which is detected from the endoscopic image.

13. detect a plurality of types of the treatment instruments from the endoscopic image, The image processing apparatus according to claim 11 or 12, wherein when the specific type of the treatment instrument is detected, the recognition process is set to be invalid.

14. The image processing apparatus according to any one of claims 1 to 10, wherein the recognition process is set to be valid when no treatment instrument is detected.

15. An image processing apparatus according to any one of claims 1 to 14, an endoscope that acquires an image of the subject, and a processor device that controls the endoscope to generate the endoscopic image, A medical service support device in which the image processing apparatus and the processor device are connected via a network.

Images