Image processing device

Abstract

The present invention provides an image processing device that prevents misidentification of results when recognition processing is performed and when recognition processing is not performed. [Solution] The image acquisition unit 54 acquires an image of the subject taken by the endoscope. The identification unit 73 uses the setting information of the operating mode, the model of the endoscope used to photograph the subject, whether or not a drug has been administered to the subject, or the setting information of the image magnification process to determine whether or not the recognition process for recognizing the subject is functioning. Information showing some or all of the information used for the determination, and information related to the recognition process are displayed on the monitor 18. The recognition process is a process that uses the image to detect the presence or absence of a lesion or a candidate lesion, or a process that uses the image to differentiate the type or progression of a lesion or a candidate lesion.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus that performs recognition processing of a subject using an image captured with an endoscope.

Background Art

[0002] In the medical field, an endoscope system including a light source device, an endoscope, and a processor device has become widespread. The light source device generates illumination light. The endoscope captures an image of a subject using an image sensor. The processor device performs generation of an image and other image processing.

[0003] The endoscope system may have additional functions in addition to simply capturing an image of a subject for observation. For example, an endoscope system having an "input mode" for operating the position and orientation of a treatment tool with a touch panel is known (Patent Document 1). In the endoscope system of this Patent Document 1, the state of the above input mode is displayed on the screen. Also, an endoscope system having a function of displaying an image indicating the state of bending of the endoscope tip portion and a character string notifying that calibration has been completed on the screen to notify an abnormality of the system is known (Patent Document 2). In addition, an endoscope system that displays the on or off state of a foot switch on the screen (Patent Document 3), an endoscope system that displays a mark indicating that air is being supplied on the screen (Patent Document 4), and an endoscope system that displays the on or off state of recording on the screen (Patent Document 5) are known.

[0004] In recent years, an endoscope system that supports diagnosis by calculating biological function information using an image of a subject is also known (Patent Document 6).

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

[0006] When obtaining information to support diagnosis (hereinafter referred to as diagnostic support information) by recognizing a subject or part of a subject with specific characteristics using medical images such as images taken using an endoscope (hereinafter referred to as endoscopic images), it is necessary to use medical images that can be processed by recognition processing when taken under specific conditions. This is because if medical images that cannot be processed by recognition processing are used, even if a result of recognition processing is obtained, the result may be inaccurate.

[0007] However, it is not always possible to acquire medical images in which recognition processing can function effectively. As a result, inaccurate recognition processing, or diagnostic support information calculated using inaccurate recognition processing results, can actually hinder the observation or diagnosis of the subject. For example, in an endoscope system with multiple observation modes using different types of illumination light, highly accurate recognition processing results may be obtained for endoscopic images obtained in a specific observation mode, but the accuracy of recognition processing may be low when using endoscopic images obtained in other observation modes.

[0008] To avoid providing inaccurate recognition processing results as described above, one possible approach is to refrain from performing recognition processing when medical images suitable for recognition processing cannot be acquired. However, simply refraining from performing recognition processing may lead users, such as physicians, to misinterpret the results of recognition processing or the results of not performing recognition processing. For example, if a device that performs recognition processing to detect potential lesions is being used, simply refraining from performing recognition processing may lead physicians to mistakenly believe that no potential lesions were detected as a result of the recognition processing. In other words, physicians may not be able to determine whether the results of recognition processing are not displayed because the processing was not performed, or because there were no objects to recognize as a result of the recognition processing.

[0009] The present invention aims to provide an image processing device that prevents misidentification of results in recognition processing and results in which recognition processing was not performed. [Means for solving the problem]

[0010] The image processing apparatus of the present invention comprises a processor, which acquires an image of a subject captured by an endoscope, and determines whether or not a recognition process for recognizing a subject is functioning using setting information for the operating mode, the model of the endoscope used to capture the subject, whether or not a drug has been administered to the subject, or setting information for image magnification processing. The processor displays on a monitor information indicating some or all of the information used for the determination, along with information related to the recognition process. The recognition process is a process that uses the image to detect the presence or absence of a lesion or a candidate lesion, or a process that uses the image to differentiate the type or progression of a lesion or a candidate lesion.

[0011] The processor preferably determines whether or not the recognition process is functioning based on the setting information of the operating mode. The operating mode is preferably a normal observation mode in which the subject is photographed with white light, or a special observation mode in which the subject is photographed using illumination light having a specific wavelength band. When the operating mode is the special observation mode, the processor preferably displays information indicating that it is the special observation mode and information regarding the recognition process on the monitor. In the special observation mode, it is preferable to photograph the subject using illumination light that contains more blue or purple components compared to white light.

[0012] The processor preferably obtains operating mode setting information from a processor device that controls the endoscope. The processor preferably obtains operating mode setting information by analyzing images. The processor preferably determines whether or not the recognition process is functional based on the model of the endoscope. The processor preferably displays information indicating the model of the endoscope and information related to the recognition process on a monitor.

[0013] The processor preferably determines whether or not the recognition process will function based on whether or not a drug has been administered. If the subject has been administered a drug, the processor preferably displays on the monitor information indicating that the subject has been administered a drug, as well as information about the subject.

[0014] The processor preferably determines whether or not the recognition process is functional based on the image magnification processing information. The processor also preferably determines whether or not the recognition process is functional based on the operating mode setting information and the image magnification processing information.

[0015] It is preferable to display on the monitor information indicating that the magnification is the first magnification and information related to the recognition process when the magnification is the first magnification, and to display on the monitor information indicating that the magnification is the second magnification and information related to the recognition process when the magnification is the second magnification, which is greater than the first magnification. It is preferable that the magnification is changed by moving the zoom lens of the endoscope. It is preferable that the processor obtains information on the image magnification process from a processor device that controls the endoscope.

[0016] The processor preferably displays on the monitor information indicating all the information used for discrimination and information related to the recognition process. It is preferable to display on the monitor information indicating at least one of the following as information indicating some of the information used for discrimination: operating mode setting information, endoscope model, whether or not drugs were administered to the subject, or image magnification processing setting information.

[0017] The information regarding the recognition process preferably includes at least one of the following: information indicating that the recognition process is functioning, information indicating that the recognition process is not functioning, and the result of the recognition process. When the processor determines that the recognition process is functioning, it is preferable to display on the monitor both the information indicating that the recognition process is functioning and the result of the recognition process as information indicating that the recognition process is functioning. As a result of the recognition process, it is preferable to display on the monitor information indicating the location of the lesion or a candidate lesion. When the processor determines that the recognition process is not functioning, it is preferable to display on the monitor information indicating that the recognition process is not functioning.

[0018] The processor preferably displays information related to the recognition process on a monitor when the information used for discrimination is switched. The recognition process preferably involves detecting the presence or absence of a lesion or candidate lesion using an image. The recognition process preferably involves differentiating the type or progression of a lesion or candidate lesion using an image. The recognition process preferably involves being performed by artificial intelligence with learning capabilities.

[0019] The processor preferably determines whether each of a plurality of recognition processes that function for different types of images functions or not, and displays information regarding the recognition process on a monitor for each recognition process.

Effect of the Invention

[0020] According to the present invention, an endoscope system capable of preventing misrecognition of the result of performing a recognition process and the result of not performing the recognition process can be provided.

Brief Description of the Drawings

[0021] [Figure 1] It is an external view of an endoscope system. [Figure 2] It is a block diagram of an endoscope system. [Figure 3] It is a block diagram of an image processing unit. [Figure 4] It is a flowchart of the first embodiment. [Figure 5] It is an example of a display for notifying that a recognition process functions. [Figure 6] It is an example of a display for notifying that a recognition process functions. [Figure 7] It is an example of a display for notifying that a recognition result does not function. [Figure 8] It is an example of a display for showing an operation mode. [Figure 9] It is an example of a display for showing that a recognition process functions by an icon. [Figure 10] It is an example of a display for showing that a recognition process does not function by an icon. [Figure 11] It is another example of a display for showing that a recognition process functions. [Figure 12] It is another example of a display for showing that a recognition process does not function. [Figure 13] It is a block diagram of a processor device when notifying whether a recognition process functions or not by a method other than display on a monitor. [Figure 14] It is a flowchart of the second embodiment. [Figure 15]This is a flowchart of the third embodiment. [Figure 16] This is a flowchart of the fourth embodiment. [Figure 17] This is a flowchart of the fifth embodiment. [Figure 18] This is a flowchart of the sixth embodiment. [Figure 19] This is an explanatory diagram showing four types of notification methods. [Figure 20] This is a block diagram of the image processing unit when the recognition unit includes multiple AIs. [Figure 21] This is a block diagram of an image processing unit having multiple recognition units. [Figure 22] This is an explanatory diagram showing a medical image processing device. [Figure 23] This is an explanatory diagram showing a diagnostic support device. [Figure 24] This is an explanatory diagram showing a medical support device. [Modes for carrying out the invention]

[0022] [First Embodiment] As shown in Figure 1, the endoscope system 10 (endoscope device) comprises an endoscope 12, a light source device 14, a processor device 16, a monitor 18, and a console 19. The endoscope 12 photographs the subject. The light source device 14 generates illumination light. The processor device 16 performs system control and image processing of the endoscope system 10. The monitor 18 is a display unit that displays images etc. from the endoscope 12. The console 19 is an input device for inputting settings etc. to the processor device 16, etc.

[0023] The endoscope 12 has an insertion section 12a for insertion into the subject, an operating section 12b provided at the base of the insertion section 12a, a bending section 12c provided at the tip of the insertion section 12a, and a tip section 12d. By operating the angle knob 12e of the operating section 12b, the bending section 12c bends. As a result, the tip section 12d points in the desired direction. In addition to the angle knob 12e, the operating section 12b is also provided with a zoom operating section 13a and a mode switching operating section 13b. By operating the zoom operating section 13a, the subject can be enlarged or reduced for imaging. Also, by operating the mode switching operating section 13b, the operating mode is switched. The operating mode refers to any operating mode of the endoscope 12, the light source device 14, or the processor device 16, or a combination of two or more of these operating modes.

[0024] Specifically, the operating mode is determined by the type of illumination light used to photograph the subject, the presence or absence of optical magnification processing, or the presence or absence of image processing (including combinations thereof). The same applies when, for convenience, we refer to the operating mode of the endoscope 12, the operating mode of the light source device 14, or the operating mode of the processor device 16. The type of illumination light refers to the type of illumination light that can be distinguished by its wavelength, wavelength band, or spectral distribution. Optical magnification processing refers to magnification processing using the zoom lens 47 (see Figure 2), i.e., optical zoom. The presence or absence of optical magnification processing refers to whether or not optical magnification is performed. The presence or absence of optical magnification processing includes cases where optical magnification processing is performed when the magnification is above a certain magnification ratio, and conversely, cases where optical magnification processing is not performed when optical zoom is used but the magnification is below a certain magnification ratio (i.e., cases where the magnitude of the magnification ratio of optical magnification processing is considered). The endoscope system 10 can perform electronic magnification processing instead of, or in combination with, optical magnification processing. Electronic magnification processing refers to the process of enlarging a portion of the endoscopic image, i.e., electronic zoom. Image processing that defines the operating mode refers to some or all of the various processes performed on the endoscopic image by the image processing unit 61 (see Figure 2). The presence or absence of image processing refers to whether or not a specific image processing is performed. The presence or absence of image processing includes cases where image processing is considered to be performed when it is performed at a certain intensity or higher, and no image processing is considered to be performed when it is performed at a lower intensity (i.e., cases where the intensity of the image processing is taken into consideration).

[0025] The endoscope system 10 includes, for example, a normal observation mode in which the subject is photographed using white light and displayed in natural colors, and a special observation mode in which the subject is photographed and displayed using illumination light having a specific wavelength band different from white light. The special observation mode includes, for example, a mode that makes it easier to observe specific tissues or structures such as fine blood vessels by photographing the subject using illumination light that contains more blue or purple components compared to the white light used in the normal observation mode, or a mode that modulates the color of the endoscopic image to make it easier to observe specific lesions, etc. Hereinafter, unless otherwise specifically required, the above subdivided special observation modes will simply be referred to as special observation modes, and only the normal observation mode and special observation modes will be distinguished.

[0026] As shown in Figure 2, the light source device 14 includes a light source unit 20 that emits illumination light and a light source control unit 22 that controls the operation of the light source unit 20.

[0027] The light source unit 20 emits illumination light to illuminate the subject. The emission of illumination light includes the emission of excitation light used to emit the illumination light. The light source unit 20 includes, for example, a laser diode (hereinafter referred to as LD), an LED (Light Emitting Diode), a xenon lamp, or a halogen lamp, and emits at least white illumination light, or excitation light used to emit white illumination light. White includes so-called pseudo-white, which is substantially equivalent to white when photographing a subject using the endoscope 12. The light source unit 20 may optionally include a phosphor that emits light when irradiated with excitation light, or an optical filter that adjusts the wavelength band, spectral spectrum, or light intensity of the illumination light or excitation light. In addition, the light source unit 20 can emit light having a specific wavelength band necessary for capturing images used to calculate biological information such as the oxygen saturation of hemoglobin contained in the subject.

[0028] In this embodiment, the light source unit 20 has four colored LEDs: V-LED20a, B-LED20b, G-LED20c, and R-LED20d. V-LED20a emits violet light VL with a central wavelength of 405nm and a wavelength band of 380-420nm. B-LED20b emits blue light BL with a central wavelength of 460nm and a wavelength band of 420-500nm. G-LED20c emits green light GL with a wavelength band of 480-600nm. R-LED20d emits red light RL with a central wavelength of 620-630nm and a wavelength band of 600-650nm. The central wavelengths of V-LED20a and B-LED20b have a range of approximately ±20nm, preferably from approximately ±5nm to approximately ±10nm.

[0029] The light source control unit 22 controls the timing of turning on, turning off, or shielding each light source constituting the light source unit 20, as well as the amount of light emitted. As a result, the light source unit 20 can emit multiple types of illumination light with different spectral characteristics. In this embodiment, the light source control unit 22 adjusts the spectral characteristics of the illumination light by inputting independent control signals to each of the LEDs 20a to 20d for turning on or off, the amount of light emitted when lit, and inserting or removing the optical filter. As a result, in normal observation mode, the light source unit 20 emits white light. The light source unit 20 can also emit illumination light consisting of at least narrowband violet light. "Narrowband" means that, in relation to the characteristics of the subject and / or the spectral characteristics of the color filter of the image sensor 48, it is substantially a single wavelength band. For example, if the wavelength band is approximately ±20 nm or less (preferably approximately ±10 nm or less) with respect to the center wavelength, this light is narrowband.

[0030] The tip 12d of the endoscope 12 is equipped with an illumination optical system 30a and an imaging optical system 30b. The illumination optical system 30a has an illumination lens 45, and illumination light is emitted towards the subject through this illumination lens 45.

[0031] The imaging optical system 30b includes an objective lens 46, a zoom lens 47, and an image sensor 48. The image sensor 48 captures the subject using reflected light from the subject (including scattered light, fluorescence emitted by the subject, or fluorescence caused by drugs administered to the subject) via the objective lens 46 and the zoom lens 47. The zoom lens 47 moves by operating the zoom control unit 13a, enlarging or reducing the image of the subject.

[0032] The image sensor 48 has one color filter from among multiple color filters for each pixel. In this embodiment, the image sensor 48 is a color sensor having primary color filters. Specifically, the image sensor 48 has R pixels having a red color filter (R filter), G pixels having a green color filter (G filter), and B pixels having a blue color filter (B filter).

[0033] The image sensor 48 can be a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor. While the image sensor 48 in this embodiment is a primary color sensor, a complementary color sensor can also be used. A complementary color sensor, for example, has cyan pixels with a cyan color filter, magenta pixels with a magenta color filter, yellow pixels with a yellow color filter, and green pixels with a green color filter. When using a complementary color sensor, the images obtained from the pixels of each color can be converted to images similar to those obtained with a primary color sensor by performing a complementary-to-primary color conversion. The same applies when a primary or complementary color sensor has one or more types of pixels with characteristics other than those described above, such as W pixels (white pixels that receive light across almost the entire wavelength band). Furthermore, while the image sensor 48 in this embodiment is a color sensor, a monochrome sensor without a color filter may also be used.

[0034] The processor device 16 includes a control unit 52, an image acquisition unit 54, an image processing unit 61, a notification unit 65, and a display control unit 66 (see Figure 2).

[0035] The control unit 52 performs overall control of the endoscope system 10, including synchronized control of the illumination timing and the timing of image capture. When various settings are entered using the console 19 or the like, the control unit 52 inputs those settings to the respective parts of the endoscope system 10, such as the light source control unit 22, the image sensor 48, or the image processing unit 61.

[0036] The image acquisition unit 54 acquires images from the image sensor 48 that capture the subject using pixels of each color, i.e., RAW images. Furthermore, RAW images are images before demosaicing is performed. Images that have undergone arbitrary processing such as noise reduction on the images acquired from the image sensor 48 are also included in the RAW images, as long as they are images before demosaicing is performed.

[0037] The image acquisition unit 54 includes a DSP (Digital Signal Processor) 56, a noise reduction unit 58, and a conversion unit 59 in order to generate an endoscopic image by performing various processing on the acquired RAW image as needed.

[0038] The DSP56 includes, for example, an offset processing unit, a defect correction processing unit, a demosaicing processing unit, an interpolation processing unit, a linear matrix processing unit, and a YC conversion processing unit (none of which are shown in the diagram). The DSP56 uses these to perform various processes on a RAW image or an image generated using a RAW image.

[0039] The offset processing unit performs offset processing on the RAW image. Offset processing is a process that reduces the dark current component from the RAW image and sets an accurate zero level. Offset processing is sometimes referred to as clamping processing. The defect correction processing unit performs defect correction processing on the RAW image. Defect correction processing is a process that corrects or generates the pixel values ​​of the RAW pixels corresponding to the defective pixels of the image sensor 48 when the image sensor 48 contains pixels (defective pixels) that have defects due to the manufacturing process or changes over time. The demosaicing processing unit performs demosaicing processing on the RAW images of each color corresponding to each color filter. Demosaicing processing is a process that generates pixel values ​​that are missing in the RAW image due to the arrangement of the color filters by interpolation. The linear matrix processing unit performs linear matrix processing on the endoscopic image generated by assigning one or more RAW images to each RGB color channel. Linear matrix processing is a process that improves the color reproducibility of the endoscopic image. The YC conversion processing unit converts the endoscopic image generated by assigning one or more RAW images to each RGB color channel into an endoscopic image having a luminance channel Y and chrominance channels Cb and Cr.

[0040] The noise reduction unit 58 applies noise reduction processing to the endoscopic image having a luminance channel Y, a chrominance channel Cb, and a chrominance channel Cr, for example, using a moving average method or a median filter method. The conversion unit 59 converts the luminance channel Y, chrominance channel Cb, and chrominance channel Cr after noise reduction processing back into an endoscopic image having channels for each of the BGR colors.

[0041] The image processing unit 61 performs necessary image processing on the endoscopic image output by the image acquisition unit 54 according to the observation mode, etc. The image processing unit 61 also performs recognition processing using the endoscopic image to recognize a subject or part of a subject that has specific characteristics. Specifically, as shown in Figure 3, the image processing unit 61 includes an image generation unit 71, a recognition unit 72, and an identification unit 73, etc.

[0042] The image generation unit 71 acquires endoscopic images from the image acquisition unit 54 and generates endoscopic images for display on the monitor 18, etc. For example, the image generation unit 71 acquires B images taken using B pixels, G images taken using G pixels, and R images taken using R pixels from the image acquisition unit 54, and generates an endoscopic image for display using all or part of these.

[0043] Furthermore, when generating an endoscopic image for display, the image generation unit 71 applies necessary image processing to the endoscopic image acquired from the image acquisition unit 54, or to the image generated using the endoscopic image acquired from the image acquisition unit 54, according to the observation mode, etc. The image processing performed by the image generation unit 71 is, for example, enhancement processing to highlight the subject or a part of the subject. Enhancement means distinguishing a specific part from other tissues or structures, etc., so that information about that specific part can be obtained. For example, processing such as enclosing a part with specific features in a frame, showing its outline, or changing the color or brightness relative to other parts (e.g., normal mucosa) is enhancement processing. "Making information obtainable" includes making the position, shape, color or brightness, and / or size (range), etc., of a specific part recognizable, as well as making biological function information (e.g., oxygen saturation or blood vessel density, etc.) of a specific part known.

[0044] The recognition unit 72 performs recognition processing to recognize the subject using the image of the subject that has been captured. "Recognizing the subject" means detecting the presence or absence of a part of the subject having a specific characteristic (including detecting the entire subject), differentiating the type or progression of the part of the subject having a specific characteristic (including differentiating the entire subject), and / or obtaining (calculating, etc.) biological function information for part or all of the subject. A part of the subject having a specific characteristic is, for example, a lesion or a candidate for a lesion (hereinafter referred to as lesion, etc.). That is, the recognition unit 72 can recognize the presence or absence of lesion, etc. through recognition processing. The recognition unit 72 can also recognize the type or progression of lesion, etc. through recognition processing. Recognition of the type of lesion, etc. means differentiating the type of lesion, etc., such as adenoma, hyperplastic polyp, or cancer, if the lesion, etc. is a polyp. Recognizing the progression of a lesion means determining the stage of cancer if the lesion is cancerous, or differentiating it according to NICE (NBI (Narrow Band Imaging) International Colorectal Endoscopic Classification) classification or JNET (The Japan NBI Expert Team) classification, etc. In this embodiment, the image of the subject is an endoscopic image acquired by the image acquisition unit 54, or an endoscopic image generated by the image generation unit 71. In this embodiment, the recognition unit 72 detects lesions through recognition processing.

[0045] The recognition unit 72 is, for example, artificial intelligence (AI) with learning capabilities. Specifically, the recognition unit 72 is an AI trained using machine learning algorithms such as neural networks (NN), convolutional neural networks (CNN), adaboost, and random forests. Furthermore, since the recognition unit 72 is trained to perform recognition processing using specific images, even if it can perform recognition processing using other images, the accuracy of the recognition processing results may be low. In this embodiment, the recognition unit 72 is an AI trained to detect lesions, etc., using endoscopic images for display obtained in normal observation mode. "Having learning capabilities" means being able to learn, and includes being trained. Note that instead of being composed of AI, the recognition unit 72 can be configured to calculate feature quantities from images and perform detection, etc., using the calculated feature quantities.

[0046] Furthermore, the recognition unit 72 inputs the results of the recognition process to the image generation unit 71 or the display control unit 66 according to the content of the recognition process. When the recognition unit 72 inputs the results of the recognition process to the image generation unit 71, the image generation unit 71 generates an endoscopic image for display that reflects the results of the recognition process. When the recognition unit 72 inputs the results of the recognition process to the display control unit 66, the display control unit 66 displays the results of the recognition process on the monitor 18 screen along with the endoscopic image acquired from the image generation unit 71. The recognition unit 72 inputs the results of the recognition process to the image generation unit 71, for example, when changing the color of the endoscopic image for display using the values ​​of biological function information that are the results of the recognition process. The recognition unit 72 inputs the results of the recognition process to the display control unit 66, for example, when indicating the location of lesions, etc., that are the results of the recognition process by displaying a frame superimposed on the endoscopic image. In this embodiment, the recognition unit 72 inputs information such as the location of detected lesions, etc., that are the results of the recognition process, to the image generation unit 71. The image generation unit 71 then generates an endoscopic image for display in which areas with lesions or other abnormalities have been enhanced.

[0047] The identification unit 73 identifies the type of image of the subject. "Identification of image type" means identifying whether or not the recognition processing performed by the recognition unit 72 is functional for a given type of image. In other words, the identification unit 73 identifies two types of images: images for which recognition processing is functional and images for which recognition processing is not functional. Images for which recognition processing is functional are those for which results of a specific accuracy or higher can be expected when recognition processing is performed, and images for which recognition processing is not functional are those for which results of less than a specific accuracy can be expected when recognition processing is performed. For example, if the recognition unit 72 is an AI, images of the same type as those used by the recognition unit 72 for training are images for which recognition processing is functional, and other images are images for which recognition processing is not functional. Specific accuracy is a standard value such as the accuracy or reliability of the recognition processing that is sufficient to support diagnosis, etc. In this embodiment, the recognition unit 72 identifies endoscopic images obtained in normal observation mode as images for which recognition processing is functional, and endoscopic images obtained in special observation mode as images for which recognition processing is not functional.

[0048] Specifically, the identification unit 73 identifies the type of image by determining the operating mode, the model of the endoscope 12 used to photograph the subject, or whether or not a drug has been administered to the subject.

[0049] The identification unit 73 can determine the operating mode using the operating mode setting information (including electrical signals generated by switching operating modes) obtained from the control unit 52. For example, if the operating mode is normal observation mode, the identification unit 73 identifies the image obtained by photographing the subject as an image in which the recognition process is functioning. The identification unit 73 can also determine the operating mode by analyzing the endoscopic image obtained from the image acquisition unit 54 or the endoscopic image generated by the image generation unit 71. For example, it can analyze the color, brightness distribution (unevenness of illumination), or size of the subject (presence or absence of pit patterns or average thickness, etc.) of the endoscopic image and use the results to determine the operating mode.

[0050] The identification unit 73 can determine the model of the endoscope 12 using information about the model of the endoscope 12 obtained from the control unit 52. Once the model of the endoscope 12 is determined, the type of image sensor 48 (such as the characteristics of the color filter) mounted on the endoscope 12 is determined. Therefore, the identification unit 73 can determine whether or not recognition processing can function on images taken using that model of endoscope 12. The information about the model of the endoscope 12 is obtained by the control unit 52 from the endoscope 12 when the endoscope 12 is connected to the light source device 14 or the processor device 16.

[0051] The identification unit 73 can determine whether or not a drug has been administered to the subject by acquiring setting information related to drug administration obtained from the control unit 52. Furthermore, the identification unit 73 can determine whether or not a drug has been administered to the subject by analyzing the endoscopic image acquired from the image acquisition unit 54 or the endoscopic image generated by the image generation unit 71. Drug administration to the subject includes not only spraying a dye such as indigo carmine onto the surface of the subject, but also intravenously injecting a fluorescent agent such as indocyanine green (ICG) into the subject.

[0052] In this embodiment, the identification unit 73 identifies the type of image by determining whether the operating mode is the normal observation mode or the special observation mode. The identification unit 73 also uses the operating mode setting information obtained from the control unit 52 to determine the operating mode.

[0053] As described above, the identification unit 73 identifies the type of image, and if the identified image is of a type that can be processed by recognition processing, and the recognition unit 72 is in a stopped state (not performing recognition processing), it activates the recognition unit 72. On the other hand, if the identified image is of a type that cannot be processed by recognition processing, and the recognition unit 72 is in an activated state (performing recognition processing), the identification unit 73 stops the recognition unit 72. As a result, the recognition unit 72 automatically activates when the image type is of a type that can be processed by recognition processing, and automatically stops when the image type is of a type that cannot be processed by recognition processing.

[0054] The notification unit 65 obtains information from the identification unit 73 regarding whether or not the recognition process is functioning. The notification unit 65 then notifies whether or not the recognition process is functioning for a specific type of image identified by the identification unit 73. "Notifying" whether or not the recognition process is functioning means making it possible for the user, such as a doctor, to know the status of whether or not the recognition process is functioning. For example, the notification unit 65 can notify whether or not the recognition process is functioning by displaying, not displaying, or changing the display of messages (strings of characters), characters, figures, and / or symbols (including displays of marks, icons, or indicators) on the screen of the monitor 18. In addition, the notification unit 65 can notify whether or not the recognition process is functioning by turning on, turning off, or flashing a lamp, making sounds (including voice), vibrating a component with a vibration function, or changing these. Of course, the notification unit 65 can also notify whether or not the recognition process is functioning by combining strings of characters, etc., with the lighting of lamps, etc. In this embodiment, the notification unit 65 displays on the monitor 18 screen whether or not the recognition process is functioning.

[0055] Furthermore, the notification unit 65 notifies whether the recognition process will function or not, at least when the image type changes. A change in image type occurs when the identification result in the identification unit 73 changes. In this embodiment, the notification unit 65 continuously notifies whether the recognition process will function or not, including when the image type changes. However, depending on the settings, the notification unit 65 can notify whether the recognition process will function or not for a predetermined time when the image type changes, and stop notifying whether the recognition process will function or not during other periods. This setting is useful when continuous notification is considered bothersome.

[0056] The display control unit 66 converts the endoscopic image output by the image processing unit 61 into a format suitable for display and outputs it to the monitor 18. As a result, the monitor 18 displays the endoscopic image. In this embodiment, the notification unit 65 inputs information indicating whether or not the recognition process is functioning. Therefore, the display control unit 66 displays this information on the screen of the monitor 18.

[0057] The endoscope system 10 configured as described above operates as follows to notify whether or not the recognition process is functioning. As shown in Figure 4, the operating mode is set by operating the mode switching operation unit 13b (step S110), and when a subject is photographed (step S111), the image acquisition unit 54 acquires the endoscope image. Then, the image generation unit 71 performs the necessary image processing to generate an endoscope image for display (step S112). Meanwhile, the identification unit 73 identifies whether or not the image is one in which the recognition process is functioning (step S113 (identification step)). If the recognition process is functioning (step S113: YES), the recognition unit 72 executes the recognition process (step S114), and the notification unit 65 notifies that the recognition process is functioning (step S115).

[0058] More specifically, when the operating mode is normal observation mode, the image generation unit 71 generates a normal observation image 121, which is an endoscopic image that displays the subject in natural colors. Therefore, as shown in Figure 5, the display control unit 66 displays the normal observation image 121 on the screen of the monitor 18. In addition, the identification unit 73 of this embodiment identifies the normal observation image 121 as an image for which recognition processing is possible, so the recognition unit 72 executes the recognition processing. Therefore, the notification unit 65 notifies this fact, for example, by displaying "AI in operation" 122. As a result, doctors and others can correctly recognize the result of the recognition processing. That is, since the normal observation image 121 does not show lesions, etc., which are the result of the recognition processing, if there is no notification by displaying "AI in operation" 122, doctors and others cannot recognize whether the lesions, etc., are not shown as a result of recognition processing or as a result of not performing recognition processing. However, the notification by the "AI in operation" display 122 allows doctors and others to correctly recognize that the recognition unit 72 has performed recognition processing and that this normal observation image 121 shows no lesions or other abnormalities.

[0059] Furthermore, as shown in Figure 6, when a subject with lesions, etc. 123 is photographed in normal observation mode, the image generation unit 71 generates a normal observation image 124 with the lesions, etc. 123 emphasized, based on the results of the recognition processing, and the display control unit 66 displays this on the monitor 18. In this case, without the notification by the "AI in operation" display 122, it may be difficult for doctors, etc. to recognize whether the lesions, etc. 123 in the normal observation image 124 are the original lesions, etc. 123 that have not undergone any enhancement processing. However, with the notification by the "AI in operation" display 122, doctors, etc. can correctly recognize that the lesions, etc. 123 are the ones that have been emphasized as a result of the recognition processing performed by the recognition unit 72.

[0060] On the other hand, as shown in Figure 7, when observing a subject in special observation mode, the image generation unit 71 generates a special observation image 131 that is different in appearance from the normal observation image 121, for example, and the display control unit 66 displays this on the screen of the monitor 18. In this case, the identification unit 73 of this embodiment identifies the special observation image 131 as an image in which recognition processing does not function, so the recognition unit 72 does not perform recognition processing. For this reason, the notification unit 65 notifies the system of this fact, for example, by displaying "AI stopped" 132. As a result, doctors and others can correctly recognize the results of the recognition processing. That is, since the special observation image 131 does not show lesions, etc., which are the result of recognition processing, if there is no notification by displaying "AI stopped" 132, doctors and others cannot recognize whether the lesions, etc., are not shown as a result of performing recognition processing or as a result of not performing recognition processing. In particular, if there is a preconceived notion that the endoscopy system 10 performs recognition processing, doctors and other medical professionals may mistakenly believe that the special observation image 131 shows no lesions, etc., as a result of recognition processing, even though the recognition unit 72 is not performing recognition processing, thus reducing the accuracy of the diagnosis. However, if there is notification from the "AI stopped" display 132, doctors and other medical professionals can recognize that recognition processing is not being performed, and therefore will observe the special observation image 131 without making the above-mentioned mistake, and thus be able to make a correct diagnosis.

[0061] As described above, the endoscopic system 10 can prevent misidentification of results when recognition processing is performed and when it is not. As a result, misdiagnosis can be prevented.

[0062] In the first embodiment described above, the notification unit 65 notifies whether the recognition process is functioning or not by displaying "AI in operation" 122 or "AI stopped" 132 on the monitor 18 screen. However, the notification unit 65 can also notify information other than whether the recognition process is functioning or not. For example, the notification unit 65 can notify some or all of the information used by the identification unit 73 to identify the type of image. In the first embodiment, the identification unit 73 determines the type of image using the setting information of the operation mode, so as shown in Figure 8, the notification unit 65 can notify the operation mode by displaying "Normal observation mode" or the like 141 on the monitor 18 screen. The same applies when the identification unit 73 determines the type of image using information other than the operation mode.

[0063] Furthermore, in the first embodiment described above, the notification unit 65 notifies whether the recognition process is functioning or not by displaying "AI in operation" 122 or "AI stopped" 132 on the monitor 18 screen, but the manner of notification regarding whether the recognition process is functioning or not is arbitrary. For example, as shown in Figure 9, the notification unit 65 can notify that the recognition process is functioning by displaying an icon 151 indicating "AI in operation" instead of the string display 122 "AI in operation". Similarly, as shown in Figure 10, the notification unit 65 can notify that the recognition process is not functioning by displaying an icon 152 indicating "AI stopped" instead of the string display 132 "AI stopped". Also, even when notifying whether the recognition process is functioning or not using a string, notification can be given by displaying "AI: ON" or "AI: Supported" instead of "AI in operation". Similarly, notification can be given by displaying "AI: OFF" or "AI: Not supported" instead of "AI stopped". In addition, as shown in Figure 11, the notification unit 65 can notify that the recognition process is functioning by adding a double frame 161 to the "Normal Observation Mode" display 141 instead of the "AI Operating" string display 122. Also, as shown in Figure 12, the notification unit 65 can notify that the recognition process is not functioning by adding a single frame 162 to, for example, the "Special Observation Mode" display 143 instead of the "AI Stopped" string display 132.

[0064] In the first embodiment described above, the notification unit 65 notifies whether the recognition process is functioning or not by displaying it on the screen of the monitor 18. However, if the notification is to be made in a manner other than displaying it on the screen of the monitor 18, the processor device 16 may be equipped with a notification device 171, as shown in Figure 13. The notification device 171 is a speaker that emits sound or voice, an indicator composed of light-emitting elements such as LEDs, or a vibrating element such as a motor or piezoelectric element. In addition, the notification unit 65 can use elements provided by the endoscope system 10 as the notification device 171. Furthermore, the notification device 171 may be provided in a device other than the processor device 16, i.e., the endoscope 12 or the light source device 14.

[0065] [Second Embodiment] In this embodiment, the endoscope system 10 has two operating modes: a magnified observation mode in which the subject is magnified and / or observed by optical and / or electronic magnification processing, and a non-magnified observation mode in which optical and / or electronic magnification processing is not used. In this case, the identification unit 73 can identify the type of image depending on whether or not magnification processing is performed.

[0066] Specifically, as shown in Figure 14, the operating mode is set by operating the mode switching operation unit 13b (step S210), the subject is photographed (step S211), the image acquisition unit 54 acquires the endoscopic image, and the image generation unit 71 performs the necessary image processing to generate an endoscopic image for display (step S212). These steps are the same as in the first embodiment.

[0067] On the other hand, the identification unit 73 identifies the type of image by determining whether the observation mode is the magnified observation mode or the non-magnified observation mode (step S213). For example, the identification unit 73 identifies an endoscopic image obtained in the magnified observation mode as an image for which recognition processing is functional (step S213: YES). Therefore, the recognition unit 72 performs recognition processing using the endoscopic image obtained in the magnified observation mode (step S214), and the notification unit 65 notifies that the recognition processing is functional. Also, the identification unit 73 identifies an endoscopic image obtained in the non-magnified observation mode as an image for which recognition processing is not functional (step S213: NO). Therefore, the recognition unit 72 does not perform recognition processing on the endoscopic image obtained in the non-magnified observation mode, and the notification unit 65 notifies that the recognition processing is not functional (step S216). The notification method is the same as in the first embodiment and its modified form.

[0068] According to the second embodiment described above, physicians and others can correctly recognize whether or not the recognition processing is functioning, without misinterpreting the results of the recognition processing and the results of not performing the recognition processing, even if the results of the recognition processing are shown or not shown in both the magnified observation mode and the non-magnified observation mode. As a result, misdiagnosis can be prevented.

[0069] In the flowchart of Figure 14, recognition processing is performed in the magnified observation mode. This is useful when the recognition unit 72 is an AI that has learned to detect and / or differentiate lesions using magnified endoscopic images. The endoscope system 10 can also perform recognition processing in the non-magnified observation mode. This is useful when the recognition unit 72 is an AI that has learned to detect and / or differentiate lesions using endoscopic images that have not been magnified. In other words, the second embodiment described above is useful when the recognition unit 72 is an AI that has learned to detect and / or differentiate lesions using endoscopic images obtained in either the magnified observation mode or the non-magnified observation mode.

[0070] Furthermore, the second embodiment described above can be combined with the first embodiment. For example, in the case of normal observation mode and magnified observation mode, normal observation mode and non-magnified observation mode, special observation mode and magnified observation mode, or special observation mode and non-magnified observation mode, recognition processing can be performed and a notification can be given that the recognition processing is functioning, while in other cases, recognition processing can not be performed and a notification can be given that the recognition processing is not functioning.

[0071] [Third Embodiment] In this embodiment, the endoscope system 10 has two operating modes: an enhancement mode in which enhancement processing is performed, and a non-enhancement mode in which enhancement processing is not performed. In this case, the identification unit 73 can identify the type of image depending on whether or not enhancement processing is performed.

[0072] Specifically, as shown in Figure 15, the operating mode is set by operating the mode switching operation unit 13b (step S310), the subject is photographed (step S311), the image acquisition unit 54 acquires the endoscopic image, and the image generation unit 71 performs the necessary image processing to generate an endoscopic image for display (step S312). These steps are the same as in the first embodiment.

[0073] On the other hand, the identification unit 73 identifies the type of image by determining whether the observation mode is the enhancement mode or the non-enhancement mode (step S313). For example, the identification unit 73 identifies an endoscopic image obtained in non-enhancement mode as an image for which recognition processing is functional (step S313: YES). Therefore, the recognition unit 72 performs recognition processing using the endoscopic image obtained in non-enhancement mode (step S314), and the notification unit 65 notifies that the recognition processing is functional. Also, the identification unit 73 identifies an endoscopic image obtained in enhancement mode as an image for which recognition processing is not functional (step S313: NO). Therefore, the recognition unit 72 does not perform recognition processing on the endoscopic image obtained in enhancement mode, and the notification unit 65 notifies that the recognition processing is not functional (step S316). The notification method is the same as in the first embodiment and its modified examples.

[0074] According to the third embodiment described above, in both the non-emphasis mode and the emphasis mode, even if the results of the recognition processing are shown or not shown, physicians and others can correctly recognize whether or not the recognition processing is functioning without misinterpreting the results of the recognition processing being performed and the results of not performing the recognition processing. As a result, misdiagnosis can be prevented.

[0075] In the flowchart of Figure 15, recognition processing is performed in the non-enhanced mode, which is useful when the recognition unit 72 is an AI that has learned to detect and / or differentiate lesions using endoscopic images that have not undergone enhancement processing. The endoscopic system 10 can also perform recognition processing in the enhanced mode. This is useful when the recognition unit 72 is an AI that has learned to detect and / or differentiate lesions using endoscopic images in which lesions have been enhanced. In other words, the third embodiment described above is useful when the recognition unit 72 is an AI that has learned to detect and / or differentiate lesions using endoscopic images obtained in either the enhanced mode or the non-enhanced mode.

[0076] Furthermore, the third embodiment described above can be combined with the first embodiment and / or the second embodiment. For example, in the case of normal observation mode and enhancement mode, normal observation mode and non-enhancement mode, special observation mode and enhancement mode, or special observation mode and non-enhancement mode, recognition processing can be performed and a notification can be given that the recognition processing is functioning, while in other cases, recognition processing can not be performed and a notification can be given that the recognition processing is not functioning. The same applies when combined with the second embodiment, or when combined with the first and second embodiments.

[0077] [Fourth Embodiment] In this embodiment, multiple endoscopes 12 of different models can be used. In this case, the identification unit 73 can identify the type of image depending on the model of the endoscope 12.

[0078] Specifically, as shown in Figure 16, the operating mode is set by operating the mode switching operation unit 13b (step S410), the subject is photographed (step S411), the image acquisition unit 54 acquires the endoscopic image, and the image generation unit 71 performs the necessary image processing to generate an endoscopic image for display (step S412). These steps are the same as in the first embodiment.

[0079] On the other hand, the identification unit 73 identifies the type of image by determining whether the model of the endoscope 12 being used is appropriate (step S413). For example, the identification unit 73 identifies an endoscopic image obtained using a specific model of endoscope 12 as an image for which recognition processing is functional (step S413: YES). Therefore, the recognition unit 72 performs recognition processing using the endoscopic image obtained using the specific model of endoscope 12 (step S414), and the notification unit 65 notifies that the recognition processing is functional. The identification unit 73 also identifies an endoscopic image obtained using an endoscope 12 of a model other than the specific model (hereinafter referred to as a non-specific model) as an image for which recognition processing is not functional (step S413: NO). Therefore, the recognition unit 72 does not perform recognition processing on an endoscopic image obtained using a non-specific model of endoscope 12, and the notification unit 65 notifies that the recognition processing is not functional (step S416). The notification method is the same as in the first embodiment and its modified examples.

[0080] According to the fourth embodiment described above, when using an endoscope 12 of a specific model or an endoscope 12 of a non-specific model, doctors and others can correctly recognize whether the recognition process is functioning or not, without misinterpreting the results of the recognition process being performed or not performed, even if the results of the recognition process are shown or not. As a result, misdiagnosis can be prevented. Since the type of image sensor 48 (color filter characteristics, etc.) installed in the endoscope 12 differs depending on the model, the characteristics of the obtained endoscopic image differ. For example, even when photographing the same subject, the color, brightness, or resolution of the endoscopic image may differ depending on the model of the endoscope 12. The fourth embodiment described above is useful when the recognition unit 72 has learned to detect and / or differentiate lesions, etc., using endoscopic images taken with an endoscope 12 of a specific model. If there are multiple types of endoscopes 12 that can obtain images in which the recognition process functions, the above "endoscope 12 of a specific model" is a group that includes multiple types of endoscopes 12.

[0081] Furthermore, the fourth embodiment described above can be combined with the first, second, and / or third embodiments. For example, recognition processing can be performed and a notification can be given that the recognition processing is functioning when using the normal observation mode and a specific model of endoscope 12, while recognition processing can be omitted in other cases and a notification can be given that the recognition processing is not functioning. The same applies when combining with other embodiments.

[0082] [Fifth Embodiment] When the identification unit 73 identifies the type of image based on whether or not a drug has been administered to the subject, the endoscope system 10 operates as shown in Figure 17. Specifically, the operating mode is set by operating the mode switching operation unit 13b (step S510), the subject is photographed (step S511), the image acquisition unit 54 acquires the endoscopic image, and the image generation unit 71 performs the necessary image processing to generate an endoscopic image for display (step S512). These steps are the same as in the first embodiment.

[0083] On the other hand, the identification unit 73 identifies the type of image by determining whether or not a drug has been administered to the subject (step S513). For example, the identification unit 73 identifies an endoscopic image obtained when no drug has been administered to the subject as an image for which recognition processing is functional (step S513: YES). Therefore, the recognition unit 72 performs recognition processing using an endoscopic image of a subject that has not been administered a drug (step S514), and the notification unit 65 notifies that the recognition processing is functional. The identification unit 73 also identifies an endoscopic image of a subject that has been administered a drug as an image for which recognition processing is not functional (step S513: NO). Therefore, the recognition unit 72 does not perform recognition processing on an endoscopic image of a subject that has been administered a drug, and the notification unit 65 notifies that the recognition processing is not functional (step S516). The notification method is the same as in the first embodiment and its modified form.

[0084] According to the fifth embodiment described above, even if the results of the recognition process are shown or not shown in both cases—when the drug is administered to the subject and when the drug is not administered to the subject—the physician or other personnel can correctly recognize whether or not the recognition process is functioning without misinterpreting the results of the recognition process being performed and the results of not performing the recognition process.

[0085] In the fifth embodiment described above, recognition processing is performed when no drug is administered to the subject. This is useful when the recognition unit 72 is an AI that has learned to detect and / or differentiate lesions, etc., using endoscopic images of subjects in which no drug has been administered. The endoscope system 10 can also perform recognition processing when a specific drug is administered to the subject. This is useful when the recognition unit 72 has learned to detect and / or differentiate lesions, etc., using endoscopic images of subjects in which a specific drug has been administered. In this case, the identification unit 73 can determine the type of drug administered to the subject using setting information or by analyzing the endoscopic image, and then identify the type of image.

[0086] The fifth embodiment described above can be combined with the first, second, third, and fourth embodiments, or a combination of these embodiments. For example, the endoscope system 10 can perform recognition processing and notify that the recognition processing is functioning when in normal observation mode and no drug is administered, and in other cases it can not perform recognition processing and notify that the recognition processing is not functioning. The same applies to other combinations.

[0087] [Sixth Embodiment] In the first embodiment, the identification unit 73 automatically starts or stops the recognition unit 72. However, instead of the recognition unit 72 automatically starting or stopping, the endoscope system 10 allows a physician or other person to manually enable or disable the recognition process. In this case, as shown in Figure 18, the operating mode is set by operating the mode switching operation unit 13b (step S610), and when a subject is photographed (step S611), the image acquisition unit 54 acquires the endoscopic image, and the image generation unit 71 performs the necessary image processing to generate an endoscopic image for display (step S612). These steps are the same as in the first embodiment.

[0088] On the other hand, the identification unit 73 uses the setting information to determine whether the recognition process is enabled or disabled (step S613), and identifies the type of image by determining the operating mode (step S614). If the recognition process is enabled by manual setting (step S613: YES), or if it is in normal observation mode (step S614: YES), the recognition unit 72 executes the recognition process (step S615), and the notification unit 65 notifies that the recognition process is functioning (step S616). If the recognition process is disabled by manual setting (step S613: NO), or if the operating mode is in special observation mode (step S614: NO), the recognition unit 72 does not execute the recognition process, and the notification unit 65 notifies that the recognition process is not functioning (step S617). The notification method is the same as in the first embodiment and its modified form.

[0089] According to the sixth embodiment described above, the notification unit 65 appropriately notifies whether or not the recognition process is functioning, so that doctors and others can correctly recognize whether or not the recognition process is functioning without misinterpreting the results of performing the recognition process or not performing the recognition process, even if the results of the recognition process are shown or not. For example, even if a doctor or others forget that they have disabled the recognition process in the manual settings, the notification unit 65 will notify that the recognition process is not functioning, so they can correctly recognize that the absence of a result of the recognition process is a result of the recognition process not functioning.

[0090] The sixth embodiment can be implemented in any combination with the first embodiment, the second embodiment, the third embodiment, the fourth embodiment, the fifth embodiment, or any multiple embodiments thereof.

[0091] [Seventh Embodiment] In the first embodiment described above, the identification unit 73 automatically starts or stops the recognition unit 72, so the notification content by the notification unit 65 is whether or not the recognition process is functioning. However, if the identification unit 73 does not automatically start or stop the recognition unit 72, it is preferable that the notification unit 65 notifies in different ways depending on whether the recognition process is functioning or not, and whether or not the recognition unit 72 is running or stopped.

[0092] Specifically, as shown in Figure 19, if the recognition unit 72 is activated and the recognition process is "enabled," and the operating mode is such that an image in which the recognition process functions can be obtained ("recognition process OK"), the notification unit 65 notifies in a first notification manner whether or not the recognition process is functioning. The first notification manner is, for example, the display 122 of "AI in operation." On the other hand, if the recognition unit 72 is stopped (not activated) and the recognition process is "invalid," and the operating mode is such that an image in which the recognition process functions can be obtained ("recognition process OK"), the notification unit 65 notifies in a second notification manner whether or not the recognition process is functioning. The second notification manner is, for example, the display of "AI not activated," "AI can perform recognition processing," or "AI stopped due to AI not being activated."

[0093] Furthermore, if the recognition unit 72 is activated and therefore the recognition process is "enabled," but the operating mode is such that an image is obtained in which the recognition process does not function ("recognition process impossible"), the notification unit 65 will notify whether or not the recognition process is functioning in a third notification mode. The third notification mode is, for example, a display such as "recognition process error," "AI not supported," or "AI stopped due to AI not supported." If the recognition unit 72 is stopped and therefore the recognition process is "invalid," and the operating mode is such that an image is obtained in which the recognition process does not function ("recognition process impossible"), the notification unit 65 will notify whether or not the recognition process is functioning in a fourth notification mode. The fourth notification mode is, for example, a display 132 of "AI stopped."

[0094] As in the seventh embodiment described above, if the notification unit 65 notifies in different ways depending on whether the recognition unit 72 is activated or stopped, doctors and others can understand the operating status of the recognition unit 72 and the results of the recognition process in detail and accurately without misinterpreting them. Furthermore, for example, by looking at the notification content of the second notification mode, it is clear that the recognition process can be performed, so even if the recognition process has been manually disabled, it is easy to enable the recognition process and receive support from the endoscopy system 10.

[0095] In the first embodiment, the notification unit 65 provides notification in either the first notification mode or the fourth notification mode. In the seventh embodiment, the first, second, third, and fourth notification modes are all displayed differently. However, the seventh embodiment only needs to include one or more patterns in which notification is provided in different modes depending on whether the recognition unit 72 is running or stopped. It does not prevent two or more of the first, second, third, and fourth notification modes from having the same display. For example, the second display mode and / or the third display mode may be the same "AI stopped" display 132 as the fourth display mode. Even in this case, at least the first and second display modes are different, so doctors and others can understand the operating state of the recognition unit 72 and the results of the recognition process in detail and accurately without misinterpreting them.

[0096] [Eighth Embodiment] In the first embodiment, etc., the recognition unit 72 is an AI that performs recognition processing to detect lesions, etc., but as shown in Figure 20, AI(1), AI(2), ..., AI(N), the recognition unit 72 can be composed of multiple AIs that function (perform recognition processing) for different types of images. AI(1) is, for example, an AI that has learned to detect lesions, etc. using an endoscopic image taken in normal observation mode with magnification of a subject that has not been administered any drug. AI(2) is, for example, an AI that has learned to differentiate lesions, etc. using an endoscopic image taken in the same conditions as AI(1). AI(3), which is not shown, is, for example, an AI that has learned to detect lesions, etc. using an endoscopic image taken in special observation mode with magnification of a subject that has not been administered any drug. The others are similar.

[0097] As described above, when the recognition unit 72 includes multiple AIs, each performing a different recognition process, the identification unit 73 identifies the type of image to determine whether one or more of these AIs are functional. The notification unit 65 then notifies whether the recognition process is functional for each of the multiple AIs. This ensures that even when the recognition unit 72 includes multiple AIs, it can accurately notify whether the recognition process is functional for each AI. The notification unit 65 can also notify that the recognition process is functional if at least one of the multiple AIs is functional. In this case, it is possible to notify that the recognition process is functional more concisely than when notifying whether the recognition process is functional for each AI individually.

[0098] In the eighth embodiment described above, one recognition unit 72 includes multiple AIs, but as shown in Figure 21, the same applies when, instead of the recognition unit 72, there are multiple recognition units, such as a first recognition unit 801, a second recognition unit 802, ..., a Mth recognition unit 803, each functioning (performing recognition processing) for different types of images. That is, when multiple recognition units (such as the first recognition unit 801 and the second recognition unit 802) are provided instead of the recognition unit 72, the notification unit 65 notifies whether or not the recognition processing is functioning for each of these multiple recognition units. This ensures that even when multiple recognition units are provided instead of the recognition unit 72, the notification unit 65 can accurately notify whether or not the recognition processing is functioning for each recognition unit. The notification unit 65 may also notify that the recognition processing is functioning when at least one of the multiple recognition units, such as the first recognition unit 801 and the second recognition unit 802, is functioning. In this case, the notification that the recognition processing is functioning can be conveyed more concisely than when the notification is given for each recognition unit whether or not the recognition processing is functioning for each individual recognition unit.

[0099] As shown in Figure 22, the recognition unit 72, identification unit 73, and / or notification unit 65 can be provided in a medical image processing device 901 that communicates with a processor device 16 and cooperates with the endoscope system 10. Also, as shown in Figure 23, the recognition unit 72, identification unit 73, and / or notification unit 65 can be provided in a diagnostic support device 911 that acquires RAW images taken by the endoscope 12, either directly from the endoscope system 10 (including those without a notification unit 65, etc.) or indirectly from a PACS (Picture Archiving and Communication Systems) 910. Furthermore, as shown in Figure 24, the recognition unit 72, identification unit 73, and / or notification unit 65 can be provided in a medical business support device 930 that connects to various inspection devices, including the endoscope system 10, such as the first inspection device 921, the second inspection device 922, ..., and the Kth inspection device 923, via a network 926.

[0100] In other words, the present invention includes a medical image processing device and its operation method, comprising: an identification unit for identifying the type of image of a subject; a recognition unit for performing recognition processing to recognize a subject using the image; and a notification unit for notifying whether or not the recognition processing is functional for a specific type of image identified by the identification unit. The present invention also includes a diagnostic support device and its operation method, comprising: an identification unit for identifying the type of image of a subject; a recognition unit for performing recognition processing to recognize a subject using the image; and a notification unit for notifying whether or not the recognition processing is functional for a specific type of image identified by the identification unit. Similarly, the present invention includes a medical business support device and its operation method, comprising: an identification unit for identifying the type of image of a subject; a recognition unit for performing recognition processing to recognize a subject using the image; and a notification unit for notifying whether or not the recognition processing is functional for a specific type of image identified by the identification unit.

[0101] The present invention also includes a method for operating an endoscope system, comprising the steps of: an identification unit identifying the type of image of a subject; a recognition unit performing recognition processing to recognize the subject using the image; and a notification unit notifying whether or not the recognition processing is functional for a specific type of image identified by the identification unit. The present invention also includes a processor device and a method for operating the same, comprising an identification unit identifying the type of image of a subject; a recognition unit performing recognition processing to recognize the subject using the image; and a notification unit notifying whether or not the recognition processing is functional for a specific type of image identified by the identification unit.

[0102] Furthermore, a capsule endoscope can be used as the endoscope 12. In this case, the light source device 14 and part of the processor device 16 can be mounted on the capsule endoscope.

[0103] In the above embodiment, the hardware structure of the processing unit that performs various processes such as the recognition unit 72, the identification unit 73, or the notification unit 65 is a various type of processor as shown below. These various processors include a CPU (Central Processing Unit), a general-purpose processor that executes software (programs) and functions as various processing units; a GPU (Graphical Processing Unit); a Programmable Logic Device (PLD), a processor whose circuit configuration can be changed after manufacturing, such as an FPGA (Field Programmable Gate Array); and a dedicated electrical circuit, a processor with a circuit configuration specifically designed to perform various processes.

[0104] A single processing unit may be composed of one of these various processors, or it may be composed of a combination of two or more processors of the same or different types (for example, multiple FPGAs, a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). Alternatively, multiple processing units may be composed of a single processor. Examples of composing multiple processing units with a single processor include, firstly, a configuration where one or more CPUs and software are combined to form a single processor, and this processor functions as multiple processing units, as is typical of computers such as clients and servers. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple processing units, on a single IC (Integrated Circuit) chip, as is typical of a System on a Chip (SoC). Thus, various processing units are configured, in terms of hardware structure, using one or more of the above-mentioned various processors.

[0105] Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit in the form of a combination of circuit elements such as semiconductor devices.

[0106] Furthermore, the present invention can be used not only in endoscopic systems, processor devices, and other related devices for acquiring endoscopic images, but also in systems or devices for acquiring medical images other than endoscopic images (including video). For example, the present invention can be applied to ultrasound examination devices, X-ray imaging devices (including CT (Computed Tomography) examination devices and mammography devices, etc.), MRI (magnetic resonance imaging) devices, etc. [Explanation of Symbols]

[0107] 10 Endoscopy Systems 12 Endoscopes 12a Insertion section 12b Operation section 12c curved section 12d Tip 12e Angle Knob 13a Zoom control section 13b Mode switching operation section 14 Light source device 16 Processor Unit 18 monitors 19 Console 20 Light source section 22 Light source control unit 30a illumination optical system 30b Imaging optical system 45 Illumination Lens 46 Objective lens 47 Zoom Lens 48 Image Sensors 52 Control Unit 54 Image acquisition unit 56 DSP 58 Noise Reduction Section 59 Conversion section 61 Image Processing Unit 65 Hochi Department 66 Display Control Unit 71 Image generation unit 72 Recognition part 73 Identification Unit 121, 124 Normal observation images 122 Display indicating that the recognition process is functioning. 131 Special Observation Images 132 Display indicating that the recognition process is not functioning. 141 Display in normal observation mode 143 Display of Special Observation Mode 151 Icon indicating AI is running 152 Icon indicating AI is stopped 161 Double frame 162 Single frame 171 Notification devices 801 1st recognition part 802 2nd recognition section 803 M recognition section 901 Medical Image Processing Equipment 910 PACS 911 Diagnostic support device 921 First Inspection Device 922 Second Inspection Device 923 K Inspection Device 926 Network 930 Medical Business Support Device S110~S617 Operation Steps

Claims

1. Equipped with a processor, The aforementioned processor, Images of the subject taken with an endoscope are acquired. Using the operating mode setting information, the model of the endoscope used to photograph the subject, whether or not a drug has been administered to the subject, or the image magnification processing setting information, it is determined whether or not the recognition process for recognizing the subject is functioning. Information showing some or all of the information used for the aforementioned determination, and information related to the recognition process are displayed on the monitor. The recognition process is an image processing device that uses the image to detect the presence or absence of a lesion or a candidate lesion, or uses the image to differentiate the type or progression of a lesion or a candidate lesion.

2. The aforementioned processor, The image processing apparatus according to claim 1, which determines whether or not the recognition process functions based on the setting information of the operating mode.

3. The image processing apparatus according to claim 1, wherein the operating mode is a normal observation mode in which the subject is photographed with white light, or a special observation mode in which the subject is photographed using illumination light having a specific wavelength band.

4. The aforementioned processor, The image processing apparatus according to claim 3, wherein, when the operating mode is the special observation mode, information indicating that it is the special observation mode and information relating to the recognition process are displayed on the monitor.

5. The image processing apparatus according to claim 3 or 4, wherein in the special observation mode, the subject is photographed using illumination light that contains a larger amount of blue or purple components compared to the white light.

6. The aforementioned processor, The image processing apparatus according to any one of claims 1 to 5, which acquires setting information of the operating mode from a processor device that controls the endoscope.

7. The aforementioned processor, The image processing apparatus according to any one of claims 1 to 5, wherein setting information of the operating mode is obtained by analyzing the aforementioned image.

8. The aforementioned processor, An image processing apparatus according to any one of claims 1 to 7, which determines whether or not the recognition process functions based on the model of the endoscope.

9. The aforementioned processor, The image processing apparatus according to claim 8, which displays information indicating the model of the endoscope and information regarding the recognition process on the monitor.

10. The aforementioned processor, An image processing apparatus according to any one of claims 1 to 7, which determines whether or not the recognition process functions based on whether or not the drug has been administered.

11. The aforementioned processor, The image processing apparatus according to claim 10, which, when a drug is administered to the subject, displays on the monitor information indicating that the drug has been administered to the subject and information regarding the subject.

12. The aforementioned processor, An image processing apparatus according to any one of claims 1 to 7, which determines whether or not the recognition process functions based on the information from the image enlargement process.

13. The aforementioned processor, The image processing apparatus according to claim 1, which determines whether or not the recognition process functions based on the setting information of the operating mode and the information of the image enlargement process.

14. When the magnification ratio of the magnification process is a first magnification, information indicating that it is a first magnification and information regarding the recognition process are displayed on the monitor. The image processing apparatus according to claim 12 or 13, wherein, when the magnification is a second magnification greater than the first magnification, information indicating that it is the second magnification and information regarding the recognition process are displayed on the monitor.

15. The image processing apparatus according to claim 14, wherein the magnification is changed by moving the zoom lens of the endoscope.

16. The aforementioned processor, The image processing apparatus according to any one of claims 1 to 15, which acquires information on the image magnification process from a processor device that controls the endoscope.

17. The aforementioned processor, The image processing apparatus according to claim 1, which displays on the monitor information showing all of the information used for the discrimination and information related to the recognition process.

18. As information that shows a part of the information used for the aforementioned determination, The image processing apparatus according to claim 1, which displays on the monitor information indicating at least one of the following: setting information for the operating mode, the model of the endoscope, whether or not a drug has been administered to the subject, or setting information for the image magnification process.

19. The information regarding the aforementioned recognition process is, An image processing apparatus according to any one of claims 1 to 18, comprising at least one of the following: information indicating that the recognition process is functional; information indicating that the recognition process is not functional; and the result of the recognition process.

20. The aforementioned processor, The image processing apparatus according to claim 19, wherein, when it is determined that the recognition process is functional, the apparatus displays on the monitor information indicating that the recognition process is functional and the result of the recognition process as information indicating that the recognition process is functional.

21. The aforementioned processor, The image processing apparatus according to claim 20, which displays on the monitor information indicating the location of the lesion or a candidate for the lesion as a result of the recognition process.

22. The image processing apparatus according to claim 20, which, when it is determined that the recognition process is not functioning, displays on the monitor information indicating that the recognition process is not functioning.

23. The aforementioned processor, The image processing apparatus according to any one of claims 1 to 22, wherein when the information used for the discrimination is switched, information relating to the recognition process is displayed on the monitor.

24. The aforementioned recognition process is, The image processing apparatus according to any one of claims 1 to 23, which involves a process for detecting the presence or absence of a lesion or a candidate lesion using the aforementioned image.

25. The aforementioned recognition process is, The image processing apparatus according to any one of claims 1 to 23, which involves a process for differentiating the type or progression of a lesion or a candidate lesion using the aforementioned image.

26. The image processing apparatus according to any one of claims 1 to 25, wherein the recognition process is performed by artificial intelligence having a learning function.

27. The aforementioned processor, For each of the multiple recognition processes that function for different types of images, it is determined whether or not each of the recognition processes functions. The image processing apparatus according to any one of claims 1 to 26, wherein information relating to the recognition process is displayed on the monitor for each of the aforementioned recognition processes.

Images