Image processing device and method of operation thereof

Abstract

To provide an image processing device and an operating method of the device that can prevent a section with specific biological information from being missed.SOLUTION: An image acquisition unit 54 acquires an endoscopic image obtained by imaging a subject with an endoscope 12. A biological information calculation unit 72 calculates biological information based on the endoscopic image. A region recognition unit 76 recognizes a region of the subject. A reference value setting unit 73 sets a reference value of the biological information based on the type of the region. A biological information image generation unit 74 generates a biological information image obtained by imaging the biological information in such a manner that a low value area in which the biological information is equal to or less than a reference value and an area in which the biological information is equal to or more than the reference value can be distinguished from each other.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an image processing device that processes medical images such as endoscopic images, and an operating method thereof. [Background technology]

[0002] In the medical field, examinations using endoscope systems are becoming widespread. An endoscope system includes, for example, an endoscope (so-called a scope) to be inserted into a subject, a light source device that generates illumination light to illuminate the subject, a processor device that obtains an endoscopic image of the subject using the endoscope, and a display that displays the endoscopic image, etc.

[0003] In recent years, an endoscope system has been known that calculates and displays biological information of a subject using an endoscope image. In this case, it is known that the biological information is calculated and displayed as a relative value. For example, an electronic endoscope system is known that calculates a reference value of blood vessel information based on image data of a set reference value area, and displays the blood vessel information of the endoscope image in pseudo-color as a relative value using the reference value (Patent Document 1). Also, an endoscope system is known that calculates the reliability of biological information based on an image, sets a reference value that is a reference for the biological information using the reliability, and generates a difference image using the difference value between the measured value and the reference value (Patent Document 2). Also, an endoscope system is known that displays an area with particularly low oxygen saturation in a color different from normal by image processing of the endoscope image (Patent Document 3). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2011-194028 A [Patent Document 2] International Publication No. 2019 / 172231 [Patent Document 3] JP 2012-139482 A Summary of the Invention [Problem to be solved by the invention]

[0005] When displaying biological information such as oxygen saturation, a reference value for the biological information is set, and the biological information is calculated and displayed as a relative value, making it easy to grasp the specific parts of the biological information. In this way, there are advantages to using a reference value for biological information, but on the other hand, when observing multiple parts or in a situation where the biological information changes over time even in the same part, there may be cases where using the same reference value repeatedly causes inconvenience.

[0006] By continuously using the same reference value, for example, in an oxygen saturation test in which oxygen saturation is imaged and displayed, problems may occur when identifying a region with a specific oxygen saturation. In order to make it easier to recognize a hypoxic region with a particularly low oxygen saturation, the reference value of oxygen saturation is set particularly low, and a hypoxic region below the reference value is displayed by an image. Then, if an oxygen saturation test is performed on another site or the same site after a certain time has passed without changing the reference value, the hypoxic region may no longer exist in the subject. In this case, the hypoxic region is not displayed. However, even if there is no display of a hypoxic region, there may actually be a subhypoxic region that is slightly higher than the reference value but is generally considered to be a hypoxic region. Both the hypoxic region and the subhypoxic region are regions with a specific oxygen saturation, and may be a lesion or a suture failure of the intestine or the like. It may be problematic if a doctor or the like does not notice the existence of a region with a specific oxygen saturation.

[0007] An object of the present invention is to provide an image processing apparatus and an operating method thereof that can prevent portions in which biometric information is unique from being overlooked. [Means for solving the problem]

[0008] The image processing device of the present invention is equipped with an image processor which acquires an endoscopic image obtained by imaging a subject with an endoscope, calculates biometric information based on the endoscopic image, recognizes a part of the subject, sets a reference value for the biometric information based on the type of part, and generates a biometric information image in which the biometric information is visualized so that low-value areas where the biometric information is below the reference value can be distinguished from areas where the biometric information is above the reference value.

[0009] The image processor preferably generates a display image based on the biometric image. The image processor preferably recognizes a part of the subject from the endoscopic image using a trained model based on machine learning.

[0010] If the image processor determines that the reference value is inappropriate when a quasi-low value area exists in the biometric information image where the biometric information is higher than the reference value and below a preset set value, the image processor preferably determines that the reference value is inappropriate, and if it determines that the reference value is inappropriate, it preferably issues a notification regarding the reference value.

[0011] It is preferable that the image processor generates color correspondence information for reference values ​​in which areas where the biometric information is equal to or less than a reference value correspond to specific colors, and generates a biometric information image based on the color correspondence information for reference values.

[0012] When the reference value is set, the image processor preferably generates reference value color correspondence information based on the set reference value. The image processor preferably generates a display image using a reference value color bar obtained by visualizing the reference value color correspondence information and a reference value indicator indicating the reference value.

[0013] When the reference value is set, the image processor preferably generates a display image using a reference value indicator indicating the set reference value. The image processor preferably calculates an average value of the biological information in the endoscopic image, and generates a display image using a reference value color bar and an average value indicator indicating the average value.

[0014] It is preferable that the image processor is connected to a display that displays the display image and a display image storage unit that stores the display image, and controls the display image stored in the display image storage unit to be displayed as a thumbnail on the display. When a reference value is set, it is preferable that the image processor controls the display image to be displayed as a thumbnail on the display after changing it based on the reference value.

[0015] The image processor preferably determines that the reference value is inappropriate when a quasi-low value region exists in the biometric information image at a rate equal to or greater than a preset rate.The image processor preferably determines that the reference value is inappropriate when a quasi-low value region exists continuously for a period equal to or greater than a preset time.

[0016] When the image processor does not determine that the reference value is appropriate, it is preferable that the image processor notify that the reference value is appropriate.When the image processor determines that the reference value is appropriate, it is preferable that the image processor switch the reference value to a higher value and set it.

[0017] When the reference value and the set value satisfy a preset condition, the image processor preferably changes the reference value to a preset value that is higher than the reference value before the change and does not exceed the set value. The biological information is preferably the oxygen saturation of the subject.

[0018] The present invention is a method for operating an image processing device equipped with an image processor, in which the image processor acquires an endoscopic image obtained by imaging a subject with an endoscope, calculates biometric information based on the endoscopic image, recognizes a part of the subject, sets a reference value for the biometric information based on the type of part, and generates a biometric information image in which the biometric information is visualized so that low-value areas where the biometric information is below the reference value can be distinguished from areas where the biometric information is above the reference value. Effect of the Invention

[0019] According to the present invention, it is possible to prevent a unique portion of biological information from being overlooked. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 is an external view of an endoscope system. [Diagram 2] FIG. 2 is an external view of an operation section of the endoscope. [Diagram 3] FIG. 1 is a block diagram of an endoscope system. [Figure 4] FIG. 2 is a block diagram of an image processing unit. [Diagram 5] 11 is an explanatory diagram illustrating a reference value set by a reference value setting unit based on a part. FIG. [Figure 6] FIG. 11 is an explanatory diagram for explaining generation of a display image. [Figure 7] This is a color bar for reference values. [Figure 8] FIG. 13 is an image diagram of a display image including a reference value color bar. [Figure 9] 13A and 13B are explanatory diagrams for explaining changes in the reference value and a display image. [Figure 10] This is a color bar for reference values ​​combined with average value indicators. [Figure 11] FIG. 13 is an image diagram of a display image including a reference value color bar combined with an average value indicator. [Figure 12] FIG. 13 is an explanatory diagram of a reference value setting menu. [Figure 13] 10 is an explanatory diagram illustrating a reference value set by a reference value setting unit based on a reference value and a setting value. FIG. [Figure 14] 11 is an explanatory diagram illustrating an example of a determination made by an appropriateness determining unit; FIG. [Figure 15] 13 is an explanatory diagram illustrating another example of the determination performed by the appropriateness determining unit. FIG. [Figure 16] FIG. 11 is an explanatory diagram illustrating yet another example of the determination performed by the appropriateness determining unit. [Figure 17] 13 is an explanatory diagram explaining that, when the appropriateness determining unit determines that the reference value is not appropriate, the reference value setting unit switches and sets the reference value. FIG. [Figure 18] FIG. [Figure 19]FIG. 13 is an explanatory diagram illustrating notification using a reference value color bar combined with average value indicators. [Figure 20] FIG. 13 is an explanatory diagram for explaining notification using a reference value color bar combined with arrows. [Figure 21] FIG. 2 is a block diagram of an image processing unit connected to a display image storage unit. [Figure 22] FIG. 11 is an explanatory diagram for explaining thumbnail display of a display image. [Diagram 23] 1 is a flowchart illustrating a flow of a series of processes performed by an image processing device. [Figure 24] FIG. 2 is a block diagram of an image processing device separate from the processor device. [Diagram 25] FIG. 1 is an explanatory diagram showing a diagnosis support device. [Figure 26] FIG. 1 is an explanatory diagram showing a medical service support device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] As shown in FIG. 1, an endoscope system 10 (endoscopic device) includes an endoscope 12 (scope), a light source device 14, a processor device 16, a display 18, and a keyboard 19. The endoscope 12 captures an image of a subject. The light source device 14 generates illumination light. The processor device 16 performs system control of the endoscope system 10. The processor device 16 also generates an endoscopic image and performs image processing on the endoscopic image as necessary. In other words, the processor device 16 functions as an image processing device. The display 18 is a display unit that displays an endoscopic image, etc. The keyboard 19 is an input device that inputs settings to the processor device 16, etc.

[0022] The endoscope 12 has an insertion section 12a to be inserted into the subject, an operation section 12b provided at the base end of the insertion section 12a, a bending section 12c provided at the tip side of the insertion section 12a, and a tip section 12d. The bending section 12c is bent by operating an angle knob 12e (see FIG. 2) of the operation section 12b. As a result, the tip section 12d faces a desired direction. As shown in FIG. 2, the operation section 12b is provided with a treatment tool insertion port 12f, a scope button 12g, and a zoom operation section 13 in addition to the angle knob 12e. The treatment tool insertion port 12f is an entrance through which a treatment tool such as a biopsy forceps, a snare, or an electric scalpel is inserted. The treatment tool inserted into the treatment tool insertion port 12f protrudes from the tip section 12d. Various operations can be assigned to the scope button 12g, and in this embodiment, it is used for an operation to set a reference value. The subject can be enlarged or reduced for photographing by operating the zoom operation section 13.

[0023] As shown in FIG. 3, the light source device 14 includes a light source section 20 that emits illumination light, and a light source control section 22 that controls the operation of the light source section 20.

[0024] The light source unit 20 emits illumination light for illuminating the subject. The illumination light includes excitation light used to emit the illumination light. The light source unit 20 includes a light source such as a laser diode, an LED (Light Emitting Diode), a xenon lamp, or a halogen lamp, and emits at least white illumination light or excitation light used to emit white illumination light. The white light includes so-called pseudo-white light that is substantially equivalent to white light in photographing the subject using the endoscope 12. The light source unit 20 includes, as necessary, a phosphor that emits light when irradiated with excitation light, or an optical filter that adjusts the wavelength band, spectral spectrum, or light amount of the illumination light or excitation light. In addition, the light source unit 20 can emit light having a specific wavelength band required for photographing an image used to calculate biological information such as oxygen saturation of hemoglobin contained in the subject.

[0025] In this embodiment, the light source unit 20 has four color LEDs, namely, V-LED 20a, B-LED 20b, G-LED 20c, and R-LED 20d. The V-LED 20a emits purple light VL having a central wavelength of 405 nm and a wavelength band of 380 to 420 nm. The B-LED 20b emits blue light BL having a central wavelength of 460 nm and a wavelength band of 420 to 500 nm. The G-LED 20c emits green light GL having a wavelength band of 480 to 600 nm. The R-LED 20d emits red light RL having a central wavelength of 620 to 630 nm and a wavelength band of 600 to 650 nm. The central wavelengths of the V-LED 20a and the B-LED 20b have a width of about ±20 nm, preferably about ±5 nm to about ±10 nm.

[0026] The light source control unit 22 controls the timing of turning on, off or blocking each light source constituting the light source unit 20, the amount of light emitted, etc. As a result, the light source unit 20 can emit a plurality of types of illumination light with different optical spectra. In this embodiment, the light source control unit 22 controls the turning on and off of each LED 20a to 20d, the amount of light emitted when turned on, the insertion and removal of optical filters, etc., independently for each of the LEDs. The light spectrum of the illumination light is adjusted by inputting a control signal. This causes the light source unit 20 to emit white light. The light source unit 20 can also emit illumination light consisting of at least narrowband light (hereinafter referred to as narrowband light). "Narrowband" refers to a substantially single wavelength band in relation to the characteristics of the subject and / or the spectral characteristics of the color filter of the image sensor 48. For example, when the wavelength band is, for example, about ±20 nm or less (preferably about ±10 nm or less), this light is narrowband. Moreover, broadband refers to a relatively wide wavelength band compared to narrowband light in relation to the characteristics of the subject and / or the spectral characteristics of the color filter of the image sensor 48. Therefore, when the wavelength band is, for example, ±20 nm or more, the light is broadband light.

[0027] An illumination optical system 30a and an imaging optical system 30b are provided at the tip 12d of the endoscope 12. The illumination optical system 30a has an illumination lens 45, through which illumination light is emitted toward a subject.

[0028] The photographing optical system 30b has an objective lens 46, a zoom lens 47, and an image sensor 48. The image sensor 48 photographs the subject using reflected light of the illumination light returning from the subject (including reflected light, scattered light, fluorescence emitted by the subject, or fluorescence caused by a drug administered to the subject, etc.) via the objective lens 46 and the zoom lens 47. The zoom lens 47 moves by operating the zoom operation unit 13, and enlarges or reduces the image of the subject.

[0029] The image sensor 48 has a color filter of one color among a plurality of color filters for each pixel. In this embodiment, the image sensor 48 is a color sensor having primary color filters. Specifically, the image sensor 48 has an R pixel having a red color filter (R filter), a G pixel having a green color filter (G filter), and a B pixel having a blue color filter (B filter).

[0030] As the image sensor 48, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor can be used. Although the image sensor 48 of this embodiment is a primary color sensor, a complementary color sensor can also be used. The complementary color sensor has, for example, a cyan pixel provided with a cyan color filter, a magenta pixel provided with a magenta color filter, a yellow pixel provided with a yellow color filter, and a green pixel provided with a green color filter. When a complementary color sensor is used, an image obtained from the pixels of each color can be converted into an image similar to an image obtained by a primary color sensor by performing complementary color-primary color conversion. The same applies to a primary color or complementary color sensor having one or more types of pixels having characteristics other than those described above, such as a W pixel (a white pixel that receives light in almost the entire wavelength band). Although the image sensor 48 of this embodiment is a color sensor, a monochrome sensor without a color filter may also be used.

[0031] In the processor device 16, programs related to processes performed by the control unit 52, image acquisition unit 54, image processing unit 61, display control unit 66, notification control unit 67, etc., which will be described later, are incorporated in a memory (not shown). The functions of the control unit 52, image acquisition unit 54, image processing unit 61, display control unit 66, and notification control unit 67 are realized by the program being operated by the control unit 52 constituted by a processor (image processor) provided in the processor device 16, which is an image processing device.

[0032] The control unit 52 performs overall control of the endoscope system 10, such as synchronous control of the illumination light irradiation timing and the image capturing timing. When various settings are input using the keyboard 19 or the like, the control unit 52 inputs the settings to each unit of the endoscope system 10, such as the light source control unit 22, the image sensor 48, or the image processing unit 61.

[0033] The image acquisition unit 54 acquires an endoscopic image obtained by capturing an image of a subject using an endoscope. More specifically, the image acquisition unit 54 acquires an image of the subject captured using pixels of each color, i.e., a RAW image, from the image sensor 48. The RAW image is an image (endoscopic image) before demosaic processing is performed. If the image is before demosaic processing, the RAW image also includes an image obtained by performing any processing, such as noise reduction processing, on the image acquired from the image sensor 48.

[0034] The image acquisition unit 54 includes a DSP (Digital Signal Processor) 56, a noise reduction unit 58, and a conversion unit 59 in order to perform various processes on the acquired RAW image as necessary to generate an endoscopic image.

[0035] The DSP 56 includes, for example, an offset processing unit, a defect correction processing unit, a demosaic processing unit, a linear matrix processing unit, and a YC conversion processing unit (none of which are shown in the figure), which the DSP 56 uses to perform various processes on a RAW image or an image generated using the RAW image.

[0036] The offset processing unit performs offset processing on the RAW image. The offset processing is a process for reducing dark current components from the RAW image and setting an accurate zero level. The offset processing is sometimes called clamp processing. The defect correction processing unit performs defect correction processing on the RAW image. The defect correction processing is a process for correcting or generating pixel values ​​of RAW pixels corresponding to defective pixels of the image sensor 48 when the image sensor 48 includes pixels (defective pixels) having defects due to the manufacturing process or changes over time. The demosaic processing unit performs demosaic processing on RAW images of each color corresponding to a color filter of each color. The demosaic processing is a process for generating pixel values ​​missing in the RAW image due to the arrangement of the color filters by interpolation. The linear matrix processing unit performs linear matrix processing on an endoscopic image generated by assigning one or more RAW images to channels of each RGB color. The linear matrix processing is a process for improving the color reproducibility of the endoscopic image. The YC conversion process performed by the YC conversion processing unit converts an endoscopic image generated by assigning one or more RAW images to each RGB color channel into an endoscopic image having a luminance channel Y and color difference channels Cb and chrominance channels Cr.

[0037] The noise reduction unit 58 performs noise reduction processing on the endoscopic image having the luminance channel Y, the color difference channel Cb, and the color difference channel Cr, for example, using a moving average method or a median filter method, etc. The conversion unit 59 reconverts the luminance channel Y, the color difference channel Cb, and the color difference channel Cr after the noise reduction processing into an endoscopic image having BGR color channels again.

[0038] The image processing unit 61 performs necessary image processing on the endoscopic image output by the image acquiring unit 54. In addition, the image processing unit 61 performs calculations using the endoscopic image output by the image acquiring unit 54. Specifically, as shown in Fig. 4, the image processing unit 61 includes a display image generating unit 71, a biometric information calculating unit 72, a reference value setting unit 73, a biometric information image generating unit 74, an appropriateness determining unit 75, and a part recognizing unit 76.

[0039] The display image generating unit 71 acquires an endoscopic image from the image acquiring unit 54, and generates an endoscopic image (hereinafter referred to as a display image) to be used for display on the display 18, etc. For example, the display image generating unit 71 acquires from the image acquiring unit 54 a B image obtained by photographing a subject using B pixels, a G image obtained by photographing a subject using G pixels, and an R image obtained by photographing a subject using R pixels, and generates a display image using all or a part of these. Furthermore, the generation of a display image performed by the display image generating unit 71 includes, in addition to obtaining an endoscopic image different from these using a plurality of endoscopic images as described above, performing necessary image processing on one endoscopic image acquired from the image acquiring unit 54 and outputting it, and outputting one endoscopic image acquired from the image acquiring unit 54 in its original form.

[0040] Furthermore, when generating a display image, the display image generating unit 71 performs necessary image processing on the endoscopic image acquired from the image acquiring unit 54. The image processing performed by the display image generating unit 71 is, for example, an enhancement process that enhances the subject or a part of the subject. Enhancement means distinguishing a specific part from other tissues or structures, etc., so that information on the specific part can be obtained. For example, enhancement processes include processes that surround a part having a specific characteristic with a frame, show the outline, or change the color or brightness relative to other parts (for example, normal mucosa, etc.).

[0041] Furthermore, the display image generating unit 71 acquires a biometric information image from the biometric information image generating unit 74, and generates a display image by superimposing the biometric information image on an endoscopic image. Therefore, the image processing performed by the display image generating unit 71 is, for example, a superimposition process for generating a display image by superimposing the biometric information image generated by the biometric information image generating unit 74 on an endoscopic image. Furthermore, the display image generating unit 71 generates a display image using a color bar, a reference value index, an average value index, or the like. The color bar is an image of color correspondence information used when generating a biometric information image. Furthermore, the reference value index indicates a reference value in the color bar. Furthermore, the average value index indicates an average value of the biometric information in the endoscopic image in the color bar. The generation of the biometric information image, etc. will be described later.

[0042] Note that "superimposing" a biometric information image on an endoscopic image means superimposing and displaying the biometric information image on the display image, etc. in a manner that allows the biometric information image to be recognized as distinct from the subject appearing in the display image, etc., at least when the display image, etc. is displayed on the display 18, etc., and includes superimposing and displaying a biometric information image separably on the display image, etc., as well as synthesizing (integrating) the biometric information image with the display image, etc. In this embodiment, the display image generating unit 71 superimposes the display image, etc. separably.

[0043] The biometric information calculation unit 72 calculates biometric information of the subject based on the endoscopic image acquired from the image acquisition unit 54. The biometric information is a numerical value or the like that indicates the overall or partial characteristics of the subject, such as oxygen saturation, blood concentration, blood vessel density, or the likelihood of having a specific morphology such as a lesion or a candidate for a lesion (including the target of a biopsy tissue test).

[0044] In this embodiment, the bioinformation calculation unit 72 calculates the oxygen saturation of the subject for each pixel using an endoscopic image acquired from the image acquisition unit 54. The oxygen saturation can be calculated using, for example, a B image of the subject photographed using narrowband light with a wavelength of about 470 nm, a G image of the subject photographed using broadband green light, and an R image of the subject photographed using broadband red light. More specifically, the ratio of the B image to the G image (hereinafter referred to as B / G) and the ratio of the R image to the G image (hereinafter referred to as R / G) are calculated for each pixel. In addition, a table or the like that associates B / G and R / G with the oxygen saturation value through experiments, simulations, or the like is prepared in advance. Then, the oxygen saturation is calculated from the calculated B / G and R / G values ​​using the table. When the bioinformation calculation unit 72 calculates the oxygen saturation, the image acquisition unit 54 acquires each of these images and provides them to the bioinformation calculation unit 72.

[0045] The reference value setting unit 73 sets a reference value for the biometric information. The reference value is a threshold value that distinguishes between an area having biometric information below this value and the other areas. By setting the reference value, color correspondence information for the reference value is generated. A biometric information image is generated by visualizing the biometric information by using the biometric information calculated by the biometric information calculation unit 72 and the color correspondence information for the reference value.

[0046] The means for setting the reference value is not limited as long as the value of the reference value can be specified. The reference value may be specified arbitrarily by the user, or may be automatically specified according to conditions, etc. In this embodiment, the reference value is the reference value of oxygen saturation, so specifically, the reference value is specified to be an arbitrary value within the range of 0% to 100%. For example, the reference value is specified arbitrarily in 1% increments within this range.

[0047] Specifically, the user can arbitrarily specify the reference value by inputting numeric keys using the keyboard 19 or the like. Similarly, the numeric value may be increased or decreased by using the arrow keys on the keyboard 19. The numeric value may also be increased or decreased by operating the scope button 12g or foot switch (not shown) of the endoscope. In this case, the numeric value may change by 1% each time the scope button 12g or the like is pressed once, and may change continuously by pressing and holding the button for a long time.

[0048] It is also possible to provide a part recognition unit 76 (see FIG. 4) that recognizes the part of the subject based on the endoscopic image. The reference value setting unit 73 may set the reference value based on the part recognized by the part recognition unit 76. For example, when the biological information is oxygen saturation, the average value of oxygen saturation may differ depending on the part. In that case, it is not necessary to reset the reference value manually, and it is preferable because the reference value can be easily changed and set.

[0049] The part recognition unit 76 recognizes a part based on an endoscopic image. For example, in the case of imaging of the upper gastrointestinal tract, the part to be recognized is the type of organ, such as the esophagus, stomach, or duodenum. In the same organ, for example, in the stomach, the type of part of the part is recognized, such as the cardia, fundus, body, greater bay, vestibule, pylorus, duodenal, or lesser bay. The recognition method may be an image processing method, or may be an image recognition method using a trained model by machine learning.

[0050] As shown in Fig. 5, for example, in an endoscopic examination, when the part recognition unit 76 recognizes that the part is the stomach based on the endoscopic image, the reference value setting unit 73 sets the reference value to B% in accordance with the correspondence information in which the part corresponds to the reference value. The reference value setting unit 73 is previously provided with this correspondence information. Then, as the examination progresses, when the part recognition unit 76 recognizes that the part is the duodenum based on the endoscopic image, the reference value setting unit 73 sets the reference value to C% in accordance with the reference value set for each part. Setting the reference value based on the type of part recognized by the part recognition unit 76 is preferable because it is possible to easily set an appropriate reference value even when the level of biological information differs depending on the part.

[0051] The biometric information image generating unit 74 generates a biometric information image in which an area where the biometric information is equal to or less than a reference value (hereinafter referred to as a low-value area) is distinguishable from an area where the biometric information is higher than the reference value. Any method can be used to image the biometric information in a manner that allows distinction between the low-value area and the other areas, as long as the low-value area can be imaged in a manner that allows distinction between the low-value area and the other areas. For example, a method of displaying different colors, patterns, or shapes between the low-value area and the other areas based on the biometric information calculated for each pixel of the endoscopic image, or a method of displaying colors, etc. only in the low-value area and not in the other areas, etc., can be used.

[0052] As a method for generating a biometric information image in which biometric information is visualized so that low-value regions and regions higher than the reference value are distinguishable, it is preferable that the biometric information image generating unit 74 generates color correspondence information for reference values ​​in which biometric information below a reference value corresponds to a specific color, and generates a biometric information image based on the color correspondence information for reference values. The color correspondence information for reference values ​​is color correspondence information in which a specific color, for example, blue, corresponds to biometric information below a reference value. As a result, pixel parts where biometric information is below the reference value are displayed blue based on the color correspondence information for reference values, and regions higher than the reference value are not displayed in color. Therefore, in the generated biometric information image, low-value regions are displayed blue, and regions higher than the reference value are colorless, making it possible to distinguish between low-value regions and the rest.

[0053] The specific color includes colorless or transparent. In this embodiment, the biometric image generating unit 74 generates a biometric image in which a color corresponding to the measured oxygen saturation level is applied to each pixel based on the measured oxygen saturation level calculated for each pixel in the endoscopic image and the reference value color correspondence information. Therefore, a biometric image in which the absolute value such as the measured oxygen saturation level corresponds to the color is generated.

[0054] Furthermore, the reference value color correspondence information may correspond biological information equal to or less than a reference value with a specific color, and may also correspond biological information equal to or greater than a preset value with a specific color different from that for biological information equal to or less than a reference value. In this embodiment, the biological information is oxygen saturation, so that a sufficiently high oxygen saturation level, for example, an oxygen saturation level of 95% or more, is associated with red. In this case, the reference value color correspondence information is information in which a low oxygen saturation region is colored blue, a region higher than the reference value and lower than 95% is colorless, and a region equal to or greater than 95% (hereinafter referred to as a high value region) is colored red.

[0055] The generated biometric information image is sent to a display image generating unit 71. The display image generating unit 71 performs a superimposition process to generate a display image by superimposing the biometric information image on the endoscopic image. The superimposition process may include a process required for superimposing the endoscopic image and the biometric information image. For example, the superimposition process may include a process such as alignment between the endoscopic image and the biometric information image and a correction of the light intensity ratio between the images.

[0056] As shown in FIG. 6, the display image generating unit 71 generates a display image 81 by superimposing an endoscope image 82 and an oxygen saturation image (biological information image) 83 in this embodiment. In the oxygen saturation image 83, a low oxygen saturation region 84 is imaged with a specific color so that the low oxygen saturation region 84 can be distinguished from a region with oxygen saturation higher than a reference value. Therefore, the oxygen saturation image 83 shows the low oxygen saturation region 84 colored, for example, blue. An intermediate region 86 where the oxygen saturation is higher than the reference value and lower than 95% is colorless, and the high oxygen saturation region 85 is colored, for example, red. In the display image 81 superimposed on the endoscope image 82, the low oxygen saturation region 84 is displayed in blue, the high oxygen saturation region 85 in red, and the intermediate region 86 in the endoscope image 82 as it is, for example, in the natural color of the subject. 6, low value regions 84 are shown with diagonal lines, high value regions 85 are shown with hatching, and intermediate regions 86 are shown without any markings. A display image 81, which is an image in which an endoscopic image 82 and an oxygen saturation image (biometric information image) 83 are superimposed, is displayed on the display 18.

[0057] It is preferable that the bioinformation image generating unit 74 generates a color bar for a reference value that visualizes the color correspondence information for the reference value. As shown in FIG. 7, for example, a color bar for a reference value 87 is an image of the color correspondence information for the measurement value. The color bar for a reference value 87 is a color bar for the reference value of oxygen saturation, so that a scale 91 indicates an oxygen saturation of 100%, a scale 93 indicates an oxygen saturation of 40% which is the reference value, and a scale 94 indicates an oxygen saturation of 0%. Moreover, a scale 92 indicates a high value region 85 where an oxygen saturation is 95%. The scale 93 indicating the reference value is provided with a reference value indicator 95, for example, in the shape of a filled triangular arrow, and the scale 92 indicating the high value region 85 is provided with a high value indicator 96, for example, in the shape of a hollow triangular arrow. The display image 81 including the color bar for reference value 87 allows the user to grasp at a glance the level of oxygen saturation that is colored based on the bioinformation image 83.

[0058] The display image generating unit 71 may generate the display image using the reference value color bar 87. The display image generating unit 71 preferably generates the display image 81 using the reference value color bar 87 and the reference value indicator 95. The reference value indicator 95 may be any one that allows the user to recognize the reference value. Therefore, the reference value indicator 95 is, for example, in the shape of an arrow or a line. It is preferable to display a combination of the reference value color bar 87 and the reference value indicator 95 near an image in which the endoscopic image 82 and the biological information image 83 are superimposed (hereinafter, referred to as a superimposed image) at a location where there is no problem in viewing the superimposed image. By displaying them close to each other, it is easy to recognize the color in the biological information image and the reference value in the superimposed image.

[0059] 8, for example, a reference value color bar 87 with a reference value indicator 95 is placed at the bottom left of a superimposed image 97 on the display screen to generate a display image 81. By displaying the reference value color bar 87 with the reference value indicator 95, the user can recognize at a glance what the current reference value is.

[0060] When the reference value setting unit 73 sets a reference value, it is preferable that the biometric information image generating unit 74 generates color correspondence information for reference values ​​based on the set reference value. That is, the reference value color correspondence information may be generated every time the reference value setting unit 73 sets a reference value. The reference value setting unit 73 changes the reference value in response to an instruction from a user, etc. When the reference value is changed, the biometric information image generating unit 74 automatically changes the color correspondence information for reference values ​​and generates a biometric information image based on the changed color correspondence information for reference values.

[0061] Furthermore, when the reference value setting unit 73 sets a reference value, it is preferable that the display image generating unit 71 generates a display image using a reference value indicator 95 indicating the set reference value. The reference value setting unit 73 changes the reference value according to an instruction from a user, etc. The biometric information image generating unit 74 generates reference value color correspondence information based on the set reference value, and generates a reference value color bar 87 based on the reference value color correspondence information. The display image generating unit 71 generates a display image 81 including the generated reference value color bar 87.

[0062] As shown in Fig. 9(A), for example, the reference value of oxygen saturation before changing the reference value is 30%, and the reference value setting unit 73 resets the reference value to a higher value of oxygen saturation, which is biological information. As shown in Fig. 9(B), by resetting the reference value, the biological information image is changed, and if there is an area below the reference value, it is displayed. In Fig. 9(B), since the reference value is set to 40%, an area in the range of oxygen saturation higher than 30% and lower than 40%, which was not displayed before changing the reference value, is displayed. In addition, each time the reference value is set, the reference value color bar 87 and the reference value indicator 95 constituting the display image 81 are changed.

[0063] The reference value setting unit 73 generates color correspondence information for reference values ​​every time it sets a reference value, and generates a biometric information image 83 in which a low value area 84 and an area higher than the reference value are distinguishable from each other, and a color bar 87 in which the color correspondence information for reference values ​​is visualized every time it sets a reference value. The biometric information image 83 and the color bar 87 for reference values ​​are generated, and a display image 81 using the biometric information image 83 and the like is also generated. When the reference value setting unit 73 sets a reference value, the display image generating unit 71 generates the display image 81 using the generated color bar 87 for reference values ​​and a reference value indicator 95 indicating the set reference value. Therefore, when the reference value is changed by the user or automatically, it is possible to automatically and immediately change the display image 81 to one based on the changed reference value. In addition, in the changed display image 81, the color bar 87 for reference values ​​with the reference value indicator 95 attached thereto can be changed and displayed. Each time a reference value is set, the reference value color bar 87 and the reference value indicator 95 that constitute the display image 81 are changed, so that the user can immediately recognize that the reference value has been changed and what the reference value will be after the change.

[0064] It is preferable that the biometric information calculation unit 72 calculates the average value of the biometric information in the endoscopic image 82, and the display image generation unit 71 generates the display image 81 using an average value indicator indicating the average value. The average value of the biometric information in the endoscopic image 82 may be, for example, an average value obtained by adding the biometric information calculated based on the endoscopic image 82, or an average value based on the median. Also, for example, it may be an average value obtained by removing a portion in the endoscopic image 82 in which the contrast is extremely different. The average value indicator may be any value that allows the user to recognize the average value. Therefore, the average value indicator may be, for example, an arrow or line shape, similar to the reference value indicator 95. However, when both the reference value indicator 95 and the average value indicator are displayed, for example, by making the shapes or colors different so that they can be distinguished from each other. It is preferable that the combination of the reference value color bar 87 and the average value indicator is displayed near the superimposed image 97 and at a location where the superimposed image 97 is visible without any problems. By displaying them close to each other, the color in the biometric information image 83 and the average value are easily recognized in the superimposed image 97.

[0065] Specifically, as shown in FIG. 10, for example, a reference color bar 87 and an average value indicator 98 are combined. As shown in FIG. 11, the display image generating unit 71 generates a display image 81 by arranging the reference color bar 87 with an average value indicator 98 at the lower left of the superimposed image 97. By displaying the average value indicator 98 on the display image 81, the user can immediately recognize what the current average value is. In addition, by combining it with the reference value indicator 95, the difference between the average value of the biological information in the endoscopic image 82 and the reference value can be recognized at a glance. For example, when the reference value and the average value are close to each other, it is not the low value area 84, but a low value area (hereinafter referred to as a quasi-low value area) where the biological information is higher than the reference value and requires attention is present, and the area is expected to be somewhat large. On the other hand, when the reference value and the average value are far from each other, it is expected that the quasi-low value area does not exist, or if it does exist, it is small in area.

[0066] The suitability determination unit 75 determines whether the reference value of the biometric information is appropriate. When the reference value is appropriate, the low value area 84 is emphasized by color or the like in the display image 81, so that the area to be noted is less likely to be overlooked. This is because the display image 81 includes a biometric information image 83 in which biometric information below the reference value and other biometric information higher than the reference value are visualized so that the low value area 84 can be distinguished from the rest.

[0067] An example of a case where the reference value is inappropriate is when the reference value is too low. Because the reference value is too low, there is no low value area 84 in the display image 81 including the biometric information image 83, and it appears that there is no area in the biometric information image 83 that has noteworthy biometric information values. However, in some cases, a quasi-low value area may be present. A quasi-low value area is an area where the biometric information is higher than the reference value but is low enough to indicate a possible pathology. Therefore, if the reference value is inappropriate, there is a possibility that the quasi-low value area will be overlooked. The quasi-low value area is used as a criterion for determining whether the reference value is appropriate or not.

[0068] The quasi-low value region is determined by a preset setting value and a reference value. The suitability determination unit 75 receives information on the biological information in the endoscopic image 82 from the biological information calculation unit 72, sets the setting value, and receives information on the reference value from the reference value setting unit 73, so that it can determine whether or not a quasi-low value region exists.

[0069] The set value is set by the suitability determination unit 75 and is set according to the part or purpose of observation, etc. The set value may be set higher than the reference value and according to the value of the reference value, the type of subject of the biometric information image 83, the part or purpose of observation, etc. For example, when the reference value is set low, the difference between the reference value and the set value is set large, and when the reference value is set high, the difference between the reference value and the set value is set small. This makes it possible to determine the quasi-low value area so that the quasi-low value area, which is an area that should be noted, is not overlooked. The set reference value and set value may be displayed, for example, on the display image 81.

[0070] In addition, when the reference value and the set value are set, a reference value setting menu may be displayed on the display. Regarding the reference value setting menu, when a reference value setting operation is accepted by operating the keyboard 19, for example, the reference value setting unit displays the reference value setting menu as shown in FIG. 12 on the display 18. The reference value can be changed, for example, between 0% and 100%, and is assigned to a slide bar 101a. The set value can also be changed, for example, between 0% and 100%, and is assigned to a slide bar 101b. When changing the reference value, the reference value is changed by operating the keyboard 19 or the like to match the slider 102a to the position on the slide bar 101a indicating the reference value to be changed. The set value is also changed by operating the keyboard 19 to match the slider 102b to the position on the slide bar 101b indicating the set value to be changed. In this embodiment, an oxygen saturation level of 40% is assigned to the reference value by the slide bar 101a, and an oxygen saturation level of 80% is assigned to the set value by the slide bar 101b.

[0071] When the reference value and the set value satisfy a preset condition, the reference value setting unit 73 preferably changes and sets the reference value to a preset value that is higher than the reference value before the change and does not exceed the set value. This is because, when the reference value is too low, the reference value can be easily changed to an appropriate value and the low value area 84 can be appropriately displayed by simply switching and setting the reference value to an appropriate value.

[0072] Furthermore, when a quasi-low value region exists, the quasi-low value region can be included in the low value region 84 by switching and setting the reference value to an appropriate value, so that the region that was a quasi-low value region can also be displayed in the bioinformation image 83, and by displaying the region that was a quasi-low value region and the low value region 84, it is possible to prevent these from being overlooked by displaying specific parts of the bioinformation. Note that the details of the preset conditions and the preset values ​​are appropriately set depending on, for example, the site or the purpose of observation.

[0073] Specifically, as shown in FIG. 13, for example, when the reference value is set to 40% and the set value is set to 80%, and detailed observation of a site that may have a lesion is performed, if the preset condition is "if the reference value is 20% or more lower than the set value, the reference value is changed to a value 10% lower than the set value", the reference value is 30% lower than the set value, which satisfies the condition "the reference value is 20% or more lower than the set value", so the reference value setting unit 73 switches and sets the reference value to 70%. It is preferable to switch the reference value when, for example, a preset time has elapsed or when the site recognition unit 76 or the like recognizes that the subject has changed. For example, when moving from a site where detailed observation was performed with a low reference value to another site and performing screening to observe whether there is a lesion, it is considered that the reference value is too low and the low value area 84 is rarely displayed in the bioinformation image 83, but by the reference value setting unit 73 switching and setting the reference value, the low value area 84 can be appropriately displayed even if the site to be observed is moved.

[0074] The appropriateness determination unit 75 determines that the reference value is not appropriate when a quasi-low value region in which the biometric information is higher than the reference value and is equal to or lower than a preset value exists in the biometric information image 83. As shown in FIG. 14, when a quasi-low value region 105 exists in the biometric information image 83, the appropriateness determination unit 75 determines that the reference value is not appropriate. Note that the quasi-low value region 105 is included in the biometric information image 83, but cannot be visually identified because it is displayed as transparent or colorless in some cases like other regions by the reference value color bar 87. Also, by looking at the reference value color bar 87, the reference value indicator 95, and the average value indicator 98, it is possible to infer that the endoscopic image 82 contains many parts where the oxygen saturation is higher than the reference value but close to the reference value, since the average value indicator 98 is near the reference value indicator 95.

[0075] The appropriateness determination unit 75 may determine that the reference value is inappropriate when the quasi-low value region 105 exists in the biometric information image 83 at a rate equal to or greater than a preset rate. When the area of ​​the quasi-low value region 105 exists in the biometric information image 83 at a rate equal to or greater than a preset rate relative to the entire area of ​​the biometric information image 83, it is preferable for the user to check the location or distribution of the quasi-low value region 105. The area of ​​the biometric information image 83 is the same as the area of ​​the endoscopic image 82 because the calculation of the biometric information is performed for the entire endoscopic image 82. By the appropriateness determination unit 75 determining that the reference value is inappropriate, the user can be prompted to check the quasi-low value region 105 even in a situation where the quasi-low value region 105 is not displayed, and the notification of the determination can be appropriately performed.

[0076] Specifically, for example, when the biological information is oxygen saturation, the reference value is 40%, and the set value is 50%, if the area of ​​the quasi-low value region 105 where the oxygen saturation is higher than 40% and lower than 50% is 10% or more of the oxygen saturation image 83, which is a preset value, the reference value is determined to be inappropriate. As shown in FIG. 15, in FIG. 15(A), the quasi-low value region 105 is less than 10% of the area of ​​the oxygen saturation image 83, so the appropriateness determination unit 75 does not determine that the reference value is inappropriate. Note that the area of ​​the oxygen saturation image 83 is the same as the area of ​​the endoscopic image 82 because the oxygen saturation is calculated for the entire endoscopic image 82. On the other hand, in FIG. 15(B), the quasi-low value region 105 is 10% or more of the area of ​​the oxygen saturation image 83, so the appropriateness determination unit 75 determines that the reference value is inappropriate.

[0077] By having the appropriateness judgment unit 75 judge the standard value based on the area proportion of the endoscopic image 82 that the quasi-low value region 105 occupies, the judgment of the standard value can be set in more detail, and it is preferable because it can prevent frequent judgments that the standard value is inappropriate and can provide appropriate notifications.

[0078] The appropriateness determination unit 75 may determine that the reference value is inappropriate when the quasi-low value region 105 continues to exist for a preset time or more in the biometric information image 83. When the quasi-low value region 105 continues to exist in the biometric information image 83, it is preferable that the user checks the location or distribution of the quasi-low value region 105. Even in a situation where the quasi-low value region 105 is not displayed, the appropriateness determination unit 75 can determine that the reference value is inappropriate, thereby urging the user to check the quasi-low value region 105, and can also appropriately notify the user of the determination.

[0079] Specifically, for example, when the biological information is oxygen saturation, the reference value is 40%, and the set value is 50%, if the area of ​​the quasi-low value region 105 where the oxygen saturation is higher than 40% and lower than 50% exists for 30 seconds or more, which is a preset time. As shown in FIG. 16, in the case of FIG. 16(A) where the quasi-low value region 105 starts to exist from time t1 seconds and does not continue at time t1+30 seconds, the appropriateness determination unit 75 determines that the reference value is appropriate. On the other hand, in the case of FIG. 16(C) where the quasi-low value region 105 continues to exist at time t1+30 seconds, the appropriateness determination unit 75 determines that the reference value is inappropriate. If the quasi-low value region 105 exists at time t1+30 seconds but does not continue for 30 seconds or more, the appropriateness determination unit 75 determines that the reference value is inappropriate. That is, in the case of Fig. 16(B), there is a quasi-low value region 105 at time t1+30 seconds, but since the time at which this quasi-low value region 105 occurred is time t2, if the quasi-low value region 105 continues to exist until time t2+30 seconds, the appropriateness determination unit 75 determines that the reference value is inappropriate, but in the case of Fig. 16(B) where the quasi-low value region 105 does not continue to exist until time t2+30 seconds, the appropriateness determination unit 75 does not determine that the reference value is inappropriate. Note that in Fig. 16, each diagonal line indicates the existence of a quasi-low value region 105.

[0080] By having the appropriateness judgment unit 75 judge the reference value based on the duration of the existence of the quasi-low value region 105, the judgment of the reference value can be set in more detail, and frequent judgments that the reference value is inappropriate can be prevented, making it possible to properly notify the user, which is preferable.

[0081] In addition, when both the ratio of the quasi-low value region 105 in the bioinformation image 83 and the duration of the quasi-low value region 105 are satisfied, the reference value may be determined to be inappropriate. For example, when the reference value for oxygen saturation is 40% and the set value is 50%, the reference value is determined to be inappropriate when the area of ​​the quasi-low value region 105, which is oxygen saturation higher than 40% and less than 50%, exists in 10% or more of the oxygen saturation image for 30 seconds or more continuously. The conditions for the determination are appropriately set according to the part or the condition of the subject. The appropriateness determination unit 75 determines the reference value using both the area ratio of the quasi-low value region 105 and the duration of the existence of the quasi-low value region 105, which is preferable because the determination of the reference value can be set in more detail and the notification can be appropriately performed by preventing the reference value from being determined to be inappropriate frequently.

[0082] When the appropriateness determination unit 75 determines that the reference value is not appropriate, it is preferable that the reference value setting unit 73 switches the reference value to a higher value. Details such as the range of change when switching the reference value are appropriately set depending on, for example, the site or the purpose of observation.

[0083] Specifically, as shown in FIG. 17, for example, when the reference value is 40% and the appropriateness judgment unit 75 judges that the reference value is inappropriate, the reference value setting unit 73 feeds back the judgment of the appropriateness judgment unit 75, thereby increasing the reference value by 10% from 40%, which is a preset range, and switches and sets the reference value to 50%.

[0084] It is preferable that the reference value setting unit 73 feeds back the judgment of the appropriateness judgment unit 75 and switches and sets the reference value to a higher value, so that even if the reference value is too low, it can be switched and set to an appropriate reference value without any trouble.

[0085] The notification control unit 67 issues a notification regarding the reference value when the appropriateness determination unit 75 determines that the reference value is not appropriate. The notification regarding the reference value is a notification for making the user aware of the presence of the quasi-low value region 105. Therefore, the notification method or notification content may be any method or content that allows the user to recognize that fact. Specifically, for example, the notification is issued by displaying the notification content on the display 18. In this case, the notification control unit 67 sends information regarding the notification by display to the display control unit 66. The display control unit 66 controls the display.

[0086] The notification content may be such that the user can recognize the presence of the quasi-low value region 105. As shown in FIG. 18, for example, the notification is made by displaying a warning display 107 somewhere on the display 18. Alternatively, for example, when the reference value color bar 87 is displayed on the display 18, the notification is made by displaying an average value indicator 98 in the shape of an arrow on the reference value color bar 87. If the average value indicator 98 is close to the reference value indicator 95, it is highly likely that the quasi-low value region 105 exists, so that the presence of the quasi-low value region 105 can be notified to the user by displaying the average value indicator 98 together with the reference value color bar 87. Note that the reference value indicator 95 and the high value indicator 96 are also displayed on the reference value color bar 87. Alternatively, as shown in FIG. 19, in order to make it difficult to miss the average value indicator 98 in the shape of an arrow, the average value indicator 98 may be displayed by changing the color of the arrow, for example, by blinking the average value indicator 98 in sequence every second.

[0087] In addition, the notification control unit 67 may notify that the reference value is appropriate when the appropriateness determination unit 75 does not determine that the reference value is inappropriate. Regarding the notification, for example, the appropriateness is notified by not notifying when the reference value is determined to be inappropriate. Also, for example, when the reference value is appropriate, the display 18 may always display that fact, and when the reference value becomes inappropriate, the display may be changed to a warning display. In addition, when the reference value color bar 87 is displayed on the display 18, the color or size of the arrow-shaped average value indicator 98 shown on the reference value color bar 87 may be changed when the reference value becomes inappropriate, to attract the user's attention.

[0088] As shown in Figure 20, for example, when an arrow-shaped average value indicator 98 is displayed on a reference value color bar 87, if the reference value is appropriate, the arrow-shaped average value indicator 98 is displayed in a normal size, and if the reference value is no longer appropriate, the arrow-shaped average value indicator 98 is displayed in a larger size.

[0089] According to the processor device 16, which is the image processing device described above, when the biological information is visualized to observe the biological information, if a reference value is set to make both the endoscopic image 82 and the biological information image 83 easier to see, a notification regarding the reference value is given when a quasi-low value region 105 that is higher than the reference value and equal to or lower than the set value exists, so that the quasi-low value region 105 is not overlooked and attention can be paid to the quasi-low value region 105. Therefore, for example, in observation using an endoscope, the image processing device can prevent overlooking not only the low value region 84 where the biological information is equal to or lower than the reference value, but also the quasi-low value region 105 where there is a possibility of a lesion or the like. In the past, when the biological information is lower than the reference value, the imaging of the biological information does not display that the biological information is lower than the reference value, so that the user may assume that there is no region with low oxygen saturation, for example, when the biological information is oxygen saturation.

[0090] In particular, for example, in the case where the oxygen saturation of the subject is generally low, if the reference value is set low in order to observe an area with particularly low oxygen saturation, or if the hypoxic area (low value area) 84 lower than the reference value disappears due to a change in the observation site or the passage of time, the hypoxic area 84 is not displayed in the oxygen saturation image, which is the bioinformation image 83. However, according to the processor device 16, if there is a subhypoxic area (sublow value area) 105 that is higher than the reference value but has a low oxygen saturation, a notification is given, so that it is possible to prevent the user from noticing both the hypoxic area 84 and the subhypoxic area 105, which are specific areas in terms of oxygen saturation.

[0091] Furthermore, when a notification regarding the reference value is given, the user can immediately change the reference value. The reference value can be easily changed, for example, in increments of 1%, using the keyboard 19, the scope button 12g, or a foot switch (not shown). Therefore, the reference value can be changed quickly and flexibly without imposing a burden on the user. Furthermore, when the reference value is changed, various types of corresponding information, such as reference value correspondence information based on the reference value or the reference value color bar 87, are automatically changed. Therefore, the user can perform observation with appropriate settings or display without much effort, and areas with specific biological information are prevented from being overlooked.

[0092] It is preferable that the processor device 16 is connected to the display 18 that displays the display image 81 and includes a display image storage unit 109 that stores the display image 81, and a display control unit 66 that controls the display of the stored display image 81 as a thumbnail on the display 18. As shown in Fig. 21, the processor device further includes the display image storage unit 109. The display image 81 generated by the display image generation unit 71 is stored in the display image storage unit 109.

[0093] The display control unit 66 performs control to display one or more display images 81 stored by the display image storage unit 109 as thumbnails on the display 18 in accordance with a preset condition. The preset condition is, for example, selection based on a preset criterion from all display images 81 displayed from a predetermined period of time in the past to the present. When the display images 81 are created for each frame, for example, one display image 81 is selected every 30 seconds, and four display images 81 selected from the period from 30 seconds to 2 minutes in the past are displayed as thumbnails. When a new display image 81 is generated, the oldest display image 81 among the display images 81 displayed as thumbnails on the display 18 is deleted, and the new display image 81 is displayed instead, and this is repeated every 30 seconds. In this case, changes in biological information can be displayed in chronological order during observation with the endoscope.

[0094] Specifically, as shown in Fig. 22, for example, when four display images 81 are selected every 30 seconds and thumbnail images 111 selected from 30 seconds to two minutes ago are displayed, the thumbnail images 111 of 30 seconds ago, 1 minute ago, 1 minute 30 seconds ago, and 2 minutes ago are displayed in order of most recent from the top to the right of the currently observed display image 81. Note that in Fig. 22, only some of the reference symbols are used to avoid cluttering the drawing.

[0095] The preset condition may be, for example, past display images 81 of the same site of the same patient. In this case, comparison of biological information of the same site or the like can be easily performed. For example, the change in oxygen saturation before and after anastomosis of the stomach of the same patient can be displayed on the same display 18. Also, for example, by using information on the acquisition time included in the endoscopic image 82, the display images 81 selected at regular time intervals from the display image storage unit 109 may be displayed as thumbnails.

[0096] In addition, among the past display images 81 displayed as thumbnails, the display images 81 may be bioinformation images 83 in which bioinformation is displayed in pseudo color. Pseudo color is one of the methods of imaging bioinformation, and displays different colors according to bioinformation based on the bioinformation calculated for each pixel of the endoscopic image 82. This method is called pseudo color because it uses pseudo colors different from the colors shown in natural observation of the endoscopic image. In order to display the bioinformation in pseudo color, for example, a method of using color correspondence information (color table) or a method of applying different gains to the spectral image of the endoscopic image according to the bioinformation can be mentioned. The color correspondence information is correspondence information in which different colors are associated with the bioinformation. In this embodiment, the bioinformation is oxygen saturation, so the color correspondence information is pseudo color correspondence information in which different colors are associated with the oxygen saturation, and is pseudo color correspondence information in which the range of oxygen saturation is associated with each color, for example, from blue to red.

[0097] The pseudo-color correspondence information can be set according to the type of subject of the bioinformation image, the purpose of observation, etc. For example, the pseudo-color correspondence information is set so that each 1% of oxygen saturation from 0% to 100% corresponds to a color of each of 101 steps from blue to red. Therefore, the pseudo-color correspondence information is correspondence information that corresponds the measured value of oxygen saturation to a color, and the value of oxygen saturation corresponds to a color on a one-to-one basis. This allows a color according to the measured value of oxygen saturation to be displayed for each pixel in the oxygen saturation image, which is a bioinformation image. Therefore, since a color according to the absolute measured value of oxygen saturation is displayed in the thumbnail image, it is possible to compare oxygen saturation between images, and it is easy to grasp the absolute change in oxygen saturation.

[0098] In addition, when the reference value setting unit 73 sets a reference value, the display control unit 66 preferably performs control to change the display image 81 to be displayed as a thumbnail on the display based on the reference value and then display the image. When the reference value setting unit 73 sets a reference value, the display image generating unit 71 generates color correspondence information for the reference value. Then, when the color correspondence information for the reference value is generated, the display image generating unit 71 generates the biometric information image 83 based on the color correspondence information for the reference value for the biometric information image 83 included in the display image 81 displayed as a thumbnail. The display image generating unit 71 may save the biometric information image 83 included in the display image 81 displayed as a thumbnail, or may acquire the display image 81 saved in the display image saving unit 109. The display image generating unit 71 regenerates the biometric information image 83 included in the display image 81 displayed as a thumbnail, and the display image 81 is also regenerated. The generated display image 81 has the biometric information image 83 changed in response to the change in the reference value. As soon as the display image 81 is generated, the display image is displayed as a thumbnail again. Furthermore, when the generated display image 81 is stored in the display image storage unit 109, for example, if a display image 81 containing an endoscopic image 82 containing the same acquisition time is stored, this past display image 81 is updated and stored. In this way, when the reference value is changed, the display image 81 displayed as a thumbnail is also changed in accordance with the changed reference value and displayed again.

[0099] When past display images 81 are displayed as thumbnails, by changing the reference value, the display images displayed as thumbnails are also displayed based on the changed reference value. This makes it possible to compare display images 81 using a single reference value between images for all display images 81 displayed on the display 18, which is preferable as it makes it easier to grasp changes in oxygen saturation.

[0100] As in the present embodiment, the biological information is preferably the oxygen saturation of the subject. By using the oxygen saturation as the biological information, the oxygen saturation of the subject can be efficiently observed using a reference value. It is known that an area of ​​the subject with low oxygen saturation may be a lesion or a suture failure of the intestine or the like. Therefore, by using the oxygen saturation as the biological information, it is effective for observation, examination, or diagnosis by an endoscope.

[0101] In addition, the low value region 84 below the reference value of the biological information and the quasi-low value region 105 have been described above, but the same can be applied to a high value region above the reference value of the biological information and a quasi-high value region.

[0102] Next, a series of processing steps performed by the image processing device will be described with reference to the flowchart shown in FIG. 23. When observation is started, an endoscopic image 82 is acquired (step ST110), and biometric information is calculated based on the acquired endoscopic image 82 (step ST120). The reference value setting unit 73 sets a reference value (step ST130). A biometric information image is generated based on the reference value (step ST140). A display image is generated based on the biometric information image (step ST150). The judgment unit judges the reference value (step ST160), and if the judgment result is that the reference value is not appropriate (YES in step ST170), a notification is made (step ST180). If the judgment result is not that the reference value is not appropriate (NO in step ST170), no notification is made. When observation is ended (YES in step ST190), the series of steps ends. If observation is to be continued (NO in step ST190), the process returns to acquiring the endoscopic image 82.

[0103] In the above embodiment, the present invention is applied to an endoscopic system that processes endoscopic images 82, but the present invention can also be applied to medical image processing systems that process medical images other than endoscopic images 82 when saving still images.

[0104] In the above-described embodiments and modified examples, the processor device 16 functions as an image processing device, but as shown in FIG. 24, in addition to the processor device 16, the image processing unit 61 can be provided with an image processing device 201 including a display image generation unit 71, a biometric information calculation unit 72, a reference value setting unit 73, a biometric information image generation unit 74, an appropriateness determination unit 75, and a part recognition unit 76.

[0105] 25, the display image generating unit 71, the biometric information calculating unit 72, the reference value setting unit 73, the biometric information image generating unit 74, the suitability determining unit 75, and the part recognizing unit 76 can be provided in a diagnosis support device 911 that acquires a RAW image captured by an endoscope 12, for example, directly from the endoscope system 10 or indirectly from a PACS (Picture Archiving and Communication Systems) 910. Also, as shown in FIG. 26, the display image generating unit 71, the biometric information calculating unit 72, the reference value setting unit 73, the biometric information image generating unit 74, the suitability determining unit 75, and the part recognizing unit 76 can be provided in a medical work support device 930 that is connected to various inspection devices, such as a first inspection device 921, a second inspection device 922, ..., a Kth inspection device 923, including the endoscope system 10, via a network 926.

[0106] The above-mentioned embodiments and modifications may be combined in part or in whole in any manner. In the above-mentioned embodiments and modifications, the endoscope 12 is a so-called flexible endoscope having a flexible insertion section 12a, but the present invention is also suitable for use with a capsule-type endoscope that is swallowed by the subject, or a rigid endoscope (lapaoscope) used in surgery, etc.

[0107] The above embodiments and variations include a method of operating an image processing device equipped with an image processor, in which the image processor acquires an endoscopic image obtained by imaging a subject with an endoscope, calculates biometric information based on the endoscopic image, sets a reference value for the biometric information, generates a biometric image in which the biometric information is visualized so that low-value areas below the reference value can be distinguished from others, generates a display image in which the biometric image is superimposed on the endoscopic image, and if a quasi-low-value area in which the biometric information is higher than the reference value and below a predetermined set value is present in the biometric information image, determines that the reference value is inappropriate, and if it determines that the reference value is inappropriate, provides a notification regarding the reference value.

[0108] In the above embodiment, the hardware structure of the processing units that execute various processes, such as the control unit 52, image acquisition unit 54, DSP 56, noise reduction unit 58, conversion unit 59, image processing unit 61, display control unit 66, and notification control unit 67 included in the processor device 16, is various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing various processes.

[0109] One processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same type or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with one processor. As an example of configuring multiple processing units with one processor, first, as represented by a computer such as a client or server, one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units. Second, as represented by a system on chip (SoC), there is a form in which a processor is used that realizes the functions of the entire system including multiple processing units with one IC (Integrated Circuit) chip. In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.

[0110] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit in the form of a combination of circuit elements such as semiconductor elements.

[0111] The present invention can be used in systems or devices that acquire medical images (including videos) other than endoscopic images, in addition to endoscope systems, processor devices, and other related devices that acquire endoscopic images, etc. For example, the present invention can be applied to ultrasound examination devices, X-ray imaging devices (including CT (Computed Tomography) examination devices and mammography devices, etc.), MRI (magnetic resonance imaging) devices, etc. [Explanation of symbols]

[0112] 10 Endoscope System 12 Endoscopy 12a Insertion part 12b Operation section 12c Curved section 12d Tip 12e Angle knob 12f Treatment tool insertion port 12g Scope button 13 Zoom operation section 14 Light source device 16 Processor Unit 18 Display 19 Keyboard 20 Light source section 20a V-LED 20b B-LED 20c G-LED 20d R-LED 22 Light source control unit 30a illumination optical system 30b Shooting optical system 45 Lighting Lens 46 Objective Lens 47 Zoom Lens 48 Image Sensor 52 Control section 54 Image acquisition unit 56 DSP 58 Noise reduction section 59 Conversion section 61 Image processing section 66 Display control unit 67 Notification control section 71 Display image generation unit 72 Biometric information calculation unit 73 Reference value setting section 74 Biometric image generation unit 75 Appropriateness Judgment Section 76 Part recognition part 81 Display images 82 Endoscopic Images 83 Biometric images (oxygen saturation images) 84 Low value area (hypoxia area) 85 High price range 86 Intermediate area 87 Reference Color Bar 91~94 scale 95 Reference Value Indicators 96 High Price Index 97 Overlay Images 98 Average Value Index 101a, 101b Slide bar 102a, 102b Slider 105 Subhypoxic region (subhypoxic region) 107 Warning display 109 Image storage for display 111 thumbnail images 201 Image processing device 910 PACS 911 Diagnostic Support Device 921 First Inspection Device 922 Second Inspection Device 923 Kth Inspection Device 926 Network 930 Medical Business Support Equipment ST110~ST190 Step

Claims

1. An image processor is provided, The image processor includes: An endoscopic image is obtained by capturing an image of a subject using an endoscope; Calculating biological information based on the endoscopic image; Recognizing a part of the subject based on the endoscopic image; setting a reference value of the biological information based on the type of the part; An image processing device that generates a biometric information image by imaging the biometric information such that a low-value region of the biometric information equal to or lower than the reference value and a region of the biometric information greater than the reference value can be distinguished.

2. The image processing apparatus according to claim 1 , wherein the image processor generates an image for display based on the biometric information image.

3. The image processing device according to claim 1 , wherein the image processor recognizes a part of the subject from the endoscopic image by using a trained model based on machine learning.

4. The image processor includes: When a quasi-low value region in which the biometric information is higher than the reference value and is equal to or lower than a preset value exists in the biometric information image, the reference value is determined to be inappropriate; 4. The image processing apparatus according to claim 1, further comprising: a notification regarding the reference value when the reference value is determined to be inappropriate.

5. 5. An image processing device according to claim 1, wherein the image processor generates color correspondence information for a reference value in which an area where the biometric information is below the reference value corresponds to a specific color, and generates the biometric information image based on the color correspondence information for the reference value.

6. The image processing device according to claim 5 , wherein, when the reference value is set, the image processor generates the reference value color correspondence information based on the set reference value.

7. 7. The image processing apparatus according to claim 5, wherein the image processor generates an image for display using a reference value color bar obtained by visualizing the reference value color correspondence information and a reference value indicator indicating the reference value.

8. The image processing device according to claim 7 , wherein, when the reference value is set, the image processor generates the display image using the reference value index indicating the set reference value.

9. The image processor calculates an average value of the biological information in the endoscopic image, 9. The image processing device according to claim 7, wherein the display image is generated using the reference color bar and an average value indicator indicating the average value.

10. a display that displays the display image and a display image storage unit that stores the display image; 3. The image processing device according to claim 2, wherein the image processor controls the display of the display images stored in the display image storage unit in a thumbnail form on the display.

11. 11. The image processing apparatus according to claim 10, wherein, when the reference value is set, the image processor performs control to change the display image, the thumbnail of which is displayed on the display, based on the reference value before displaying the image.

12. 5. The image processing apparatus according to claim 4, wherein the image processor determines that the reference value is inappropriate when the quasi-low value region exists in the biometric information image at a rate equal to or greater than a preset rate.

13. 5. The image processing apparatus according to claim 4, wherein said image processor determines that said reference value is inappropriate when said quasi-low value region continues to exist for a preset time or longer.

14. 14. The image processing device according to claim 1, wherein the image processor notifies the user that the reference value is appropriate when the image processor does not determine that the reference value is inappropriate.

15. 15. The image processing device according to claim 1, wherein the image processor, when determining that the reference value is inappropriate, switches the reference value to a higher value.

16. 5. The image processing device according to claim 4, wherein the image processor changes and sets the reference value to a preset value that is higher than the reference value before the change and does not exceed the set value when the reference value and the set value satisfy a preset condition.

17. The image processing device according to claim 1 , wherein the biological information is an oxygen saturation level of the subject.

18. 1. A method of operating an image processing apparatus having an image processor, comprising: The image processor includes: An endoscopic image is obtained by capturing an image of a subject using an endoscope; Calculating biological information based on the endoscopic image; Recognizing a part of the subject based on the endoscopic image; setting a reference value of the biological information based on the type of the part; A method for operating an image processing device that generates a biometric information image by imaging the biometric information so that a low-value region of the biometric information below the reference value and a region of the biometric information greater than the reference value can be distinguished.

Images