Endoscope system, generation method of biological parameter, and program

Abstract

To provide an endoscope system, a generation method of a biological parameter image, and a program capable of generating a biological parameter image with high accuracy and high robustness.SOLUTION: An endoscope system includes a processor. The processor performs control of generating an alignment first image by performing positioning processing on a first endoscope image on the basis of a plurality of second endoscope images, acquiring an alignment monochrome image set including the plurality of alignment first images, calculating a degree of oxygen saturation of an observation target on the basis of the acquired alignment first image set, generating a biological parameter image on the basis of the degree of oxygen saturation, and displaying the biological parameter image on a display.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an endoscope system, a method for generating a biological parameter image, and a program. [Background technology]

[0002] In recent years, in the medical field using endoscopes, technologies for generating images of biological parameters such as oxygen saturation imaging have become known. Oxygen saturation imaging is a technology that calculates hemoglobin oxygen saturation from a small number of spectral information points, such as two or three, in visible light.

[0003] A known technique for oxygen saturation imaging is to use image signals in four wavelength bands to accurately calculate oxygen saturation, thereby generating an oxygen saturation image in which the influence of yellow pigment, which interferes with endoscopic observation, is corrected (Patent Document 1).Another known technique is to calculate the amount of movement of an object to be observed using first and second image signals obtained by capturing images of the object illuminated with first and second illumination lights, respectively, and then calculate the oxygen saturation using a method according to the amount of movement (Patent Document 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-177961 [Patent Document 2] Japanese Patent Application Laid-Open No. 2016-185398 Summary of the Invention [Problem to be solved by the invention]

[0005] With the advancement of image processing technology, there has been a demand for generating biological parameter images with higher accuracy and robustness.

[0006] An object of the present invention is to provide an endoscope system, a method and a program for generating a biological parameter image that can generate a biological parameter image with higher accuracy and higher robustness. [Means for solving the problem]

[0007] The endoscopic system of the present invention comprises a light source device that alternately emits a plurality of first illumination lights and a second illumination light having a broader band than the first illumination lights as illumination light, an endoscope having an imaging sensor that images an object of observation illuminated with the illumination light, a processor device having a processor that generates a biological parameter image based on the endoscopic image obtained by the imaging sensor and controls the display of the biological parameter image on the display, and a display, wherein the plurality of first illumination lights are light of wavelength bands that each have a different dependence on oxygen saturation, and the processor generates an aligned first image by performing an alignment process on a first endoscopic image obtained by imaging the object of observation illuminated with the first illumination light based on a plurality of second endoscopic images obtained by imaging the object of observation illuminated with the second illumination light, obtains an aligned first image set including the plurality of aligned first images, calculates the oxygen saturation of the object of observation based on the obtained aligned first image set, and generates a biological parameter image based on the oxygen saturation.

[0008] The alignment process preferably includes a motion amount calculation process and a motion amount correction process, the motion amount calculation process being a process of calculating the motion amount of a first endoscopic image based on a plurality of second endoscopic images, and the motion amount correction process being a process of generating an aligned first image by performing correction based on the motion amount on the first endoscopic image whose motion amount has been calculated.

[0009] The light source device preferably alternately emits the second illumination light and any one of the plurality of first illumination lights for each frame. The center wavelength of each of the plurality of first illumination lights is preferably one of 450 nm, 470 nm, 540 nm, 620 nm, 690 nm, and 850 nm. There are five first illumination lights, and the center wavelengths of the five lights are preferably 470 nm, 540 nm, 620 nm, 690 nm, and 850 nm.

[0010] Preferably, the light source device illuminates the observation object with light-emitting units that emit all of the plurality of first illumination lights in a predetermined order, and the processor generates an aligned first image set based on the plurality of first endoscopic images obtained by the light-emitting units. Preferably, the light source device illuminates the observation object with light-emitting units that emit all of the plurality of first illumination lights in order of decreasing central wavelength. Preferably, the second illumination light is generated by emitting two or more of the plurality of first illumination lights. Preferably, the second illumination light is generated by simultaneously emitting first illumination lights of the plurality of first illumination lights having central wavelengths of 450 nm, 540 nm, and 620 nm.

[0011] The motion amount calculation process is preferably a process of calculating the motion amount of the first endoscopic image based on two second endoscopic images obtained in frames immediately before and after the first endoscopic image. The processor preferably calculates the oxygen saturation of the object of observation for each pixel of the aligned first image included in the aligned first image set.

[0012] The endoscope has a normal observation mode in which the light source device continuously emits the second illumination light and the processor controls the display to continuously display the second endoscopic image, and a biological parameter image acquisition mode in which the light source device alternately emits the second illumination light and any of the plurality of first illumination lights and the processor controls the display to display the biological parameter image, and it is preferable that the endoscope has a switching operation unit for switching between the normal observation mode and the biological parameter image acquisition mode, which switching operation unit is operated by the endoscope operator.

[0013] When the endoscope operator switches from normal observation mode to biometric parameter image acquisition mode by operating the switching operation unit, it is preferable that the processor generates a biometric parameter image based on the first aligned image set acquired first after the switch, controls the display of the generated biometric parameter image, and then automatically returns to normal observation mode.

[0014] It is preferable that the light source device illuminates the object to be observed using light-emitting units that emit all of the multiple first illumination lights in a predetermined order, and the processor calculates a total amount of motion by adding up the amount of motion in each of the multiple first endoscopic images obtained in the light-emitting units, and if the total amount of motion is within a predetermined range, acquires an aligned first image set including the multiple aligned first images obtained in the light-emitting units, and calculates oxygen saturation based on the acquired aligned first image set.

[0015] The light source device illuminates the object to be observed using light-emitting units that emit all of the multiple first illumination lights in a predetermined order, and the processor calculates a total amount of motion by adding up the amount of motion in each of the multiple first endoscopic images obtained in the light-emitting units, and preferably adjusts the resolution to be reduced so that the greater the total amount of motion, the greater the degree of reduction in the resolution of the aligned first images of the object for calculating the oxygen saturation of the object to be observed.

[0016] A method for generating a biological parameter image executed in an endoscopic system comprising a light source device, an endoscope, and a processor device, wherein the light source device alternately emits a plurality of first illumination lights and a second illumination light having a broader band than the first illumination lights as illumination light, the endoscope has an imaging sensor that images an object of observation illuminated by the light source device, and the processor device has a processor that generates a biological parameter image based on an endoscopic image obtained by the imaging sensor, wherein the plurality of first illumination lights are light of wavelength bands each having a different dependency on oxygen saturation, and the method includes the steps of: generating an aligned first image by performing alignment processing on a first endoscopic image obtained by imaging an object of observation illuminated with the first illumination light based on a plurality of second endoscopic images obtained by imaging the object of observation illuminated with the second illumination light; obtaining an aligned monochromatic image set including the plurality of aligned first images; calculating the oxygen saturation of the object of observation based on the aligned first image set; and generating a biological parameter image based on the oxygen saturation.

[0017] The present invention is a program to be executed by a processor of an endoscopic system including a light source device that alternately emits a plurality of first illumination lights and a second illumination light having a broader band than the first illumination light as illumination light, an endoscope having an imaging sensor that images an object of observation illuminated with the illumination light, and a processor device having a processor that generates a biological parameter image based on the endoscopic image obtained by the imaging sensor and controls the display of the biological parameter image on a display, and causes the processor to perform the functions of generating an aligned first image by performing alignment processing on a first endoscopic image obtained by imaging an object of observation illuminated with the first illumination light based on a plurality of second endoscopic images obtained by imaging the object of observation illuminated with the second illumination light, obtaining an aligned first image set including a plurality of aligned first images, calculating the oxygen saturation of the object of observation based on the aligned first image set, generating a biological parameter image based on the oxygen saturation, and controlling the display of the biological parameter image on a display. [Effects of the Invention]

[0018] According to the present invention, a biological parameter image can be generated with higher accuracy and robustness. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 10 is an explanatory diagram illustrating alignment of a single-color image. [Figure 2] 1 is a schematic diagram of an endoscopy system for the digestive tract. [Figure 3] 10A and 10B are explanatory diagrams illustrating the display mode of an endoscopic image on a display and an extended display in a normal observation mode. [Figure 4] 10A and 10B are explanatory diagrams showing the display mode of an endoscopic image on the display and the extended display in the oxygen saturation mode. [Figure 5] (A) shows an oxygen saturation image of the inside of the gastrointestinal tract, and (B) shows an oxygen saturation image of the serosal side of the gastrointestinal tract on an expanded display. [Figure 6] FIG. 2 is a block diagram showing the functions of the endoscope system. [Figure 7] 1 is a graph showing the emission spectra of monochromatic light b, sb, g, a, r, and ir. [Figure 8] FIG. 10 is an explanatory diagram showing a light emission pattern in an oxygen saturation mode. [Figure 9] 1 is a graph showing the spectral sensitivity of an image sensor. [Figure 10] 10 is a table showing image signals obtained in a normal observation mode. [Figure 11] 10 is a table showing spectral image signals obtained in an oxygen saturation mode. [Figure 12] 1 is a graph showing the relationship between the central wavelength of monochromatic light b, sb, g, a, r, and ir and the reflectance of hemoglobin. [Figure 13] FIG. 2 is a block diagram showing the functions of an advanced processor device. [Figure 14] FIG. 10 is an explanatory diagram illustrating calculation of a motion amount. [Figure 15] 10A and 10B are explanatory diagrams illustrating motion amount correction based on a change in illuminance. [Figure 16] 1 is a graph showing the relationship between reflectance R and ratio μa / μs′. [Figure 17] 10 is a flowchart showing a series of steps in an oxygen saturation mode. [Figure 18] FIG. 10 is an explanatory diagram showing the oxygen saturation, hemoglobin concentration, bilirubin concentration, and scattering wavelength-dependent parameters of the spectroscopic model that are minimum required for calculating biological parameters. [Figure 19] 10A and 10B are explanatory diagrams showing oxygen saturation, hemoglobin concentration, bilirubin concentration, and scattering wavelength-dependent parameters of a spectroscopic model required for improving the accuracy of calculation of biological parameters. [Figure 20] FIG. 10 is an explanatory diagram showing the oxygen saturation, hemoglobin concentration, bilirubin concentration, and scattering wavelength-dependent parameters of the spectroscopic model required to further improve the accuracy of the calculation of biological parameters. [Figure 21] 10 is a graph showing spectral reflectance determined by a spectral model. [Figure 22] FIG. 10 is a block diagram showing the functions of an extended processor device according to a second embodiment. [Figure 23] 10 is a table showing reference spectroscopic signals for each spectroscopic model. [Figure 24] 10 is a graph showing spectral reflectance determined in a phantom. [Figure 25] FIG. 10 is a block diagram showing the functions of an extended processor device according to a third embodiment. [Figure 26] 10 is a table showing reference spectroscopic signals for each phantom. [Figure 27] FIG. 10 is a block diagram showing the functions of an extended processor device according to a fourth embodiment. [Figure 28] FIG. 10 is an explanatory diagram showing a method for acquiring a reference spectroscopic signal in the fourth embodiment. [Figure 29] 1 is a schematic diagram of an endoscope system for laparoscopy. [Figure 30] FIG. 2 is an explanatory diagram showing an imaging unit having three monochrome imaging sensors. [Figure 31] FIG. 1 is an explanatory diagram showing an imaging unit having an imaging sensor for one color. [Figure 32] 1 is a graph showing an emission spectrum of broadband illumination light. [Figure 33] FIG. 2 is an explanatory diagram showing a pixel arrangement in a spectroscopic image sensor. [Figure 34] FIG. 10 is an explanatory diagram showing another light emission pattern in the oxygen saturation mode. DETAILED DESCRIPTION OF THE INVENTION

[0020] [First embodiment] An example of an embodiment of an endoscopic system, etc., of the present invention will be described. First, the background to the development of one of the following embodiments will be described. Endoscopic oxygen saturation imaging is a technology for calculating hemoglobin oxygen saturation from a small amount of spectral information of visible light. Conventionally, a technology is known for creating and displaying an oxygen saturation image from two or three spectral signals acquired by switching the illumination light in two or three frames of a video of endoscopic images acquired by an endoscope. However, if more spectral signals are used to create more accurate and robust oxygen saturation images, it is necessary to acquire many spectral images by switching the illumination light over more frames. In such a case, the influence of misalignment between images corresponding to different wavelengths in a set of spectral images used to create a biological parameter image, such as an oxygen saturation image, is likely to be greater.

[0021] In other words, by acquiring many spectroscopic images and obtaining biological parameters such as oxygen saturation based on these spectroscopic images, it is possible to calculate biological parameters with high accuracy, but there is a risk that the impact of positional deviation between spectroscopic images will become greater.

[0022] The endoscopic system, method and program for generating a biological parameter image of the present invention are configured to include a specific light source device, endoscope, processor device and display, respectively, thereby achieving the contradictory characteristics of high accuracy in biological parameters such as oxygen saturation and hemoglobin concentration, and high robustness against movement of the object being observed and / or the endoscope, etc., and making it possible to generate oxygen saturation images, etc. with greater accuracy and robustness and display them on the display.

[0023] The endoscopic system of the present invention includes a light source device, an endoscope, a processor device, and a display. These devices are connected to each other and capable of communicating with each other if necessary. The connection or communication may be wired or wireless. The endoscopic system may be any type of endoscopic system as long as it is an endoscopic system that acquires biological parameter images such as oxygen saturation images. Therefore, the endoscopic system may be any of an upper endoscope, a lower endoscope, a laparoscope, etc.

[0024] In this specification, a biological parameter image refers to an image in which biological parameters such as oxygen saturation and hemoglobin concentration of an object of observation are visualized using a heat map, numerical display, etc. Therefore, an oxygen saturation image refers to an image in which the oxygen saturation of an object of observation is visualized using a heat map, etc. Furthermore, an endoscopic image refers to an image obtained using an endoscope, and includes still images and videos.

[0025] The light source device alternately emits, as illumination light, a plurality of first illumination lights and any one of second illumination lights having a broader band than the first illumination light. In this embodiment, monochromatic light is emitted as the first illumination light, and polychromatic light is emitted as the second illumination light. Therefore, the light source device continuously emits illumination light in the following order: polychromatic light, any one of monochromatic light, polychromatic light, and any one of monochromatic light. Polychromatic light is normal light that an endoscope operator (hereinafter referred to as a user) normally uses during observation using an endoscope, and is illumination light that allows the observation object to be observed in natural colors. Therefore, polychromatic light is preferably white light obtained by emitting multiple monochromatic lights. Note that in this specification, white light also includes white-equivalent light, which has fewer short-wavelength components than normally used white light. White-equivalent light is also normal light that allows the observation object to be observed in natural colors.

[0026] The multiple monochromatic lights are multiple lights with different wavelength bands. Furthermore, the multiple monochromatic lights are lights with wavelength bands that have different dependencies on oxygen saturation. One piece of spectral information can be obtained from one monochromatic light. The multiple monochromatic lights are composed of two or more monochromatic lights. A larger number of monochromatic lights is preferable because it increases the amount of image information obtained and increases the accuracy of the calculated biological parameters. However, it is preferable to limit the number of monochromatic lights to a level that does not impair the effects of the present invention due to the complexity of the light source device and the calculation of the biological parameters. For example, in order to preferably calculate biological parameters such as oxygen saturation, it is preferable that the multiple monochromatic lights be three or more types with different wavelength bands.

[0027] An endoscope has an image sensor that captures an object illuminated by illumination light emitted by a light source device. The image sensor generates one still image, which is an endoscopic image, in one exposure period, as a unit of one frame. Therefore, each frame is obtained in chronological order. In an endoscopic system, the light source device, the image sensor, etc. are synchronized, so that endoscopic images of the object illuminated by each illumination light can be obtained. For example, still images are generated one frame at a time in the following order: a still image illuminated by polychromatic light, a still image illuminated by one of monochromatic lights, a still image illuminated by polychromatic light, and a still image illuminated by one of monochromatic lights.

[0028] The processor device includes a processor that generates a biological parameter image by performing processing based on an endoscopic image acquired by an imaging sensor and controls display of the biological parameter image on a display. The biological parameter image is generated based on the endoscopic image, which is a still image acquired by an endoscope.

[0029] The processor first generates an aligned monochromatic image (aligned first image) by performing alignment processing on a monochromatic endoscopic image (first endoscopic image) based on a plurality of multicolor endoscopic images (second endoscopic images) (hereinafter referred to as multicolor images) obtained by capturing an image of an object illuminated with polychromatic light. A monochromatic image is an endoscopic image obtained by capturing an image of an object illuminated with monochromatic light. Therefore, an aligned monochromatic image is a spectral image and is an image obtained by performing alignment processing on a monochromatic image. A plurality of aligned monochromatic images are obtained and are used as an aligned monochromatic image set (aligned first image set), and the oxygen saturation of the object of observation is calculated based on this aligned monochromatic image set.

[0030] Registration requires information on the amount of motion between the two images to be aligned. However, because the monochromatic images to be aligned have different wavelengths of illumination light, the contrast and other appearance of the blood vessels of the tissue being observed in the image vary greatly from image to image. For example, when blue light, green light, etc., contained in white light is used as illumination light, the blood vessels of the target tissue are clearly visible in the resulting endoscopic image, but when red light, near-infrared light, etc., the blood vessels are not clearly visible. Therefore, there are images in which it is difficult to extract feature points for calculating the amount of motion between images. Therefore, frame images captured using a common polychromatic illumination light are used to calculate the amount of motion between frame images captured using monochromatic light.

[0031] Since a multicolor image is an image of an object illuminated by the same illumination light, such as white light, it can be said that what appears in the image is more homogeneous and contains more image information than when the object is illuminated by monochromatic light. As mentioned above, since multiple monochromatic lights have different wavelength bands, when each of these lights is used as illumination light, the images obtained may contain different things.

[0032] Therefore, comparing multicolor images allows for a more accurate understanding of image misalignment than comparing monochromatic light images that depict different objects. By performing alignment processing on monochromatic images using multiple multicolor images as a reference, it is possible to perform alignment processing with high precision even when there are multiple monochromatic images. In the aligned monochromatic images, which are monochromatic images after such alignment processing, misalignment due to the movement of the observation subject, the movement of the endoscope, etc. has been processed and eliminated.

[0033] As a method for performing the alignment process for a monochromatic image based on a multi-color image, various processes can be adopted as long as the method can perform the alignment process for a monochromatic image based on a multi-color image. For example, the alignment process can be performed by using a monochromatic image selected from a plurality of monochromatic images as a reference, and aligning several frames of monochromatic images before and after the reference monochromatic image so that they match the selected monochromatic image.

[0034] As an example of the alignment process, first, a monochrome image is set as an even frame, and the amount of motion between odd frames of a multi-color image adjacent to the monochrome image is calculated, and the position of the even frame of the monochrome image sandwiched between them is estimated based on the calculated amount of motion. Next, the amount of motion between the monochrome images is calculated based on the estimated positions of the even frames of the multiple monochrome images. Then, one frame of the multiple monochrome images is used as a reference, and the other monochrome images can be aligned based on the amount of motion relative to the reference frame.

[0035] In this case, the alignment process preferably includes a motion amount calculation process and a motion amount correction process. The motion amount calculation process calculates the amount of motion between two multi-color images, for example, between two frames captured before and after the acquisition of a monochromatic image, i.e., between two frames captured in chronological order before and after the monochromatic image was captured, to determine the amount of motion between the multi-color images.

[0036] Specifically, the motion amount calculation process preferably involves dividing the multi-color image into a plurality of predetermined regions and calculating the motion amount. The method for calculating the motion amount will be described later. The calculated motion amount is then used to estimate the motion amount of the monochrome image. That is, based on two frames of multi-color images captured at different times, the motion amount can be estimated by interpolation for monochrome images captured between the times at which these multi-color images were captured. The interpolation method may be any commonly used method, such as linear interpolation or various interpolation methods.

[0037] The motion amount correction process is a process of correcting the amount of motion of each monochrome image using multiple monochrome images whose motion amounts have been estimated. For example, one reference monochrome image (hereinafter referred to as the reference monochrome image) can be selected from the multiple monochrome images whose motion amounts have been estimated, and the positions of the other monochrome images can be aligned with the reference monochrome image based on their respective motion amounts. Monochrome images that have undergone the above-described alignment process are aligned monochrome images.

[0038] A plurality of aligned monochromatic images are generated to form an aligned monochromatic image set. The aligned monochromatic images are spectral images composed of a plurality of monochromatic lights, each of which has a different dependence on oxygen saturation. Therefore, the aligned monochromatic image set contains a plurality of spectral image information, making it possible to calculate biological parameters such as oxygen saturation with greater accuracy. Furthermore, because these spectral images are precisely aligned, biological parameters can be calculated with high robustness against the movement of the observation target, the endoscope, and the like.

[0039] As shown in FIG. 1, specifically, when five monochromatic lights are used and image capture is performed by switching the illumination light frame by frame, eleven endoscopic images are acquired in chronological order based on the time t at which the images are acquired: a first white-light image NP1, a first monochromatic image MP1 obtained using the first monochromatic light, a second white-light image NP2, a second monochromatic image MP2 obtained using the second monochromatic light, a third white-light image NP3, a third monochromatic image MP3 obtained using the third monochromatic light, a fourth white-light image NP4, a fourth monochromatic image MP4 obtained using the fourth monochromatic light, a fifth white-light image NP5, a fifth monochromatic image MP5 obtained using the fifth monochromatic light, and a sixth white-light image NP6. The order of monochromatic images acquired using the first monochromatic light is repeated thereafter. Note that different shading in the figure indicates endoscopic images acquired using illumination light of different wavelength bands. In the following description, the first through fifth white-light images will be referred to simply as white-light images NP unless the order of the frames is specified. The sixth white light image NP6 is similar to the first white light image NP1, and after the sixth white light image NP6 is acquired, the first monochrome image MP1 is obtained.

[0040] The five monochrome images, namely, the first monochrome image MP1, the second monochrome image MP2, the third monochrome image MP3, the fourth monochrome image MP4, and the fifth monochrome image MP5, are aligned by an alignment processing unit 61 (see FIG. 13) using multicolor images of the previous and next frames. The alignment processing unit 61 includes a motion amount calculation unit 64 and a motion amount correction unit 65 (see FIG. 13).

[0041] The motion amount calculation unit 64 estimates the amount of motion of each of the five monochromatic images using the multi-color images of the previous and next frames. The amount of motion of the first monochromatic image MP1 is estimated by interpolation based on the amount of motion calculated from the first multi-color image NP1 and the second multi-color image NP2. The first monochromatic image MP1 is aligned based on the amount of motion of the first monochromatic image MP1. The second to fifth monochromatic images MP2 to MP5 are aligned in a similar manner. The alignment method will be described later.

[0042] The aligned first monochromatic image MP1 becomes an aligned first monochromatic image AMP1. Similarly, an aligned second monochromatic image AMP2, an aligned third monochromatic image AMP3, an aligned fourth monochromatic image AMP4, and an aligned fifth monochromatic image AMP5 are generated.

[0043] These aligned monochromatic images are combined into one set of aligned monochromatic images. The oxygen saturation, which is a biological parameter of the object of observation, is calculated based on one aligned monochromatic image set consisting of multiple aligned monochromatic images containing image information from light in wavelength bands that have different dependencies on oxygen saturation. Therefore, as described above, the biological parameters can be calculated with higher accuracy. Note that each of the multiple aligned monochromatic images has been aligned based on a multicolor image. Therefore, the biological parameters can be calculated with higher accuracy and robustness.

[0044] The display displays biological parameter images in nearly real time. Furthermore, by displaying multicolor images, the display also displays endoscopic images in multicolor light in nearly real time. Therefore, when operating the endoscope, the user can perform endoscopic operations, diagnoses, etc., while viewing the natural-colored endoscopic images and biological parameter images, such as oxygen saturation images, displayed on the display in nearly real time, without feeling any discomfort.

[0045] With the above-described configuration, the endoscope system and the like of the present invention can generate biological parameter images with higher accuracy and robustness.

[0046] Hereinafter, an embodiment will be described with reference to the drawings. As shown in Fig. 2, an endoscopic system 10 includes an endoscope 12, a light source device 13, a processor device 14, a display 15, a processor-side user interface 16, an extended processor device 17, and an extended display 18. The endoscope 12 is optically or electrically connected to the light source device 13 and electrically connected to the processor device 14. The extended processor device 17 is electrically connected to the light source device 13 and the processor device 14. Note that the display of the present invention includes the extended display 18 in addition to the display 15.

[0047] The endoscope 12 has an insertion section 12a, an operating section 12b, a bending section 12c, and a tip section 12d. The insertion section 12a is inserted into the body of the subject. The operating section 12b is provided at the base end of the insertion section 12a. The bending section 12c and the tip section 12d are provided on the tip side of the insertion section 12a. The bending section 12c is bent by operating the angle knob 12e of the operating section 12b. The tip section 12d is oriented in a direction desired by the user by bending the bending section 12c. A forceps channel (not shown) is provided from the insertion section 12a to the tip section 12d for inserting a treatment tool or the like. The treatment tool is inserted into the forceps channel from the forceps port 12j.

[0048] An optical system for forming an image of a subject and an optical system for irradiating the subject with illumination light are provided inside the endoscope 12. The operation unit 12b is provided with an angle knob 12e, a mode change switch 12f, a still image acquisition command switch 12h, and a zoom operation unit 12i. The mode change switch 12f is a change operation unit used to change the observation mode.

[0049] The endoscope system 10 has two observation modes: a normal observation mode and an oxygen saturation mode, which is a biological parameter image acquisition mode. The user switches between these modes by operating the mode selector switch 12f. In the normal observation mode, the light source device 13 continuously emits white light, which is a polychromatic light, and the processor controls the display to continuously display a polychromatic image. In the oxygen saturation mode, the light source device 13 alternately emits white light and one of multiple monochromatic lights, and the processor controls the display to display a biological parameter image. The user can switch between the normal observation mode and the oxygen saturation mode by operating the mode selector switch 12f.

[0050] The still image acquisition command switch 12h is used to command acquisition of a still image of the subject. The zoom operation unit 12i is used to operate to enlarge or reduce the observation target. The mode change switch 12f and the still image acquisition command switch 12h are included in the scope-side user interface 19 for performing various operations on the processor device 14.

[0051] The light source device 13 emits illumination light. The processor device 14 performs system control of the endoscope system 10 and also generates endoscopic images by performing image processing on image signals transmitted from the endoscope 12. The display 15 displays endoscopic images transmitted from the processor device 14. The processor-side user interface 16 has a keyboard, mouse, microphone, tablet, foot switch, touch pen, etc., and accepts input operations such as function settings.

[0052] As shown in Fig. 3, in normal observation mode, a white light image with natural coloring obtained by capturing an image of an observation target using white light as illumination light is displayed on the display 15, while nothing is displayed on the extended display 18. Note that in the figure, only a portion of the display 15 or 18 may be shown.

[0053] 4, in the oxygen saturation mode, the oxygen saturation of the object of observation is calculated, and the calculated oxygen saturation is visualized as an oxygen saturation image OP, which is displayed on the extended display 18. Also, in the oxygen saturation mode, a white light image NP is displayed on the display 15. The oxygen saturation image may also be displayed on the display 15.

[0054] The endoscope system 10 is used as an upper endoscope for examining the inside of the digestive tract, such as the stomach and large intestine, and the endoscope 12 is a flexible endoscope. For example, in the oxygen saturation mode, as shown in FIG. 5(A), an internal digestive tract oxygen saturation image GOP, which visualizes the oxygen saturation state inside the digestive tract, is displayed on the extended display 18. The endoscope system 100, which will be described later, is used in laparoscopic surgery, etc., which visualizes the serosa on the outside of organs, and the endoscope 12, which is a laparoscope, is a rigid endoscope. For example, in the oxygen saturation mode, as shown in FIG. 5(B), a serosal side oxygen saturation image SOP, which visualizes the oxygen saturation state on the serosal side of the large intestine, is displayed on the extended display 18.

[0055] In the oxygen saturation mode, it is possible to calculate the oxygen saturation more accurately in the following cases. - When observing a predetermined target area (e.g., esophagus, stomach, large intestine) - In environments other than those with ambient lighting - When there is no residue, residual fluid, mucus, blood, or fat remaining on the mucous membrane or serous membrane When dye is not sprayed onto the mucous membrane When the endoscope 12 is more than 7 mm away from the object to be observed When the endoscope 12 is not too far away from the object to be observed and is at an appropriate distance Area where the illumination light is sufficiently shining on the object to be observed When there is little specular reflection from the object being observed - The inner 2 / 3 area of ​​the entire oxygen saturation image When the movement of the endoscope 12 is small, or when there is little movement of the patient or the object of observation, such as pulsation or breathing. When blood vessels deep in the gastrointestinal mucosa cannot be observed

[0056] 6, the light source device 13 includes a light source section 20 and a light source processor 21 that directly controls the light source section 20. The central control section 53 of the processor device 14 controls the entire endoscope system 10, and controls the light source processor 21 (described later), etc. The extended processor device 17 includes an imaging processor 37, and performs processing based on the endoscopic image sent from the processor device 14 to generate a biological parameter image. This allows the various components of the endoscope system 10 to operate in conjunction with each other.

[0057] The light source unit 20 emits at least one of polychromatic light and monochromatic light as illumination light. The light source unit 20 has, for example, multiple semiconductor light sources, which are turned on or off, and when turned on, the light emission amount of each semiconductor light source is controlled to emit illumination light that illuminates the observation object. The light source unit 20 is equipped with LEDs (Light Emitting Diodes) as light sources, and includes six LEDs: a v-LED 20g (Violet-Light Emitting Diode), a b-LED 20a (Blue-Light Emitting Diode), a sb-LED 20b (Sky Blue-Light Emitting Diode), a g-LED 20c (Green-Light Emitting Diode), an a-LED 20d (amber-Light Emitting Diode), an r-LED 20e (Red-Light Emitting Diode), and an ir-LED 20f (Infra Red-Light Emitting Diode).

[0058] The number of light sources provided in the light source unit 20 is not limited to six. Any number may be provided that can obtain spectral images that allow accurate acquisition of biological parameters such as oxygen saturation or hemoglobin concentration, and at least three types of LEDs, sb-LED 20b, g-LED 20c, and a-LED 20d, may be provided. In addition to the above LEDs, a v-LED (Violet-Light Emitting Diode) or the like may be provided for use as a polychromatic white light illumination light.

[0059] Light emitted from each of the LEDs 20a to 20f is incident on a light guide 25 via an optical path combining unit 23 formed of a mirror, a lens, etc. The light guide 25 is built into the endoscope 12 and a universal cord (a cord that connects the endoscope 12 with the light source device 13 and the processor device 14). The light guide 25 propagates the light from the optical path combining unit 23 to the tip 12d of the endoscope 12.

[0060] The tip 12d of the endoscope 12 is provided with an illumination optical system 30 and an imaging optical system 31. The illumination optical system 30 has an illumination lens 32, and illumination light propagated by the light guide 25 is irradiated onto the observation object via the illumination lens 32. The imaging optical system 31 has an objective lens 35 and an imaging sensor 36. Reflected light from the observation object irradiated with the illumination light is incident on the imaging sensor 36 via the objective lens 35. As a result, an image of the observation object is formed on the imaging sensor 36.

[0061] The imaging sensor 36, which captures an image of an object to be observed, is preferably a color imaging sensor. Each pixel of the imaging sensor 36 is provided with either a B pixel (blue pixel) having a B (blue) color filter, a G pixel (green pixel) having a G (green) color filter, or an R pixel (red pixel) having an R (red) color filter. The spectral transmittances of the B color filter, G color filter, and R color filter, which determine the spectral sensitivity of the imaging sensor 36, will be described later. Note that the imaging sensor 36 is preferably a color imaging sensor with a Bayer array in which the ratio of the number of B pixels, G pixels, and R pixels is 1:2:1, for example.

[0062] A CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal-Oxide Semiconductor) image sensor can be used as the image sensor 36. Also, instead of the primary color image sensor 36, a complementary color image sensor equipped with complementary color filters of C (cyan), M (magenta), Y (yellow), and G (green) may be used. When a complementary color image sensor is used, image signals of four colors, CMYG, are output, and by converting the four CMYG image signals into three RGB image signals by complementary color-primary color conversion, image signals of each RGB color similar to those of the image sensor 36 can be obtained.

[0063] The imaging sensor 36 is driven and controlled by an imaging processor 37. Control of each mode in the imaging processor 37 will be described later. A CDS / AGC circuit 40 (Correlated Double Sampling / Automatic Gain Control) performs correlated double sampling (CDS) and automatic gain control (AGC) on the analog image signal obtained from the imaging sensor 36. The image signal that passes through the CDS / AGC circuit 40 is converted into a digital image signal by an A / D converter 41 (Analog / Digital). The digital image signal after A / D conversion is input to the processor device 14.

[0064] The processor device 14 includes a DSP (Digital Signal Processor) 45, an image processing unit 50, a display control unit 52, and a central control unit 53. Programs relating to various processes are stored in a program memory (not shown) in the processor device 14. The central control unit 53, which is made up of a processor, executes the programs in the program memory, thereby realizing the functions of the DSP 45, the image processing unit 50, the display control unit 52, and the central control unit 53.

[0065] The DMP 45 performs various signal processing on the image signals received from the endoscope 12, such as defect correction, offset processing, gain correction, demosaic processing, linear matrix processing, white balance processing, gamma conversion, YC conversion processing, and noise reduction processing. In defect correction processing, signals from defective pixels of the imaging sensor 36 are corrected. In offset processing, dark current components are removed from the image signals that have been subjected to defect correction processing, and an accurate zero level is set. In gain correction processing, the signal level of each image signal is adjusted by multiplying the image signals of each color after offset processing by a specific gain. The image signals of each color after gain correction processing are then subjected to demosaic processing and linear matrix processing to improve color reproducibility.

[0066] After linear matrix processing, white balance processing is performed, and then gamma conversion processing is performed to adjust the brightness and saturation of each image signal. After that, YC conversion processing is performed, and the luminance signal Y and color difference signals Cb and Cr are output to the DMP 45. The DMP 45 performs noise reduction processing using, for example, the moving average method or the median filter method.

[0067] The image processing unit 50 performs various image processing on the image signal from the DMP 45. Image processing includes color conversion processing such as 3x3 matrix processing, tone conversion processing, and 3D LUT (Look Up Table) processing, color enhancement processing, and structure enhancement processing such as spatial frequency enhancement. The image processing unit 50 performs image processing according to the mode. In the normal observation mode, the image processing unit 50 performs image processing for the normal observation mode to generate a white light image. In the oxygen saturation mode, the image processing unit 50 generates a white light image and transmits the image signal from the DMP 45 to the extension processor device 17 via the image communication unit 51. The extension processor device 17 generates an oxygen saturation image based on the transmitted image signal of the endoscopic image.

[0068] The display control unit 52 performs display control for displaying on the display 15 image information such as the white light image NP or the oxygen saturation image OP from the image processing unit 50, and other information.

[0069] The extended processor device 17 receives image signals from the processor device 14 and performs various image processing. In the oxygen saturation mode, the extended processor device 17 calculates the oxygen saturation level and generates an oxygen saturation image OP by visualizing the calculated oxygen saturation level. The extended processor device 17 will be described in detail later. The generated oxygen saturation image OP is displayed on the extended display 18.

[0070] The functions of the extended processor device 17 may be performed by the processor device 14, or both the extended processor device 17 and the processor device 14 may be configured to perform the functions of the extended processor device 17. In this case, different biological parameters may be generated by the extended processor device 17 and the processor device 14. Therefore, the processor device having a processor for generating biological parameter images may be only the processor device 14, only the extended processor device 17, or both the processor device 14 and the extended processor device 17, etc.

[0071] The illumination light in each mode will be described. The central wavelength of each of the multiple monochromatic lights is preferably one of 470 nm, 540 nm, 620 nm, 690 nm, and 850 nm. As will be described later, the monochromatic lights having these central wavelengths are light in wavelength bands that differ from one another in their dependence on oxygen saturation (see FIG. 11). Three or more of these multiple monochromatic lights are used. Preferably, there are five monochromatic lights, each with a central wavelength of 470 nm, 540 nm, 620 nm, 690 nm, and 850 nm. In the case of three monochromatic lights, the central wavelengths can be 540 nm, 620 nm, and 690 nm. Each of these lights can be emitted by an LED light source. In each mode, the light can be turned on or off to control the emission of polychromatic light, monochromatic light, etc.

[0072] As shown in Figure 7, the v-LED 20g emits monochromatic light v with a central wavelength of 410 nm. The b-LED 20a emits monochromatic light b with a central wavelength of 450 nm. The sb-LED 20b emits monochromatic light sb with a central wavelength of 470 nm. The g-LED 20c emits monochromatic light g with a central wavelength of 540 nm. The a-LED 20d emits monochromatic light a with a central wavelength of 620 nm. The r-LED 20e emits monochromatic light r with a central wavelength of 690 nm. The ir-LED 20f emits monochromatic light ir with a central wavelength of 850 nm. Note that the central wavelength of each monochromatic light may or may not be the same as the peak wavelength. Also, the graphs of light intensity at each wavelength in the figure are schematic and may not represent actual light intensity.

[0073] The polychromatic light is preferably generated by emitting two or more monochromatic lights from among a plurality of monochromatic lights. In the normal observation mode, the v-LED 20g, the b-LED 20a, the g-LED 20c, and the r-LED 20e are simultaneously turned on to emit polychromatic light including monochromatic light v with a central wavelength of 410 nm, monochromatic light b with a central wavelength of 450 nm, monochromatic light g with a central wavelength of 540 nm, and monochromatic light r with a central wavelength of 690 nm.

[0074] As shown in Figure 8, in the oxygen saturation mode, monochromatic light having central wavelengths of 450 nm, 470 nm, 540 nm, 620 nm, 690 nm, and 850 nm is emitted in sequence in the even-numbered frames of frame 2, frame 4, frame 6, frame 8, and frame 10, respectively, and polychromatic light is emitted between each emission of monochromatic light. The polychromatic light in the oxygen saturation mode is polychromatic light including monochromatic light b having a central wavelength of 450 nm, monochromatic light g having a central wavelength of 540 nm, and monochromatic light r having a central wavelength of 690 nm, and is emitted in the odd-numbered frames of frame 1, frame 3, frame 5, frame 7, and frame 9. In this way, the light source device 13 preferably illuminates the observation object with light-emitting units that emit each of the multiple monochromatic lights in ascending order of central wavelength, or preferably illuminates the observation object with light-emitting units that emit each of the multiple monochromatic lights in a predetermined order. Furthermore, it is preferable that the light source device 13 alternately emits polychromatic light and any one of a plurality of monochromatic lights for each frame.

[0075] Therefore, if there are five types of monochromatic light, each with a central wavelength of 470 nm, 540 nm, 620 nm, 690 nm, and 850 nm, and polychromatic light and these monochromatic lights are emitted alternately every frame, as shown in FIG. 8, light emission is repeated for 10 frames, from frame 1 to frame 10. In this case, since frames 1 to 10 are repeated, frame 1 to frame 1 constitutes a light emission unit. Furthermore, one light emission unit consists of 10 frames, from frame 1 to frame 10, and 10 endoscopic images are acquired per light emission unit.

[0076] Specifically, when light source device 13 repeatedly emits light in light emission units for 10 frames from frame 1 to frame 10, it emits monochromatic light sb, g, a, r, and ir with central wavelengths of 470 nm, 540 nm, 620 nm, 690 nm, and 850 nm in even-numbered frames 2, 4, 6, 8, and 10, respectively, as illumination light, and emits polychromatic light including monochromatic light b with a central wavelength of 450 nm, monochromatic light g with a central wavelength of 540 nm, and monochromatic light r with a central wavelength of 690 nm in odd-numbered frames 1, 3, 5, 7, and 9 between the even frames.

[0077] The spectral transmittance of the B, G, and R color filters will now be described. As shown in FIG. 9, the B color filter BF provided in the B pixel of the image sensor 36 transmits mainly light in the blue band, specifically light with a wavelength band of 380 to 560 nm (blue transmission band). The peak wavelength at which the transmittance is maximum is around 460 to 470 nm. The G color filter GF provided in the G pixel of the image sensor 36 transmits mainly light in the green band, specifically light with a wavelength band of 450 to 630 nm (green transmission band). The R color filter RF provided in the R pixel of the image sensor 36 transmits mainly light in the red band, specifically light with a wavelength band of 580 to 900 nm (red transmission band).

[0078] The control of each mode in the imaging processor 37 will be described. As shown in FIG. 10, in the normal observation mode 54a, the imaging processor 37 controls the imaging sensor 36 to capture an image of an object illuminated with polychromatic light including monochromatic light b, monochromatic light g, and monochromatic light r, frame by frame. The polychromatic illumination light is continuously emitted, and frame by frame imaging is continued. As a result, Bc image signals are output from the B pixels of the imaging sensor 36, Gc image signals are output from the G pixels, and Rc image signals are output from the R pixels. A white-light image NP is generated based on the Bc, Gc, and Rc image signals. The white-light image NP is a polychromatic image because it is an endoscopic image obtained by capturing an image of an object illuminated with polychromatic light, which simultaneously emits multiple monochromatic lights.

[0079] 11 , in the oxygen saturation mode 54b, in odd-numbered frames 1, 3, 5, 7, and 9, the imaging processor 37 controls the imaging of an object illuminated with polychromatic light including monochromatic light b, monochromatic light g, and monochromatic light r. As a result, in frames 1, 3, 5, 7, and 9, Bc image signals are output from the B pixels of the imaging sensor 36, Gc image signals are output from the G pixels, and Rc image signals are output from the R pixels. First to fifth white light images NP1, NP2, NP3, NP4, and NP5 are generated for each of the odd-numbered frames 1, 3, 5, 7, and 9 based on the Bc, Gc, and Rc image signals.

[0080] Meanwhile, in frame 2 of the even-numbered frames, the imaging processor 37 controls the imaging of the object illuminated with monochromatic light sb. As a result, B1 image signals are output from the B pixels of the imaging sensor 36, G1 image signals are output from the G pixels, and R1 image signals are output from the R pixels. Similarly, in frames 4, 6, 8, and 10 of the even-numbered frames, the imaging processor 37 controls the imaging of the object illuminated with monochromatic light g, a, r, and ir. As a result, B2, B3, B4, and B5 image signals are output from the B pixels of the imaging sensor 36, G2, G3, G4, and G5 image signals are output from the G pixels, and R2, R3, R4, and R5 image signals are output from the R pixels.

[0081] The B1, G1, and R1 image signals form a first monochrome image MP1. The B2, G2, and R2 image signals form a second monochrome image MP2. The B3, G3, and R3 image signals form a third monochrome image MP3. The B4, G4, and R4 image signals form a fourth monochrome image MP4. The B5, G5, and R5 image signals form a fifth monochrome image MP5.

[0082] To calculate biological parameters such as oxygen saturation, the B1 image signal, B2 image signal, G2 image signal, R3 image signal, R4 image signal, B5 image signal, G5 image signal, and R5 image signal are used from the image signals obtained in frames 2, 4, 6, 8, and 10 of the even-numbered frames. For the B5 image signal, G5 image signal, and R5 image signal, the MR5 image signal obtained by combining (adding) these three image signals is used. Hereinafter, when distinguishing from the reference image described later, the B1 image signal will be referred to as the B1 image signal. img Image signal, B2 for B2 image signal img Image signal, G2 for G2 image signal img Image signal, G3 for R3 image signal img Image signal, R3 for image signal img Image signal, R4 for image signal img Image signal, MR5 image signal, MR5 img This is referred to as the image signal. img Image signal, B2 img Image signal, G2 img Image signal, R3 img Image signal, R4 img Image signal, B5 img Image signal, G5 img Image signal, MR5 img The image signal is expressed in either the pre-registration or post-registration state.

[0083] The B1 image signal contains image information related to monochromatic light sb among the light transmitted through the B color filter BF. The B2 image signal contains image information related to monochromatic light g among the light transmitted through the B color filter BF. The G2 image signal contains image information related to monochromatic light g among the light transmitted through the G color filter GF. The R3 image signal contains image information related to monochromatic light a among the light transmitted through the R color filter RF. The R4 image signal contains image information related to monochromatic light r among the light transmitted through the R color filter. The B5 image signal contains image information related to monochromatic light ir among the light transmitted through the B color filter. The G5 image signal contains image information related to monochromatic light ir among the light transmitted through the G color filter. The R5 image signal contains image information related to monochromatic light ir among the light transmitted through the R color filter.

[0084] 12, the image information related to the monochromatic light sb includes image information of the wavelength band sb whose reflection spectrum changes with changes in the oxygen saturation of blood hemoglobin. Similarly, the image information related to the monochromatic light g, a, r, and ir includes image information of the wavelength bands g, a, r, and ir whose reflection spectrum changes with changes in the oxygen saturation of blood hemoglobin. Note that curve 55a represents the reflection spectrum of reduced hemoglobin, and curve 55b represents the reflection spectrum of oxygenated hemoglobin.

[0085] The extension processor device 17 will now be described. As shown in Fig. 13, the extension processor device 17 includes an image acquisition section 60, a registration processing section 61, a biological parameter calculation section 62, a display control section 63, and a reference image setting section 66. The registration processing section 61 includes a motion amount calculation section 64 and a motion amount correction section 65. The extension processor device 17 has programs related to various processes stored in a program memory (not shown). A central control section (not shown) formed by a processor executes the programs in the program memory, thereby realizing the functions of the image acquisition section 60, the registration processing section 61, the biological parameter calculation section 62, and the display control section 63.

[0086] The image acquisition unit 60 acquires, via the processor device 14, multicolor images and monochromatic images obtained by capturing images of an object illuminated with polychromatic light and monochromatic light using the imaging sensor 36. The processor calculates oxygen saturation by performing processing based on the monochromatic images. As described above, the processor then performs alignment processing on the monochromatic images. The alignment processing includes a motion calculation process by the motion calculation unit 64 and a motion correction process by the motion correction unit 65. The motion calculation process calculates the amount of motion of a monochromatic image based on multiple multicolor images, and the motion correction process generates an aligned monochromatic image by performing correction based on the amount of motion on the monochromatic image whose motion amount has been calculated.

[0087] The motion amount calculation unit 64 included in the alignment processing unit 61 estimates and calculates the amount of motion for each monochrome image, generating a motion amount-containing monochrome image, which is a monochrome image to which a motion amount has been assigned (see FIG. 1). Furthermore, the motion amount correction unit 65 performs alignment by correcting each motion amount-containing monochrome image based on the amount of motion calculated by the motion amount calculation unit 64. These processes can be performed by known methods, such as the method described in Japanese Patent Application Laid-Open No. 2016-185398. In this embodiment, alignment is performed by calculating the amount of motion in the horizontal direction of the image, calculating the amount of motion in the vertical direction of the image, and correcting these motion amounts. Therefore, alignment includes correcting the amount of motion in the horizontal direction, correcting the amount of motion in the vertical direction, and a combination of these.

[0088] The horizontal motion amount of an image is calculated as follows. Specifically, when calculating the horizontal motion amount using the first multi-color image NP1G and the second multi-color image NP2G, as shown in Fig. 14, the two images, the first multi-color image NP1G and the second multi-color image NP2G, for which the relative motion amount is to be calculated, are each divided into a predetermined number of regions (M × N rectangular regions Mij (i = 1, ..., M, j = 1, ..., N) with M vertical and N horizontal regions), and block matching is performed between small regions Mij at the same position in each image to calculate a representative motion vector Vij for each small region. The first multi-color image NP1G and the second multi-color image NP2G are G channel images. The motion vector Vij is assigned to the central pixel of each small region Mij, and the motion vectors of the other pixels are calculated by linear interpolation taking into account the representative motion vectors of the surrounding small regions (vector diagram VNP). This calculation is performed using the G channel image, in which blood vessels are visible with high contrast. This is because the blood vessels visible in the image allow for accurate calculation of the motion amount.

[0089] After obtaining the amount of motion as a vector based on the multi-color images, the amount of motion of the monochrome image captured between the two multi-color images is estimated. The amount of motion of the monochrome image can be estimated by halving the sum of the vectors of the amount of motion obtained based on the two multi-color images. This is because the monochrome image was captured at exactly the midpoint between the two multi-color images. Thus, the amount of motion of the monochrome image is obtained by interpolating the amounts of motion in the two multi-color images, and the amount of motion of the monochrome image is obtained as a vector by, for example, linear interpolation. The amount of motion is obtained for each monochrome image, and a motion-amount-containing monochrome image is created. Based on this amount of motion, horizontal alignment of each monochrome image is performed. The horizontal alignment described above is performed for the first through fifth monochrome images MP1 through MP5.

[0090] Therefore, the motion amount calculation process by motion amount calculation section 64 is preferably a process of calculating the motion amount of the monochrome image based on two multi-color images obtained in frames immediately before and after the monochrome image.

[0091] Next, when calculating the amount of movement perpendicular to the image, if there is movement that changes the distance to the object being observed between frames, the illuminance will change between frames, which will cause the spectral signal ratio between frames to vary from the true value and become a noise factor in the calculated oxygen saturation. Therefore, the illuminance change between frames is corrected using the following procedure.

[0092] A specified region is set in the center of each odd-numbered frame image (using the R channel image), and the average pixel value within the region is calculated (r(i), i = 1, 3, ..., 11). Using these, an estimate of the value in the even-numbered frame is calculated by averaging the values ​​in the adjacent odd-numbered frames (r(2) = (r(1) + r(3)) / 2, ...). Using these even-numbered frame values ​​r(2), r(4), ..., the entire pixel values ​​of the even-numbered frame images (frames 2, 4, 8, and 10, see Figure 11) after the horizontal alignment are multiplied by r(6) / r(2), r(6) / r(4), r(6) / r(8), and r(6) / r(10), respectively. This allows the illumination change between frames to be corrected using frame 6 as the reference.

[0093] As shown in FIG. 15, a predetermined region is set in the center of the first multi-color image NP1R and the second multi-color image NP2R, which are odd-numbered frames, and the average pixel values ​​within the region are r(1) and r(3), respectively. The first multi-color image NP1R and the second multi-color image NP2R are R channel images. For the first monochromatic image MP1, which is an even-numbered frame acquired between the first multi-color image NP1R and the second multi-color image NP2R, the entire pixel values ​​of the horizontally aligned frame image (see FIG. 14) are multiplied by r(6) / r(2), to obtain an aligned first monochromatic image AMP1. This allows correction for alignment based on the amount of movement in the direction perpendicular to the image, for the first monochromatic image MP1 after horizontal alignment, using the third monochromatic image MP3 of frame 6 as the reference.

[0094] Similarly, for the second monochrome image MP2R, all pixel values ​​of the horizontally aligned frame image (see FIG. 14) are multiplied by r(6) / r(4) to obtain an aligned second monochrome image AMP2. Because the third monochrome image N3R is the reference, no multiplication is performed here to obtain an aligned third monochrome image AMP3. Similarly, for the fourth monochrome image MP4R, all pixel values ​​of the horizontally aligned frame image (see FIG. 14) are multiplied by r(6) / r(8) to obtain an aligned fourth monochrome image AMP4. Similarly, for the fifth monochrome image NP5R, all pixel values ​​of the horizontally aligned frame image (see FIG. 14) are multiplied by r(6) / r(10) to obtain an aligned fifth monochrome image AMP5.

[0095] The first aligned monochromatic image AMP1 to the fifth aligned monochromatic image AMP5 are images obtained within a single emission unit, and are therefore considered an aligned monochromatic image set. As the user continues observation in the oxygen saturation mode, a time series of aligned monochromatic image sets is obtained one after the other.

[0096] The registration processing unit 61 may determine the degree of misalignment when aligning each monochromatic image when generating an aligned monochromatic image set. That is, the motion amount calculation unit 64 calculates a total motion amount for each light-emitting unit by adding up the motion amounts between each monochromatic image acquired in that light-emitting unit. The calculated total motion amount may then be regarded as the degree of misalignment, and it may be determined whether the misalignment exceeds a predetermined threshold. The determination of the misalignment between each monochromatic image acquired in that light-emitting unit may be performed by summing up the motion vectors between all the monochromatic images and comparing them with a preset threshold. Alternatively, the threshold may be determined by summing up only the magnitude of the motion vectors, or by using a specific direction as the threshold.

[0097] If the total amount of movement is within a preset range, an aligned image set including multiple aligned monochromatic images obtained in the light-emitting unit may be acquired. As a result, if the accuracy and reliability of the calculated oxygen saturation level are low, such as when the positional deviation is large, the oxygen saturation image is not displayed on the extended display 18. This improves the reliability of the displayed oxygen saturation image.

[0098] If the misalignment between the monochromatic images acquired per emission unit exceeds a predetermined threshold, the resolution of each monochromatic image may be adjusted according to the degree of misalignment before being generated as an aligned monochromatic image set. The resolution adjustment process may be performed by, for example, averaging the image size. In this case, the resolution may be reduced by an average of 1 / 8 if the degree of misalignment is determined to be small, or by an average of 1 / 16 if the degree of misalignment is determined to be large. The resolution of each monochromatic image may be reduced before generating the aligned monochromatic image set. It is preferable that the resolution of the aligned monochromatic image set with reduced resolution is restored to its original resolution after the biological parameter calculation unit 62 calculates the oxygen saturation level, and then an oxygen saturation image is generated.

[0099] The biological parameter calculation unit 62 calculates biological parameters so that a calculated error value based on the error between the reference image and the aligned monochromatic image falls within a specific range (fitting process). The reference image is determined based on a spectroscopic model that uses biological parameters as arguments, and is therefore defined as a function of the biological parameters. In this embodiment, four biological parameters are used: oxygen saturation S, hemoglobin concentration Hb, bilirubin concentration Bb, and scattering wavelength-dependent parameter b.

[0100] A reference image is defined for each aligned monochrome image. img Image signal, B2 img Image signal, G2 img Image signal, R3 img Image signal, R4 img Image signal and MR5 img As a reference image corresponding to the image signal, B1 std Image signal, B2std Image signal, G2 std Image signal, R3 std Image signal, R4 std Image signal and MR5 std The image signal is defined.

[0101] The calculated error value Diff is preferably calculated by the square error calculation formula based on (Formula 1). Diff=W B1 (B1 std (S,Hb,Bb,b)-B1 img ) 2 +W B2 (B2 std (S,Hb,Bb,b)-B2 img ) 2 +W G1 (G2 std (S,Hb,Bb,b)-G2 img ) 2 +W R3 (R3 std (S,Hb,Bb,b)-R3 img ) 2 +W R4 (R4 cal (S,Hb,Bb,b)-R4 img ) 2 +W MR5 (MR5 std (S,Hb,Bb,b)-MR5 img ) 2 ...(Formula 1) In addition, B1 std (S,Hb,Bb,b), B2 std (S,Hb,Bb,b), G2 std (S,Hb,Bb,b), R3 std (S,Hb,Bb,b), R4 std (S,Hb,Bb,b) and MR5 std (S, Hb, Bb, b) are B1 std Image signal, B2 std Image signal, G2 std Image signal, R3 std Image signal, R4 stdImage signal and MR5 std It represents the image signal, and is a function of the oxygen saturation S, hemoglobin concentration Hb, bilirubin concentration Bb, and scattering wavelength-dependent parameter b. img , B2 img , G2 img , R3 img , R4 img , and MR5 img are B1 img Image signal, B2 img Image signal, G2 img Image signal, R3 img Image signal, R4 img Image signal and MR5 img It represents the image signal. B1 , W B2 , W G2 , W R3 , W R4 , W MR5 represents a weighting factor for the squared difference value of each term.

[0102] When the calculated error value Diff is calculated by the above (Equation 1), the biological parameter calculation unit 62 calculates the biological parameters so that the calculated error value Diff becomes minimum. As a result, the oxygen saturation level S, the hemoglobin concentration Hb, the bilirubin concentration Bb, and the scattering wavelength-dependent parameter b are obtained as biological parameters. The biological parameters may be calculated for each pixel, or may be calculated for each pixel region having multiple pixels.

[0103] In this embodiment, it is preferable to calculate the biological parameters using N or more reference spectral signals at different wavelengths determined by N-1 (N is an integer greater than 3) biological parameters including oxygen saturation, and N or more first spectral signals corresponding to the wavelengths of the N or more reference signals. By calculating the biological parameters using N first spectral signals that are greater than N-1, which is the number of biological parameters, the N-1 biological parameters can be calculated with high accuracy.

[0104] In this embodiment, in order to accurately calculate the four biological parameters, namely, the oxygen saturation level S, the hemoglobin concentration Hb, the bilirubin concentration Bb, and the scattering wavelength-dependent parameter b, six image signals, namely, a B1 image signal, a B2 image signal, a G2 image signal, an R3 image signal, an R4 image signal, and an MR5 image signal, are used as the measured first spectral image signals.

[0105] The display control unit 63 displays the calculated oxygen saturation level as an visualized oxygen saturation image on the extended display 18 in various display modes. For example, it is preferable to display a still image of the oxygen saturation level on the extended display 18 in parallel with a moving white light image created by polychromatic light. Furthermore, when displaying the moving white light image and the moving oxygen saturation image in parallel, it is preferable to display the moving oxygen saturation image at a reduced frame rate. As described above, it is preferable that the white light image be generated based on a polychromatic image obtained by capturing an image of an object illuminated with polychromatic light using the imaging sensor 36. It is preferable that the polychromatic image be the Bc, Gc, and Rc image signals obtained in odd-numbered frames (frames 1, 3, 5, 7, and 9) in the oxygen saturation mode.

[0106] The display of the oxygen saturation image may be controlled based on the switching of the observation mode. When the user switches from the normal observation mode to the biological parameter image acquisition mode by operating the mode switch 12f, which is a switching operation unit, an oxygen saturation image, which is a biological parameter image, may be generated based on the first set of aligned monochromatic images acquired after the switch, and the generated oxygen saturation image may be displayed on the extended display 18. Note that when the oxygen saturation image is displayed on the extended display 18, the observation mode may automatically return to the normal observation mode. In this case, the oxygen saturation image may remain displayed on the extended display 18 until the next oxygen saturation image is generated.

[0107] In addition to oxygen saturation, hemoglobin concentration, bilirubin concentration, and scattering wavelength-dependent parameters may be calculated as biological parameters, and these may also be displayed as images on the extended display 18. For example, a hemoglobin concentration image based on hemoglobin concentration may be displayed on the extended display 18. Because a combination of hemoglobin concentration and oxygen saturation can be used as an index of congestion, it is preferable to display an oxygen saturation image and a hemoglobin concentration image on the extended display 18 alongside a white light image. Also, an index combining oxygen saturation and hemoglobin concentration may be visualized and displayed on the extended display 18. For example, a pseudocolor image in which oxygen saturation is assigned to color differences Cr and Cb and hemoglobin concentration is assigned to luminance Y is displayed on the extended display 18. In this case, the higher the hemoglobin concentration in the pseudocolor image, the darker it appears.

[0108] The reference image setting unit 66 sets a reference image based on the emission spectrum of the illumination light, the spectral reflectance of the living body determined from the spectral model, and the spectral sensitivity of the image sensor 36. The reference image is preferably normalized by a reference emission value corresponding to the emission spectrum.

[0109] In this embodiment, B1 std (S, Hb, Bb, b) are set according to (Equation 2).

number

[0110] In this embodiment, B2 std (S, Hb, Bb, b) are set according to (Equation 3).

number

[0111] In this embodiment, G2 std (S, Hb, Bb, b) are set according to (Equation 4).

number

[0112] In this embodiment, R3 std (S, Hb, Bb, b) are set according to (Equation 5).

number

[0113] In this embodiment, R4 std(S, Hb, Bb, b) are set according to (Equation 6).

number

[0114] In this embodiment, MR5 std (S, Hb, Bb, b) are set according to (Equation 7).

number

[0115] The spectral reflectance R(λ;S,Hb,Bb,b) of a living body is calculated from the ratio μa / μs' between the absorption coefficient μa(λ;Hb,S,Bb) and the scattering coefficient μs'(λ;b). The relationship between the spectral reflectance R and the ratio μa / μs' is shown in Figure 16. Note that the functional form of the spectral reflectance R (ratio μa / μs') may be stored in a table, or may be approximated and stored using a simple function such as a polynomial. In Figure 16, both the spectral reflectance R and the ratio μa / μs' are logarithmic.

[0116] The absorption coefficient μa(λ;Hb,S,Bb) is expressed by (Equation 8). μa(λ;Hb,S,Bb) =Hb×{S / 100×μaHbO2(λ)+(1-S / 100)×μaHb(λ)} +Bb×μaBb(λ)...(Formula 8) In (Equation 8), μaHbO2(λ) represents the absorption coefficient of oxygenated hemoglobin, Hb(λ) represents the absorption coefficient of reduced hemoglobin, μaBb(λ) represents the absorption coefficient of bilirubin, and λ represents the wavelength (similar to Equation (9)).

[0117] The scattering coefficient μs′(λ;b) is expressed by (Equation 9). μs´(λ;b)=14cm -1 ×(λ / 500nm) -b ...(Formula 9)

[0118] Next, a series of steps in the oxygen saturation mode will be described with reference to the flowchart in Fig. 17. While the user is observing an object in the normal observation mode, the mode selector switch 12f is operated to switch to the oxygen saturation mode (step ST010). Polychromatic light and monochromatic light are alternately used as illumination light (step ST020). The image acquisition unit 60 captures polychromatic and monochromatic images of the object illuminated with polychromatic and monochromatic light using the image sensor 36 (step ST030).

[0119] The registration processing unit 61 calculates the amount of motion for the single-color image using the multi-color image (step ST040). The registration processing unit 61 calculates the amount of motion for the multiple single-color images obtained in the light-emitting unit, and performs registration based on the calculated amount of motion (step ST050).

[0120] When performing alignment, the alignment processing unit 61 may determine whether the misalignment is within a predetermined threshold in the multiple aligned monochromatic images obtained in the light-emitting unit. If it is within the threshold (Y in step ST060), an aligned monochromatic image set is generated (step ST070). If it is outside the threshold (N in step ST060), the resolution of the multiple aligned monochromatic images is adjusted (step ST080). The resolution adjustment may be performed according to the degree of misalignment. For example, the images may be reduced in size on average and then generated as an aligned monochromatic image set (step ST080). If the degree of misalignment is determined to be medium, the average reduction of the images may be 1 / 8, and if the degree of misalignment is determined to be large, the average reduction may be 1 / 16.

[0121] For the generated set of aligned monochromatic images, the oxygen saturation is calculated for each pixel of the aligned monochromatic images, and an oxygen saturation image is generated based on the calculated oxygen saturation and displayed on the extended display 18 (step ST090). If the oxygen saturation mode is to be continued (Y in step ST100), observation using the specific illumination light is continued. If the oxygen saturation mode is to be ended (N in step ST100), the oxygen saturation mode is ended.

[0122] [Second embodiment] In the second embodiment, when calculating a biological parameter, a plurality of fixed values ​​are defined in advance in the spectral model, rather than being used as arguments (variables), and biological parameters that satisfy a condition are calculated from among the fixed values. In the second embodiment, M sets of reference images used in calculating the biological parameters are set for each spectral model from the emission spectrum of the illumination light, the spectral reflectance of the living body determined from the spectral model, and the spectral sensitivity of the image sensor 36. Note that, apart from the setting of the reference image, the process is the same as in the first embodiment.

[0123] Specifically, 16 spectroscopic models are used as the minimum spectroscopic models required to calculate biological parameters (M=16). In this case, as shown in Fig. 18, a total of 16 spectroscopic models SP1 to SP16 are used for two cases of oxygen saturation, "high oxygen saturation" and "low oxygen saturation," two cases of hemoglobin concentration, "high hemoglobin concentration" and "low hemoglobin concentration," two cases of bilirubin concentration, "high bilirubin concentration" and "low bilirubin concentration," and two cases of scattering wavelength-dependent parameters, "high scattering wavelength-dependent parameter" and "low scattering wavelength-dependent parameter." In this case, 16 sets of reference spectroscopic signals are set. In FIG. 18, "S" represents oxygen saturation, "Hb" represents hemoglobin concentration, "Bb" represents bilirubin concentration, and "b" represents a scattering wavelength-dependent parameter, with "high" representing a high concentration or a high parameter and "low" representing a low concentration or a low parameter (the same applies to FIGS. 19 and 20 (in the case of FIGS. 19 and 20, "medium" represents a medium concentration or an intermediate parameter)).

[0124] Furthermore, 81 spectroscopic models (M=81) are used as spectroscopic models necessary to improve the calculation accuracy of biological parameters. In this case, as shown in Fig. 19, a total of 81 spectroscopic models SP1 to SP81 are used for three cases of oxygen saturation: "high oxygen saturation," "medium oxygen saturation," and "low oxygen saturation," three cases of hemoglobin concentration: "high hemoglobin concentration," "medium hemoglobin concentration," and "low hemoglobin concentration," three cases of bilirubin concentration: "high bilirubin concentration," "medium bilirubin concentration," and three cases of scattering wavelength-dependent parameters: "high scattering wavelength-dependent parameter," "medium scattering wavelength-dependent parameter," and "low scattering wavelength-dependent parameter." In this case, 81 sets of reference spectroscopic signals are set.

[0125] Furthermore, as a spectroscopic model required to further improve the accuracy of calculation of biological parameters, as shown in Fig. 20, an intermediate concentration may be set for oxygen saturation between "high oxygen saturation" and "middle oxygen saturation," and an intermediate concentration may be set between "middle oxygen saturation" and "low oxygen saturation." Similar intermediate concentrations may also be set for other hemoglobin concentration, bilirubin concentration, and scattering wavelength-dependent parameters. In this case, a total of M spectroscopic models SP1 to SPM are used. In this case, M sets of reference spectroscopic signals are set.

[0126] In the following, for the sake of simplicity, the following description will be given using some of the spectral models SPX1 to SPX4 used to calculate the biological parameters. As shown in Fig. 21, the spectral reflectance RLX1 determined by the spectral model SPX1 is determined by the biological parameter LPX1. The biological parameter LP1 includes an oxygen saturation SX1 for low oxygen saturation, a hemoglobin concentration HbX1 for high hemoglobin concentration, a medium bilirubin concentration BbX1, and a medium scattering wavelength-dependent parameter bX1.

[0127] The spectral reflectance RL2X determined by the spectral model SPX2 is determined by the biological parameter LPX2. The biological parameter LP2 includes the oxygen saturation SX2 at high oxygen saturation, the hemoglobin concentration HbX2 at high hemoglobin concentration, the medium bilirubin concentration BbX1, and the medium scattering wavelength-dependent parameter bX1. The spectral reflectance RLX3 determined by the spectral model SPX3 is determined by the biological parameter LPX3. The biological parameter LPX3 includes the oxygen saturation SX3 at low oxygen saturation, the hemoglobin concentration HbX3 at low hemoglobin concentration, the medium bilirubin concentration BbX1, and the medium scattering wavelength-dependent parameter bX1. The spectral reflectance RLX4 determined by the spectral model SPX4 is determined by the biological parameter LPX4. The biological parameter LPX4 includes the oxygen saturation SX4 at high oxygen saturation, the hemoglobin concentration HbX4 at low hemoglobin concentration, the medium bilirubin concentration BbX1, and the medium scattering wavelength-dependent parameter bX1.

[0128] As shown in Fig. 22, the reference image setting unit 71 of the second embodiment sets a reference image for the spectral model SPX1 from the emission spectrum of the illumination light, the spectral reflectance RLX1, and the spectral sensitivity of the image sensor 36. The method of setting the reference image is the same as in the first embodiment. As shown in Fig. 23, the reference image for the spectral model SPX1 obtained as described above includes six B1 std Image signal, B2 std Image signal, G2 std Image signal, R3 std Image signal, R4 std Image signal and MR5 std The image signals are also included. Similarly, the reference images for the spectral models SP2, SP3, and SP4 are also included. std Image signal, B2 std Image signal, G2 std Image signal, R3 std Image signal, R4 std Image signal and MR5 std The image signal is included. In the same way, the reference images for the spectroscopic models SPX2, SPX3, and SPX4 are set.

[0129] The biological parameter calculation unit 70 of the second embodiment selects a specific reference image from M sets of reference images that has the smallest calculated error value based on the error from the aligned monochromatic image. Then, specific biological parameters defined by the spectral model corresponding to the specific reference image are calculated as biological parameters. Specifically, when four sets of reference images for spectral models SPX1, SPX2, SPX3, and SPX4 are used, if the reference image for spectral model SPX1 has the smallest calculated error value based on the error from the aligned monochromatic image, the reference image for spectral model SPX1 is selected as the specific reference image. In this case, the biological parameter LPX1 defined by the spectral model SPX1 is calculated as the biological parameter. That is, the oxygen saturation level S1 and the hemoglobin concentration Hb1 are calculated as biological parameters. It is preferable that the calculated error value be a value obtained by square error, as in the first embodiment.

[0130] [Third embodiment] In the third embodiment, when calculating biological parameters, instead of a spectroscopic model, multiple phantoms that imitate the spectroscopic model are used, and biological parameters that satisfy conditions are calculated from the biological parameters determined by the phantoms. In the third embodiment, M sets of reference images used to calculate the biological parameters are set for each phantom based on the emission spectrum of the illumination light, the spectral reflectance of the living body determined by spectroscopic measurement of the phantom, and the spectral sensitivity of the image sensor 36. Note that, apart from the setting of the reference images, the process is the same as in the first embodiment.

[0131] Specifically, for the phantoms required at a minimum for calculating the biological parameters, it is preferable to use phantoms corresponding to the 16 spectral models shown in FIG. 18. In this case, 16 sets of reference spectral signals are set. Furthermore, for the phantoms required to improve the accuracy of calculating the biological parameters, it is preferable to use phantoms corresponding to the 81 spectral models shown in FIG. 19. In this case, 81 sets of reference spectral signals are set. Furthermore, for the phantoms required to further improve the accuracy of calculating the biological parameters, it is preferable to use phantoms corresponding to M spectral models shown in FIG. 20. In this case, M sets of reference spectral signals are set.

[0132] For simplicity of explanation, the following description will be given using phantoms FHX1 to FHX4, some of the phantoms used to calculate biological parameters. As shown in Fig. 24, phantom FHX1 has a spectral reflectance RLX1 determined by a biological parameter LPX1. The biological parameter LPX1 includes an oxygen saturation SX1 for low oxygen saturation, a hemoglobin concentration HbX1 for high hemoglobin concentration, a medium bilirubin concentration BbX1, and a medium scattering wavelength-dependent parameter bX1.

[0133] Phantom FHX2 also has a spectral reflectance RLX2 determined by biological parameters LPX2. The biological parameters LPX2 include oxygen saturation SX2 at high oxygen saturation, hemoglobin concentration HbX2 at high hemoglobin concentration, medium bilirubin concentration BbX1, and medium scattering wavelength-dependent parameter bX1. Phantom FHX3 also has a spectral reflectance RLX3 determined by biological parameters LPX3. The biological parameters LPX3 include oxygen saturation SX3 at low oxygen saturation, hemoglobin concentration HbX3 at low hemoglobin concentration, medium bilirubin concentration BbX1, and medium scattering wavelength-dependent parameter bX1. Phantom FH4 also has a spectral reflectance RLX4 determined by biological parameters LPX4. The biological parameters LPX4 include oxygen saturation SX4 at high oxygen saturation, hemoglobin concentration HbX4 at low hemoglobin concentration, medium bilirubin concentration BbX1, and medium scattering wavelength-dependent parameter bX1. The above spectral reflectances RLX1 to RLX4 are calculated in advance by measuring a phantom with a spectroscopic measurement device. The calculated spectral reflectances RLX1 to RLX4 are stored in the extension processor device 17 in advance.

[0134] As shown in Fig. 25, the reference image setting unit 81 of the third embodiment sets a reference image for the phantom FHX1 from the emission spectrum of the illumination light, the spectral reflectance RLX1, and the spectral sensitivity of the image sensor 36. The method of setting the reference image is the same as in the first embodiment. As shown in Fig. 26, the reference image for the phantom FHX1 obtained as described above includes six B1 std Image signal, B2 std Image signal, G2 std Image signal, R3 std Image signal, R4 std Image signal and MR5 std The reference images for phantoms FHX2 to FHX4 also contain six B1 signals. std Image signal, B2 std Image signal, G2 std Image signal, R3 std Image signal, R4 std Image signal and MR5 stdThe image signals are included. In the same way, reference images for phantoms FHX2, FHX3, and FHX4 are set.

[0135] The biological parameter calculation unit 80 of the third embodiment selects a specific reference image from M sets of reference images that has the smallest calculated error value based on the error from the aligned monochromatic image. Then, specific biological parameters defined by the spectral model corresponding to the specific reference image are calculated as biological parameters. Specifically, when four sets of reference images for phantoms FHX1 to FHX4 are used, if the reference image for phantom FT1 has the smallest calculated error value based on the error from the aligned monochromatic image, the reference image for phantom FHX1 is selected as the specific reference image. In this case, the biological parameter LPX1 defined for phantom FHX1 is calculated as the biological parameter. That is, the oxygen saturation level S1 and the hemoglobin concentration Hb1 are calculated as the biological parameters. It is preferable that the calculated error value be a value obtained by square error, as in the first embodiment.

[0136] [Fourth embodiment] In the fourth embodiment, when calculating biological parameters, instead of a spectral model, multiple phantoms that mimic the spectral model are used, and each phantom is illuminated with monochromatic light and imaged to obtain a reference image. Then, among the reference images, biological parameters determined by the phantom corresponding to a reference image that satisfies the conditions are calculated as biological parameters. Therefore, in the fourth embodiment, the reference image acquisition unit 90 shown in FIG. 27 acquires M sets of reference images in advance for each phantom by illuminating the phantom with monochromatic light and imaging it. Note that the process is the same as the first embodiment except for the acquisition of the reference image.

[0137] Specifically, for the minimum number of phantoms required to calculate the biological parameters, it is preferable to use phantoms corresponding to the 16 spectral models shown in FIG. 18. In this case, 16 sets of reference spectral signals are acquired. Furthermore, for phantoms required to improve the accuracy of calculation of the biological parameters, it is preferable to use 81 phantoms corresponding to the spectral models shown in FIG. 19. In this case, 81 sets of reference spectral signals are acquired. Furthermore, for phantoms required to further improve the accuracy of calculation of the biological parameters, it is preferable to use phantoms corresponding to M spectral models shown in FIG. 20. In this case, M sets of reference spectral signals are acquired.

[0138] For simplicity, the following description will be given using phantoms FHX1 to FHX4, some of the phantoms used to calculate biological parameters. As shown in FIG. 28, phantom FHX1 is illuminated with monochromatic light sb, g, a, r, and ir, and an image is captured by the image sensor 36 for each monochromatic light illumination. This capture results in B1, G1, and R1 image signals, B2, G2, and R2 image signals, B3, G3, and R3 image signals, B4, G4, and R4 image signals, and B5, G5, and R5 image signals. Of these image signals, the B1, B2, G2, R3, and R4 image signals, and the MR5 image signal obtained by combining the B5, G5, and R5 image signals, are each used as a reference image.

[0139] That is, as shown in FIG. 26, in the case of the phantom FHX1, B1 std Image signal, B2 std Image signal, G2 std Image signal, R3 std Image signal, R4 std Image signal and MR5 std Similarly, for phantoms FHX2 to FHX4, a reference image is obtained for each phantom by illuminating the phantoms FHX2 to FHX4 with monochromatic light and capturing an image, just as in the case of phantom FHX1.

[0140] The biological parameter calculation unit 91 of the fourth embodiment selects a specific reference image from M sets of reference images that has the smallest calculated error value based on the error from the aligned monochromatic image. Then, specific biological parameters defined for the phantom corresponding to the specific reference image are calculated as biological parameters. Specifically, when four sets of reference images for phantoms FHX1 to FHX4 are used, if the reference image for phantom FHX1 has the smallest calculated error value based on the error from the aligned monochromatic image, the reference image for phantom FHX1 is selected as the specific reference image. In this case, the biological parameter LPX1 defined for phantom FHX1 is calculated as the biological parameter. That is, the oxygen saturation level S1 and the hemoglobin concentration Hb1 are calculated as the biological parameters. It is preferable that the calculated error value be a value obtained by square error, as in the first embodiment.

[0141] As the endoscope, an endoscope 12, which is a flexible endoscope for the digestive tract, or an endoscope that is a rigid endoscope for laparoscopy may be used. When a rigid endoscope is used, an endoscope system 200 shown in Fig. 29 is used. The endoscope system 200 includes an endoscope 101, a light source device 13, a processor device 14, a display 15, a processor-side user interface 16, an extended processor device 17, and an extended display 18. In the following, parts of the endoscope system 100 that are common to the endoscope system 10 will be omitted, and only the differences will be described.

[0142] 30, the imaging unit 103 separates light from the endoscope 101 into light of multiple wavelength bands and acquires image signals based on the separated wavelength bands. The imaging unit 103 includes dichroic mirrors 105, 106, and 107 and monochrome imaging sensors 110, 111, 112, and 113. The dichroic mirror 105 reflects light in the blue band of the light reflected from the endoscope 101 and transmits light with longer wavelengths than the light in the blue band. The light in the blue band reflected by the dichroic mirror 105 is incident on the imaging sensor 110. Monochromatic light sb, the blue band portion of the monochromatic light g, and monochromatic light ir are incident on the imaging sensor 110.

[0143] Dichroic mirror 106 reflects light in the green wavelength range out of the light that has passed through dichroic mirror 105, and transmits light with longer wavelengths than the light in the green wavelength range. The light in the green wavelength range reflected by dichroic mirror 106 is incident on image sensor 111. The light in the green wavelength range out of the monochromatic light g and the monochromatic light ir are incident on image sensor 111. On the other hand, the light that has passed through dichroic mirror 106 is incident on image sensor 112. The monochromatic light r and the monochromatic light ir are incident on image sensor 112.

[0144] In normal mode, the image sensors 110, 111, and 112 output Bc, Gc, and Rc image signals in response to the incidence of monochromatic light b, monochromatic light g, and monochromatic light r, respectively. In oxygen saturation mode, in odd-numbered frames (frames 1, 3, 5, 7, and 9), the image sensors 110, 111, and 112 output Bc, Gc, and Rc image signals in response to the incidence of monochromatic light b, monochromatic light g, and monochromatic light r, respectively. Meanwhile, in even-numbered frames in oxygen saturation mode, in frame 2, the image sensor 110 outputs a B1 image in response to the incidence of monochromatic light sb. In frame 4, the image sensor 110 outputs a B2 image signal in response to the incidence of light in the blue band of the monochromatic light g, and the image sensor 111 outputs a G2 image signal in response to the incidence of monochromatic light g.

[0145] In frame 6, the image sensor 112 outputs an R3 image signal in response to the incidence of monochromatic light a. In frame 8, the image sensor 112 outputs an R4 image signal in response to the incidence of monochromatic light r. In frame 10, the image sensor 110 outputs a B5 image signal in response to the incidence of monochromatic light ir, the image sensor 111 outputs a G5 image signal in response to the incidence of monochromatic light ir, and the image sensor 112 outputs an R5 image signal in response to the incidence of monochromatic light ir. These three image signals, B5, G5, and R5, are combined to generate an MR5 image signal.

[0146] When using an endoscope that is a rigid endoscope for laparoscopy, an endoscope system 200 that images an object to be observed using another imaging method may be used instead of the endoscope system 100 that images an object to be observed using three monochrome image sensors 110-112. As shown in FIG. 31 , the endoscope system 200 uses a one-sensor type abdominal endoscope 201 that has one color image sensor 203. The image sensor 203 is provided in an imaging unit 205 of the endoscope 201. The spectral sensitivity of the image sensor 203 is the same as that of the image sensor 36. The rest is the same as that of the endoscope system 100.

[0147] In the above embodiment, the aligned monochromatic images are acquired by sequentially switching between monochromatic lights and capturing images with the image sensor 36. However, instead, the aligned monochromatic images may be acquired by illuminating with broadband illumination light and capturing images with a spectroscopic image sensor that disperses the broadband illumination light into monochromatic lights. Specifically, as shown in FIG. 32, the broadband illumination light has a wavelength range of 300 nm to 1000 nm. As the spectroscopic image sensor, a snapshot mosaic hyperspectral image sensor is preferably used.

[0148] In this case, as shown in Fig. 33, the spectral image sensor 300 has a color filter array, each consisting of nine pixels (3 pixels vertically and 3 pixels horizontally), arranged in tiles. In the color filter array, pixels are arranged with color filters having transmission bands with center wavelengths of 450 nm, 470 nm, 500 nm, 540 nm, 620 nm, 690 nm, and 850 nm. In Fig. 33, "450 nm, 470 nm, 500 nm, 540 nm, 620 nm, 690 nm, and 850 nm" represent pixels with color filters having the respective transmission band center wavelengths. By capturing an image of an object illuminated with broadband illumination light using the spectroscopic imaging sensor 300, the spectroscopic imaging sensor 300 outputs a b image signal from the 450 nm pixel, an sb image signal from the 470 nm pixel, an sg image signal from the 500 nm pixel, an lg image signal from the 540 nm pixel, an a image signal from the 620 nm pixel, an r image signal from the 60 nm pixel, and an ir image signal from the 850 nm pixel.

[0149] As described above, when image signals are output from the spectroscopic imaging sensor 300, each pixel has its own image signal, and no image signals are output from other pixels, resulting in a discrete distribution of image signals. To address this, demosaic processing, which is an interpolation process between pixels, is performed to compensate for the pixel signals of other pixels. As a result, each pixel has image signals for all wavelength ranges (b image signal, sb image signal, sg image signal, sl image signal, a image signal, r image signal, and ir image signal). For demosaic processing, it is preferable to use the method described in JP-A-2023-529189, for example.

[0150] Regarding the image signal after demosaicing, the b image signal is B1 img The sb image signal is treated as B2 img The image signal that combines the sg and sl image signals is treated as a G2 image signal. img The image signal is treated as an R3 img The image signal is treated as R4 imgThe IR image signal is treated as an MR5 img The image signal is treated as an image signal, and the innate parameters are calculated in the same manner as in the above embodiment.

[0151] In the above embodiment, monochromatic light is emitted as the first illumination light while polychromatic light is emitted as the second illumination light in the oxygen saturation mode. However, other light may be emitted. For example, as shown in Fig. 34, when monochromatic light b, monochromatic light g, and monochromatic light a are simultaneously emitted as the second illumination light in frames 1, 3, and 5, monochromatic light sb and monochromatic light a may be simultaneously emitted as the first illumination light in frame 2, and monochromatic light g and monochromatic light r may be simultaneously emitted as the first illumination light in frame 4. Frame 6 emits only monochromatic light ir as the first illumination light.

[0152] In this case, in frame 2, the image signal output from the B pixel of the image sensor 36 is B1 img The image signal output from the R pixel is taken as R3 img In frame 4, the image signal output from the B pixel of the image sensor is designated as B2 img The image signal output from the G pixel is G2 img The image signal output from the R pixel is taken as R2 img For frame 6, as with frame 10 in the above embodiment, the image signals output from the B pixels, G pixels, and R pixels of the image sensor 36 are respectively designated as B5 img Image signal, G5 img Image signal, R5 img As described above, when the first illumination light is emitted, the number of frames can be reduced to six frames (frames 1 to 6).

[0153] In the above embodiment, the hardware structure of processing units that perform various processes, such as the image acquisition unit 60, the registration processing unit 61, the motion amount calculation unit 64, the motion amount correction unit 65, the biological parameter calculation unit 62, the display control unit 63, the reference image setting unit 66, the biological parameter calculation unit 70, the reference image setting unit 71, the biological parameter calculation unit 80, the reference image setting unit 81, the reference image acquisition unit 90, and the biological parameter calculation unit 91, is various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a GPU (Graphical Processing Unit), a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacturing such as an FPGA (Field Programmable Gate Array), and a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for performing various processes.

[0154] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (e.g., multiple FPGAs, a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). Multiple processing units may also be configured with a single processor. Examples of multiple processing units configured with a single processor include a multi-layered configuration, such as a client or server computer, where one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units. A mono-layered configuration, such as a system-on-chip (SoC), uses a processor that realizes the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip. In this way, the various processing units are configured with one or more of the above-mentioned various processors as a hardware structure.

[0155] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit formed by combining circuit elements such as semiconductor elements, and the hardware structure of the memory unit is a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). [Explanation of symbols]

[0156] 10 Endoscopy System 12 Endoscopy 12a Insertion part 12b Operation section 12c curved section 12d Tip 12e Angle Knob 12f Mode switch 12h Still image acquisition command switch 12i Zoom control 12j forceps mouth 13 Light source device 14 Processor unit 15 Display 16 Processor-side user interface 17 Extended Processor Unit 18 Extended Display 19 Scope-side user interface 20 Light source section 21 Light Source Processor 20a b-LED 20b sb-LED 20c g-LED 20d a-LED 20e r-LED 20f ir-LED 20g v-LED 23 Optical path coupling section 25 Light Guide 30 Illumination optical system 31 Imaging optical system 32 Lighting lens 35 objective lens 36 imaging sensors 37 Imaging processor 40 CDS / AGC circuit 41 A / D converter 45 DSP 50 Image processing section 51 Image Communication Department 52 Display control unit 53 Central Control Unit 54a Normal observation mode 54b Oxygen Saturation Mode 55a, 55b curve 60 Image acquisition unit 61 Alignment processing unit 62 Biological parameter calculation unit 63 Display control unit 64 Motion amount calculation unit 65 Motion compensation section 66 Reference image setting section 70 Biological parameter calculation unit 71 Reference image setting section 80 Biological parameter calculation unit 81 Reference image setting section 90 Reference image acquisition unit 91 Biological parameter calculation unit 100 Endoscopy System 101 Endoscope 103 Imaging unit 105, 106 Dichroic mirror 110, 111, 112 Image sensor 200 Endoscopy System 201 Endoscope 203 Image Sensor 205 Imaging unit 300 Spectroscopic imaging sensor FHX1 Phantom FHX2 Phantom FHX3 Phantom FHX4 Phantom NP white light image NP1~NP6 White light image MP1 First monochrome image MP2 Second monochrome image MP3 3rd monochrome image MP4 4th monochrome image MP5 5th monochrome image AMP1 1st aligned monochrome image AMP2 Second aligned monochrome image AMP3 1st aligned monochrome image AMP4 4th aligned monochrome image AMP5 5th aligned monochrome image SPX1 Spectroscopic Model SPX2 Spectroscopic Model SPX3 Spectroscopic Model SPX4 Spectroscopic Model OP oxygen saturation image RLX1 spectral reflectance RLX2 spectral reflectance RLX3 spectral reflectance RLX4 spectral reflectance GOP Internal gastrointestinal oxygen saturation image SOP Serosal oxygen saturation image BF B color filter GF G color filter RF R color filter Bc Bc image signal Gc Gc image signal Rc Rc image signal ST010~ST100 steps

Claims

1. a light source device that alternately emits a plurality of first illumination lights and a second illumination light having a broader band than the first illumination lights; an endoscope having an image sensor that captures an image of an observation target illuminated by the illumination light; a processor device having a processor that generates a biological parameter image based on the endoscopic image obtained by the imaging sensor and controls displaying the biological parameter image on a display; the display, the plurality of first illumination lights are light beams in wavelength bands that have different dependences on oxygen saturation level, The processor: generating an aligned first image by performing a registration process on a first endoscopic image obtained by imaging the object illuminated with the first illumination light based on a plurality of second endoscopic images obtained by imaging the object illuminated with the second illumination light; obtaining an aligned first image set including a plurality of the aligned first images; Calculating an oxygen saturation level of the object based on the acquired aligned first image set; An endoscope system that generates the biological parameter image based on the oxygen saturation.

2. the alignment process includes a motion amount calculation process and a motion amount correction process, the motion amount calculation process is a process of calculating a motion amount of the first endoscope image based on a plurality of the second endoscope images, The endoscope system according to claim 1 , wherein the motion amount correction process is a process of generating the aligned first image by performing correction based on the motion amount on the first endoscope image for which the motion amount has been calculated.

3. The endoscope system according to claim 1 , wherein the light source device alternately emits the second illumination light and any one of the plurality of first illumination lights for each frame.

4. The endoscope system according to claim 1 , wherein the central wavelength of each of the plurality of first illumination lights is any one of 450 nm, 470 nm, 540 nm, 620 nm, 690 nm, and 850 nm.

5. The endoscope system according to claim 1 , wherein there are five first illumination lights, and the central wavelengths of the lights are 470 nm, 540 nm, 620 nm, 690 nm, and 850 nm, respectively.

6. the light source device illuminates the observation object with light-emitting units that emit all of the plurality of first illumination lights in a predetermined order, The endoscopic system of claim 1 , wherein the processor generates the aligned first image set based on a plurality of the first endoscopic images obtained in the light-emitting unit.

7. The endoscope system according to claim 6 , wherein the light source device illuminates the observation object with the light-emitting units that emit the plurality of first illumination lights in ascending order of center wavelength.

8. The endoscope system according to claim 1 , wherein the second illumination light is generated by emitting two or more of the first illumination lights out of the plurality of first illumination lights.

9. The endoscope system according to claim 1 , wherein the second illumination light is obtained by simultaneously emitting the first illumination light beams having central wavelengths of 450 nm, 540 nm, and 620 nm from among the plurality of first illumination light beams.

10. The endoscopic system according to claim 2, wherein the motion amount calculation process is a process of calculating the motion amount of the first endoscopic image based on two second endoscopic images obtained in frames immediately before and after the first endoscopic image.

11. The endoscope system of claim 1 , wherein the processor calculates the oxygen saturation of the object for each pixel of the aligned first image included in the aligned first image set.

12. a normal observation mode in which the light source device continuously emits the second illumination light and the processor continuously displays the second endoscopic image on the display; a biological parameter image acquisition mode in which the light source device alternately emits the second illumination light and any one of the plurality of first illumination lights, and the processor controls the display to display the biological parameter image; The endoscope system according to claim 1 , wherein the endoscope has a switching operation unit for switching between the normal observation mode and the biological parameter image acquisition mode, the switching operation unit being operated by an endoscope operator.

13. When the endoscope operator operates the switching operation unit to switch from the normal observation mode to the biological parameter image acquisition mode, The processor: generating the biological parameter image based on the first aligned image set acquired first after the switching; Controlling the display of the generated biological parameter image on the display; The endoscope system according to claim 12, wherein the endoscope system then automatically returns to the normal observation mode.

14. the light source device illuminates the observation object with light-emitting units that emit all of the plurality of first illumination lights in a predetermined order, The processor: calculating a total amount of movement by adding up the amount of movement in each of the plurality of first endoscope images obtained in the light emission unit; When the total amount of movement is within a preset range, the aligned first image set including the aligned first images obtained in the light-emitting unit is acquired; The endoscope system according to claim 2 , wherein the oxygen saturation level is calculated based on the acquired aligned first image set.

15. the light source device illuminates the observation object with light-emitting units that emit all of the plurality of first illumination lights in a predetermined order, The processor: calculating a total amount of movement by adding up the amount of movement in each of the plurality of first endoscope images obtained in the light emission unit; The endoscopic system according to claim 2, wherein the resolution is adjusted to be reduced so that the greater the total amount of movement, the greater the degree of reduction in the resolution of the aligned first image of the object for which the oxygen saturation of the object to be observed is calculated.

16. A method for generating a biological parameter image executed in an endoscope system including a light source device, an endoscope, and a processor device, comprising: the light source device alternately emits, as illumination light, any one of a plurality of first illumination lights and a second illumination light having a broader band than the first illumination lights; the endoscope has an image sensor that captures an image of an observation target illuminated by the light source device, the processor device has a processor that generates a biological parameter image based on the endoscopic image obtained by the imaging sensor, the plurality of first illumination lights are light beams in wavelength bands that have different dependences on oxygen saturation level, the processor: generating an aligned first image by performing a registration process on a first endoscopic image obtained by imaging the object illuminated with the first illumination light, based on a plurality of second endoscopic images obtained by imaging the object illuminated with the second illumination light; acquiring an aligned monochromatic image set comprising a plurality of said aligned first images; calculating an oxygen saturation level of the object based on the aligned first image set; generating the biological parameter image based on the oxygen saturation.

17. a light source device that alternately emits a plurality of first illumination lights and a second illumination light having a broader band than the first illumination lights; an endoscope having an image sensor that captures an image of an observation target illuminated by the illumination light; a processor device having a processor that generates a biological parameter image based on an endoscopic image obtained by the imaging sensor and controls displaying the biological parameter image on a display, the processor, generating an aligned first image by performing a registration process on a first endoscopic image obtained by imaging the object illuminated with the first illumination light based on a plurality of second endoscopic images obtained by imaging the object illuminated with the second illumination light; obtaining an aligned first image set including a plurality of the aligned first images; Calculating an oxygen saturation level of the object based on the first aligned image set; generating a biological parameter image based on the oxygen saturation; a program for executing a function of controlling the display of the biological parameter image on the display;

Images