Image Calibration Device
The image calibration device for endoscopes addresses image quality issues by calculating and applying correction parameters for positional and brightness differences, enhancing image quality and reducing artifacts in combined images.
Patent Information
- Application Number
- JP2021032945
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-03-02
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2041-03-02
AI Technical Summary
Existing endoscope systems face issues with image quality deterioration and artifacts when combining two images due to inadequate correction parameters, inappropriate chart placement, and polarization or wavelength dependence, especially in varying observation modes.
An image calibration device that includes an optical path splitter, image sensor, and calibration processor to calculate and apply correction parameters for positional and brightness differences between two images, using specific charts and illumination methods to ensure optimal alignment and color tone matching.
The device reduces image quality deterioration and artifacts by obtaining optimal correction parameters in advance, ensuring high-precision alignment and consistent brightness and color tone across different observation modes.
Smart Images

Figure 0007582882000001 
Figure 0007582882000002 
Figure 0007582882000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an image calibration device, for example, to an image calibration device for endoscopic image synthesis used in the medical and industrial fields. [Background technology]
[0002] Endoscopes are devices that are widely used in the medical and industrial fields. In particular, in the medical field, images obtained by an endoscope inserted into a body cavity are used for diagnosis and treatment of the observed site.
[0003] It is generally known that in devices equipped with an image sensor, such as an endoscope system, the depth of field narrows as the pixel count of the image sensor increases. That is, when the pixel pitch (the vertical and horizontal dimensions of one pixel) is reduced in order to increase the number of pixels in the image sensor, the permissible circle of confusion also becomes smaller accordingly, and the depth of field of the imaging device narrows.
[0004] The following Patent Documents 1, 2, 3, and 4 propose a depth expanding endoscope that simultaneously acquires two images with different focuses and combines the images.
[0005] There are endoscopes that split the image from the endoscope objective optical system into two using a polarizing beam splitter (PBS) and synthesize the two split images to expand the depth, expand the dynamic range, and improve resolution. In this case, as a pre-processing before the image synthesis process, a calibration process is performed to geometrically align the two images and also correct the brightness and color tone to be approximately the same. [Prior art documents] [Patent documents]
[0006] [Patent Document 1] International Publication No. 2013 / 027459 [Patent Document 2] International Publication No. 2013 / 061819 [Patent Document 3] International Publication No. 2014 / 002740 [Patent Document 4] International Publication No. 2016 / 043107 Summary of the Invention [Problem to be solved by the invention]
[0007] Patent Document 1 proposes an endoscope system with magnified depth that uses image processing to correct differences in position, magnification, rotation, color tone, and brightness between two images and synthesizes the images. Patent Documents 2 and 3 propose an endoscope system that stores correction values as unique correction parameters, and when a target endoscope is connected to an image processor, reads out the correction parameters and performs real-time correction. Patent Document 4 proposes an endoscope system that switches correction parameters according to the observation mode of the endoscope so that an optimal synthetic depth of field can be obtained even in narrowband observation.
[0008] Patent Documents 1, 2, 3, and 4 do not disclose a specific method or configuration for acquiring the correction parameters themselves. For this reason, if optimal correction is not performed when combining two images, there is a problem that the image quality of the combined image may deteriorate or artifacts (noise) may occur.
[0009] In aligning the two images, a chart for alignment is placed on the endoscope objective optical system, images are captured, and calibration is performed, but if the chart is placed at an inappropriate observation distance, precise alignment cannot be performed. In addition, when correcting brightness and color tone, if an inappropriate chart or light source is used, correct calibration cannot be performed due to polarization dependence and wavelength dependence, and artifacts are likely to occur in the composite image. Furthermore, endoscope systems have various observation modes, and since the characteristics of brightness and color tone differ for each mode, there is a problem that switching the observation mode can deviate from the optimal correction state.
[0010] The present invention has been made in consideration of these problems, and aims to provide an image calibration device that reduces degradation in image quality and artifacts (noise) in the synthesized image by obtaining optimal correction parameters in advance when synthesizing two images in an endoscopic system. [Means for solving the problem]
[0011] In order to solve the above-mentioned problems and achieve the object, an image calibration device according to at least some embodiments of the present invention includes, in order from the object side, an objective optical system, and a subject image obtained by the objective optical system, which is arranged on an optical path. long and an image sensor that acquires two images from the two optical images, the image calibration device comprising: The first chart, The second chart, The third chart, a light source for irradiating the first chart, the second chart, and the third chart with illumination light; a stage on which the first chart, the second chart, and the third chart are placed; A calibration processor; The calibration processor a first calculation unit that calculates a spatial frequency characteristic value of each of first two images obtained by capturing an image of the first chart; a second calculation unit that calculates a difference in a relative positional relationship between two second images obtained by capturing images of the second chart with an optical device in a state in which the second chart is disposed at a position of the first chart where the spatial frequency characteristic values of the two first images are equivalent; a third calculation unit that calculates a difference in relative brightness ratio between two third images obtained by capturing images of a third chart illuminated by a light source using an optical device, for each RGB signal; The parameter calculation unit calculates a first correction parameter for correcting the difference in the positional relationship and a second correction parameter for correcting the difference in the brightness ratio. Effect of the Invention
[0012] The present invention has the advantage of providing an image calibration device that reduces degradation of image quality and artifacts (noise) in the synthesized image by obtaining optimal correction parameters in advance when synthesizing two images in an endoscopic system. [Brief description of the drawings]
[0013] [Figure 1] 1 is a perspective view of an image calibration device according to an embodiment of the present invention; [Diagram 2] FIG. 1 is a diagram illustrating a configuration of an image calibration device according to an embodiment. [Diagram 3] 1 is a diagram showing a cross-sectional configuration of an objective optical system, an optical path splitting unit, and an image pickup element of an optical device installed in an image calibration device (normal observation state). [Figure 4] 2 is a schematic diagram illustrating a depolarizing plate, an optical path splitting section, and an image sensor included in the optical device. FIG. [Diagram 5] FIG. 2 is a schematic diagram illustrating the configuration of an image pickup element included in the optical device. [Figure 6] 4 is a flowchart showing a calibration process of the image calibration device according to the embodiment. [Figure 7] 1A is a diagram showing a first chart, and FIG. 1B is a diagram showing two images of the first chart captured by an imaging element. [Figure 8] FIG. 13 is a diagram showing spatial frequency characteristic values of two images of the first chart. [Figure 9] FIG. 2 is another perspective configuration diagram of the image calibration device according to the embodiment. [Figure 10] FIG. 13 is a diagram showing a second chart. [Figure 11] FIG. 13 is a diagram showing two images of a second chart captured by an imaging element. [Figure 12] FIG. 2 is another perspective configuration diagram of the image calibration device according to the embodiment. [Figure 13]FIG. 13 is a diagram showing two images of a third chart captured by an imaging element. [Figure 14] FIG. 13 is a diagram showing the spectral distribution characteristics of a third chart. [Figure 15] FIG. 1 is a functional block diagram showing a configuration of an endoscope system.
[0014] An image calibration device according to an embodiment will be described in detail below with reference to the drawings. Note that the present invention is not limited to the embodiment.
[0015] Fig. 1 is a perspective view of the configuration of an image calibration device 100 according to an embodiment of the present invention. Fig. 2 is a diagram showing another configuration of the image calibration device 100.
[0016] The image calibration device 100 is an image calibration device 100 for an optical device 200 having, in order from the object side, an objective optical system OBL (FIG. 3), an optical path splitting section 120 that splits a subject image obtained by the objective optical system OBL into two optical images with different optical path differences, and an image pickup element 122 that acquires two images from the two optical images. The optical device 200 is included in an endoscope system 1 (FIG. 15) that can extend the depth of field. The configurations of the endoscope system 1 and the optical device 200 will be described later.
[0017] As shown in FIG. 2, the image calibration device 100 includes a first chart 101 (FIG. 7), a second chart 102 (FIG. 10), a third chart 103 (FIG. 2), a light source 115 that irradiates illumination light to these charts 101, 102, and 103 via a light guide path 116, a stage 104 on which the first chart 101, the second chart 102, and the third chart 103 are placed, and a calibration processor 108 (FIG. 2).
[0018] The processor device 4, which will be described later, has a light source 115. The light source 115 supplies light for illuminating the first, second, and third charts 101, 102, and 103. The light from the light source 115 is guided by a light guide path 116 in the direction in which the first chart 101, the second chart 102, and the third chart 103 are placed.
[0019] The light guide 116 can be composed of an optical fiber or the like. The first chart 101 and the second chart 102 are transmissive charts. The light emitted from the light guide 116a illuminates the first chart 101 or the second chart 102 from the left side of the paper. The light emitted from the light guide 116b is guided into the third chart 103 (integrating sphere).
[0020] For example, light from a light source 115 is guided through a light guide path 116 to light up the backlight. The guided light illuminates a first chart 101 or a second chart 102 of a transmissive type that is provided alongside the backlight.
[0021] The optical device 200 captures images of a first chart 101 and a second chart 102 illuminated by a backlight.
[0022] As for third chart 103 (integrating sphere), illumination light guided from light source 115 through light guide path 116b illuminates the inside of the integrating sphere.
[0023] Optical device 200 captures an image of the inside of the integrating sphere. As a modified example, light source 115 may be separately installed externally. Furthermore, illumination light from light source device 3 (FIG. 15) for an endoscope illumination optical system included in endoscope system 1 may be used as light source 115.
[0024] 2, the calibration processor 108 includes a first calculation unit 110, a second calculation unit 111, a third calculation unit 112, and a parameter calculation unit 113.
[0025] The first calculation unit 110 calculates the spatial frequency characteristic values of each of the first two images 101a and 101b (FIG. 7(b)) obtained by capturing an image of the first chart 101.
[0026] Next, the second calculation unit 111 calculates a difference in the relative positional relationship between the two second images 102a and 102b (FIG. 11) obtained by capturing the second chart 102 with the optical device 200 in a state where the second chart 102 is placed at a position L where the spatial frequency characteristic values of the first two images 101a and 101b are equivalent.
[0027] The third calculation unit 112 calculates the difference in relative brightness ratio for each RGB signal between two third images 103a, 103b (Figure 13) obtained by capturing an image of a third chart 103 illuminated by a light source using an optical device 200.
[0028] The parameter calculation unit 113 calculates a first correction parameter for correcting the difference in the positional relationship between the two images 102a and 102b, and a second correction parameter for correcting the difference in the brightness ratio.
[0029] First, a description will be given of an example of an optical device 200 that is a target for calibration by the image calibration device 100. Fig. 3 is a diagram showing a cross-sectional configuration of an objective optical system OBL, an optical path splitting unit 120, and an image sensor 122 included in the optical device 200 installed in the image calibration device 100 (normal observation state).
[0030] The objective optical system OBL is made up of three groups: a first group G1, a second group G2, and a third group G3, each having lenses L1-L10.
[0031] The optical device 200 has, in order from the object side, an objective optical system OBL, a depolarization plate 121a, an optical path splitting section 120 that splits light from the objective optical system OBL into two, and an image sensor 122 that captures the two split images. The depolarization plate 121a is disposed in the optical path between the objective optical system OBL and the optical path splitting section 120. The depolarization plate 121a has a function of eliminating polarization.
[0032] FIG. 4 is a schematic diagram of the depolarizing plate 121a, the optical path splitting section 120, and the imaging element included in the optical device 200. As shown in FIG.
[0033] The light emerging from the objective optical system OBL passes through a depolarization plate 121a and enters the optical path splitting section 120. The depolarization plate 121a has a function of depolarizing the light with a simple configuration, as will be described later with reference to FIG.
[0034] The optical path splitting section 120 has a polarizing beam splitter 121 that splits the subject image into two optical images with different focuses, and an image pickup element 122 that captures the two optical images to obtain two images.
[0035] 5, the polarizing beam splitter 121 includes an object-side prism 121b, an image-side prism 121e, a mirror 121c, and a λ / 4 plate 121d. The object-side prism 121b and the image-side prism 121e both have beam splitting surfaces that are inclined at 45 degrees with respect to the optical axis AX.
[0036] A polarization splitting film 121f is formed on the beam splitting surface of the object-side prism 121b. The object-side prism 121b and the image-side prism 121e form a polarization beam splitter 121 by abutting their beam splitting surfaces with the polarization splitting film 121f interposed between them.
[0037] The mirror 121c is provided near the end face of the prism 121b on the object side via a λ / 4 plate 121d. The image sensor 122 is attached to the end face of the prism 121e on the image side via a cover glass CG. I is an image forming surface (imaging surface).
[0038] The subject image from the objective optical system OBL is separated into a P-polarized component (transmitted light) and an S-polarized component (reflected light) by a polarization separation film 121f provided on the beam split surface of the object-side prism 121b, and is separated into two optical images: an optical image on the reflected light side and an optical image on the transmitted light side.
[0039] The optical image of the S-polarized component is reflected by the polarization separation film 121f to the side opposite the image sensor 122, passes through the A optical path, passes through the λ / 4 plate 121d, is reflected by the mirror 121c, and is folded back towards the image sensor 122. The folded optical image passes through the λ / 4 plate 121d again, whereby the polarization direction of the image is rotated by 90°, and passes through the polarization separation film 121f to be formed on the image sensor 122.
[0040] The optical image of the P-polarized component passes through the polarization separation film 121f, passes through the B optical path, and is reflected by a mirror surface provided on the opposite side to the beam splitting surface of the image-side prism 121e that turns back vertically toward the image sensor 122, and is imaged on the image sensor 122. At this time, the prism glass path is set so that a predetermined optical path difference of, for example, about several tens of μm occurs between the A optical path and the B optical path, and two optical images with different focuses are imaged on the light receiving surface of the image sensor 122.
[0041] In other words, the object-side prism 121b and the image-side prism 121e are arranged so that the optical path length on the reflected light side is shorter (smaller) than the optical path length (glass path length) on the transmitted light side leading to the image sensor 122 in the object-side prism 121b, so that the object-side prism 121b and the image-side prism 121e can separate the subject image into two optical images with different focus positions.
[0042] In addition, the optical path length (glass path length) of each of the two optical images is not limited to the configuration of this embodiment as long as an arbitrary optical path length difference can be obtained. For example, the optical path length of the reflected light side in the object-side prism 121b to the image sensor 122 may be arranged to be longer (larger) than the optical path length (glass path length) of the transmitted light side.
[0043] Fig. 5 is a schematic diagram of the image sensor 122. As shown in Fig. 5, the image sensor 122 has two light receiving regions (effective pixel regions) 122a and 122b in the entire pixel region of the image sensor 122 in order to individually receive and capture two optical images having different focus positions.
[0044] FIG. 6 is a flowchart showing the calibration process of the image calibration device 100 according to the embodiment.
[0045] In step S1, the first chart 101 is placed in a holder 106 on a stage 104. FIG.
[0046] The first chart 101 is a chart for determining spatial frequency characteristic values, and has a chart pattern including a black-and-white edge ridge 101bk. The first chart 101 is arranged so that the visual field centers 122ac and 122bc of the two images 101a and 101b intersect with the black-and-white edge ridges 101abk and 101bbk, respectively.
[0047] In general, for evaluating spatial frequency characteristic values (here, synonymous with resolution), visual resolution, limit resolution (SIN wave chart), SFR (Spatial Frequency Response) chart, etc. defined in ISO12233 are used. In this embodiment, it is particularly preferable that the first chart 101 is an SFR chart. The reason for this will be described below.
[0048] In order to obtain the observation distance L of the second chart 102 described later, it is necessary to obtain spatial frequency characteristic values for each observation distance at an arbitrary step width in each of the A and B optical paths. When using a chart for evaluating the visual resolution or the limit resolution, it is necessary to prepare a chart with a different pitch pattern for each observation distance or to consider a correction coefficient according to the observation magnification. In that case, it is not preferable because the configuration of the image calibration device 100 becomes complicated and the cost becomes high. For this reason, in this embodiment, an SFR chart that is not affected by the observation magnification is used.
[0049] In step S2, the image sensor 122 captures an image 101a that is an image focused on a near point (hereinafter referred to as a "near point image") and an image sensor 122 captures an image 101b that is an image focused on a far point (hereinafter referred to as a "far point image").
[0050] The image calibration device 100 also includes a drive unit 114 and a storage unit 109 .
[0051] An endoscope with an extended depth of field can switch between a normal focus (observing a distant object) mode and a close focus (observing a close object) mode by driving a focus lens (the second group G2 in FIG. 3). In this embodiment, it is preferable to perform calibration in the normal focus mode. This is because when performing calibration in the close focus mode, the observation distance setting between the first chart and the optical device 200 needs to be relatively more accurate than in the normal focus mode, and the actuator and control software tend to be expensive.
[0052] The driving unit 114 changes the observation distance of the first chart 101 relative to the stage 104 and the optical device 200 in any step (step S5). The driving unit 114 is, for example, a motor or an actuator.
[0053] Note that it is only necessary to change the observation distance of the first chart 101 relative to the optical device 200. For this reason, the position of the optical device 200 held by the holder 107 may be changed by driving the stage 105 instead of the stage 104.
[0054] The storage unit 109 stores a still image of the first chart 101 for each viewing distance. The first calculation unit 110 calculates the spatial frequency characteristic value of each of the first two images calculated from the still images of the first chart 101 stored in the storage unit 109. The first calculation unit 110 calculates the viewing distance L at which the spatial frequency characteristic value is the same for the two images 101a, 101b (steps S3, S4).
[0055] 8 shows the spatial frequency characteristic values of two images 101a (near point image, plotted with triangles) and image 101b (far point image, plotted with circles) of the first chart 101. The first calculation unit 110 calculates the observation distance L at which the spatial frequency characteristic values of the two images 101a and 101b are the same. The horizontal axis represents the defocus amount (unit: mm), and the vertical axis represents the spatial frequency characteristic value (unit: %). The spatial frequency characteristic value is found by SFR (Spatial Frequency Response).
[0056] The second chart 102 is placed on the holder 106 of the stage 104 at the observation distance L calculated by the first calculation unit 110 (step S6). If the first chart 101 and the second chart 102 are placed on the same plane of the chart substrate, there is no need to place the second chart 102 on the holder 106 again. The storage unit 109 stores the still images 102a, 102b of the second chart (step S7). The parameter calculation unit 113 calculates a first correction parameter based on the still images 102a, 102b of the second chart and the difference in positional relationship calculated by the second calculation unit 111 (steps S8, S9).
[0057] In the optical device 200, when performing image synthesis according to the spatial frequency characteristic value of the images, it is necessary to perform alignment correction of the two images with high accuracy. Here, if one of the two images has a low spatial frequency characteristic value (i.e., is out of focus), the accuracy of the alignment correction decreases. As a result, the resolution of the synthesized image deteriorates and artifacts (noise) occur, which is undesirable.
[0058] In contrast, in this embodiment, an observation distance L is calculated at which the spatial frequency characteristic values of the two images 101a, 101b of the first chart 101 are the same. Then, the second chart 102 for alignment is placed at the calculated observation distance L. This has the effect of making it possible to obtain a correction parameter for alignment (first correction parameter).
[0059] Fig. 9 shows a configuration in which a second chart is arranged in the image calibration device 100. Fig. 10 is a diagram showing a second chart 102. Fig. 11 is a diagram showing two images 102a and 102b of the second chart captured by the optical device 200.
[0060] The second chart 102 is a grayscale checker pattern in which areas having at least two or more different gradations are randomly arranged.
[0061] A typical black and white pattern chart uses edge matching using only two gradations. In contrast, this embodiment uses a pattern in which multiple gradations are randomly arranged. Specifically, an error function of the luminance difference of multiple gradation patterns between the two images 102a and 102b of the second chart 102 is calculated. Then, the calculated value of the error function is minimized to align the two images. This enables highly accurate alignment.
[0062] In a preferred aspect of this embodiment, the parameter calculation unit 113 calculates the first correction parameter using one of the second two images 102a, 102b as a reference.
[0063] The purpose of the calibration in this embodiment is to avoid artifacts and overlapping images in a composite image by relatively eliminating the difference between the two images. For this reason, obtaining the absolute value of the difference between the two images does not have much meaning. It is very difficult to derive the absolute value of the difference between the two images (correct coordinate value) considering the measurement means and manufacturing errors, and even if it is possible to set the reference parameters, the alignment algorithm is very complicated and large in scale, which is not preferable. Therefore, in this embodiment, it is desirable to use one of the two images as the reference image.
[0064] In addition, in a preferred embodiment of the present invention, the third chart 103 is an integrating sphere. The spectral characteristics of the illumination light guided to the integrating sphere are the same as those when the optical device 200 is used.
[0065] Fig. 12 is another perspective view of the third chart 103 (integrating sphere) of the image calibration device 100 according to this embodiment. Fig. 13 is a diagram showing two images 103a and 103b of the third chart 103 captured by the optical device 200.
[0066] When correcting the relative brightness difference between the two images, it is desirable to perform the correction using illumination light having characteristics equivalent to the light source characteristics (spectral characteristics) used in the endoscope system 1 product. For example, if a commercially available LED illumination device is used, it will have a completely different color tone and brightness from the endoscope light source, and will deviate from the optimal second correction parameters. Even if further correction of color tone and brightness is performed in the image processing unit, this is not preferable because it will increase the circuit size and increase image noise.
[0067] The inner surface of the integrating sphere, which is the third chart 103, corresponds to a perfect diffusion surface. Therefore, the light emitted from the integrating sphere has almost no polarization dependency. In optical device 200 using a polarizing beam splitter in optical path splitting section 120, when correcting color tone and brightness, if a subject having polarization dependency is used as a reference (collision), uniform image division is not possible. For this reason, correct correction is difficult for the two images. Therefore, in this embodiment, an integrating sphere with extremely little polarization dependency is used as the subject (step S10).
[0068] As described above, the third calculation unit 112 calculates the difference in relative brightness ratio between the two images 103a, 103b of the third chart 103 obtained by capturing the third chart 103 illuminated by a light source using the optical device 200, for each RGB signal (step S11).
[0069] The parameter calculation unit 113 calculates a second correction parameter for correcting the difference in brightness ratio between the two images 103a and 103b (step S12).
[0070] 14(a), (b), and (c) are conceptual diagrams of the spectral distribution characteristics emitted from light source 115. The horizontal axis indicates the wavelength band (nm), and the vertical axis indicates an arbitrary intensity Is. As described above, with respect to third chart 103 (integrating sphere), the illumination light guided from light source 115 through light guide path 116b illuminates the inside of the integrating sphere. Optical device 200 captures an image of the inside of the integrating sphere. Therefore, the optical device 200 is configured to guide light having the spectral distribution characteristics shown in FIG. 14(a), (b), and (c) to the integrating sphere through light guide path 116b.
[0071] The optical device 200 is an endoscope system 1 (FIG. 15) that can switch between different observation modes. ) The illumination light from the light source is switched for each observation mode of the endoscope system 1. The parameter calculation unit 113 calculates the second correction parameter for each observation mode.
[0072] In addition to an observation mode using white light, an endoscope also has an observation mode using a narrow band of wavelengths. In brightness and color calibration, there are cases where matching is not appropriate using correction parameters that are intended for white light only. For this reason, in this embodiment, correction parameters can be calculated and stored for each illumination light used for each observation mode.
[0073] The illumination light used for brightness and color tone calibration is illumination light corresponding to each observation mode of the endoscope, and brightness and color tone calibration can be performed for each observation mode using multiple illumination lights, at least broadband light of 400 to 700 nm used in the normal observation mode and narrowband light used in the narrowband observation mode.
[0074] In general, polarizing elements such as a polarizing beam splitter, a phase difference plate, and a depolarizing plate have wavelength dependency. Therefore, better image quality can be obtained by setting correction parameters for each observation mode. Note that there are individual differences in endoscopes that extend the depth of field due to manufacturing errors in optical elements and assembly. Therefore, unique correction parameters are required for each individual (optical device 200) as in this embodiment.
[0075] The brightness and color correction parameters obtained by calibration are switched in conjunction with each observation mode of the endoscope to correct the image.
[0076] The calibration of brightness and color tone is performed using an integrating sphere that has little dependence on polarization or wavelength as the subject, and the light source has spectral characteristics equivalent to those of the product. It is desirable to use a light guide that also has spectral transmittance characteristics equivalent to those of the product.
[0077] As described above, this embodiment has the advantage that it is possible to perform highly accurate alignment and calibration of brightness and color tone independent of wavelength or polarization, and that optimal composite images can be obtained in various observation modes.
[0078] 15 is a functional block diagram showing the configuration of an endoscope system storing the above-mentioned correction parameters. The endoscope system 1 of this embodiment includes an endoscope 2 to be inserted into a subject, a light source device 3 that supplies illumination light to the endoscope 2, a processor device 4, and an image display unit 5.
[0079] The processor device 4 has a function of performing image processing, but also has other functions. The processor device 4 has an actuator control unit 25, an image processor 30, and a control unit 39. The image display device 5 displays the image signal generated by the processor device 4 as an endoscopic image.
[0080] The endoscope 2 has a long and thin insertion section 6 to be inserted into the subject, and an operation section 7 provided at the rear end of the insertion section 6. A light guide cable 8 extends outward from the operation section 7. One end of the light guide cable 8 is detachably connected to the light source device 3 via a connection section 8a. The light guide cable 8 has a light guide 9 on the inside. A part of the light guide 9 is disposed inside the insertion section 6.
[0081] The light source device 3 incorporates a lamp 11 such as a xenon lamp as a light source. The light source is not limited to the lamp 11 such as a xenon lamp, and a light emitting diode (abbreviated as LED) may be used. The amount of light passing through the illumination light, for example, white light, generated by the lamp 11 is adjusted by the diaphragm 12. The illumination light is then focused by the condenser lens 13 and enters the entrance end surface of the light guide 9. The opening diameter of the diaphragm 12 can be changed by the diaphragm driver 14.
[0082] The light guide 9 transmits illumination light generated by the light source device 3 to the tip 6a of the insertion section 6. The transmitted illumination light is emitted from the tip surface of the light guide 9. An illumination lens 15 is disposed at the tip 6a so as to face the tip surface. The illumination lens 15 emits illumination light from an illumination window 15a. This illuminates the observation target site inside the subject.
[0083] An observation window 20 is provided at the tip 6a next to the illumination window 15a. Light from the observation target area passes through the observation window 20 and enters the tip 6a. An objective optical system OBL is disposed behind the observation window 20. The objective optical system OBL is composed of a lens group 16 and an optical path splitter 120.
[0084] The lens group 16 includes a lens 16a and a lens 21. The lens 21 is movable along the optical axis. This allows focusing to be performed. In order to move the lens 21, an actuator 22 is provided.
[0085] One image pickup element 122 (not shown) is disposed in the optical path splitting section 120. Two optical images are simultaneously formed on the light receiving surface of the image pickup element 122. The two optical images are picked up by the image pickup element 122.
[0086] The operation unit 7 is connected to the processor device 4 via a cable 24. A signal connector 24a is provided at the connection point with the processor device 4. Various information is transmitted between the endoscope 2 and the processor device 4 via the cable 24. The signal connector 24a has a correction parameter storage unit 37.
[0087] The correction parameter storage unit 37 stores (information on) correction parameters used for image correction. The correction parameters differ for each endoscope. Assume that an endoscope having unique endoscope identification information is connected to the processor device 4. In this case, based on the endoscope identification information, correction parameters unique to the connected endoscope are read from the correction parameter storage unit 37. Based on the read correction parameters, image correction is performed in the image correction processing unit 32. The control unit 39 determines whether or not to perform correction.
[0088] The actuator 22 is controlled by an actuator control section 25. To this end, the actuator 22 and the actuator control section 25 are connected via a signal line 23. In addition, the imaging element is connected to the image processor 30 via a signal line 27a. A signal from the imaging element is input to the image processor 30. In addition, information from a switch 26 provided in the operation section 7 is also transmitted to the processor device 4 via the signal line 27a.
[0089] If the optical path length of the first optical path is slightly different from that of the second optical path, two focused optical images are formed in front of and behind the imaging plane. The amount of deviation of the optical images from the imaging plane is small. Therefore, two optical images are formed on the imaging plane, with only a portion of the image being in focus.
[0090] The two optical images are captured by the imaging element. An image signal obtained by the imaging is input to an image processor 30 via a signal line 27a. The image processor 30 has an image readout section 31, an image correction processing section 32, an image synthesis processing section 33, a subsequent image processing section 34, an image output section 35, and a light adjustment section 36.
[0091] The image reading unit 31 reads out image signals of a plurality of images from the input image signals. Here, the number of optical images and the number of images are both two.
[0092] In an optical system that forms two optical images, geometric differences may occur. The geometric differences include relative deviations (differences) between the two optical images, such as deviations (differences) in magnification, positional deviations (differences), and deviations (differences) in the rotational direction. It is difficult to completely eliminate these differences during the manufacture of the objective optical system OBL. However, if the deviations (differences) become large, for example, the composite image will appear double. For this reason, it is preferable to correct the above-mentioned geometric differences in the image correction processing unit 32.
[0093] The image correction processing unit 32 performs image correction on the two read images. For example, the image correction processing unit 32 performs a process of correcting at least one of the differences in relative magnification, position, and rotation between the two images.
[0094] Furthermore, the image correction processing unit 32 performs color correction. To this end, the image correction processing unit 32 has a color correction unit (not shown). In the color correction, a process is performed to make the relative brightness and saturation of the two images approximately equal in at least one arbitrary specific wavelength band. The color correction may be performed in the image correction processing unit 32 without providing a color correction unit.
[0095] Image correction processing unit 32 changes the luminance of one of the two images so that it approximately matches the luminance of the other image. Also, image correction processing unit 32 changes the saturation of one image so that it approximately matches the saturation of the other image.
[0096] As described above, in the method of acquiring an image with a large depth of field, only the areas in focus are extracted from multiple images, and the extracted areas are synthesized. In the endoscope system of this embodiment, the difference in brightness and color tone between multiple images can be reduced. Therefore, unevenness in brightness and difference in color tone can be reduced in the synthesized image.
[0097] In addition, in a method for improving the color reproducibility of an image, image synthesis is performed using two images. If there is a difference in brightness or color tone between two optical images, the two images obtained by imaging will also have a difference in brightness or color tone. In the endoscope system of this embodiment, even if there is a difference in brightness or color tone between multiple images, the difference in brightness or color tone can be reduced. Therefore, the color reproducibility of the synthesized image can be further improved.
[0098] In the image synthesis processing unit 33, first, the spatial frequency characteristic values are compared using two images. This comparison is performed for each of the spatially identical pixel regions in the two images. Next, the pixel region with the relatively higher spatial frequency characteristic value is selected. Then, one image is generated using the selected pixel region. In this manner, one synthetic image is generated from the two images. Note that, if the difference in spatial frequency characteristic values between the two images is small, a synthetic image may be generated after synthetic image processing in which each image is weighted and added together.
[0099] In the subsequent image processing unit 34, image processing such as edge enhancement, gamma correction, etc. is performed on the composite image. The image output unit 35 outputs the image that has been subjected to the image processing to the image display device 5.
[0100] In the light adjustment unit 36, a light adjustment signal for adjusting the light to a reference brightness is generated from the image read by the image reading unit 31. The light adjustment signal is output to the aperture driving unit 14 of the light source device 3. The aperture driving unit 14 adjusts the opening amount of the aperture 12 in accordance with the light adjustment signal so as to maintain the reference brightness.
[0101] The image calibration device described above may satisfy a plurality of requirements at the same time. This is preferable for obtaining a good endoscope optical system. The combination of the requirements may be arbitrary.
[0102] Furthermore, the present embodiment is not limited to a calibration device for an endoscope system that extends the depth of field, but is also useful as an image calibration device for an optical device that performs stereoscopic vision from left-eye and right-eye images, for example.
[0103] Various embodiments of the present invention have been described above, but the present invention is not limited to these embodiments. Within the scope of the spirit of the present invention, embodiments that are formed by appropriately combining the configurations of these embodiments also fall within the scope of the present invention.
[0104] (Additional Note) The following configuration of the invention is derived from these embodiments. (Additional note 1) An image calibration method for an optical device having, in order from an object side, an objective optical system, an optical path splitting unit that splits an object image obtained by the objective optical system into two optical images having different optical path differences, and an image sensor that obtains two images from the two optical images, the method comprising: providing a first chart, a second chart, and a third chart; irradiating the first chart, the second chart, and the third chart with illumination light; placing the first chart, the second chart, and the third chart; a first calculation step of calculating a spatial frequency characteristic value of each of first two images obtained by capturing an image of the first chart; a second calculation step of calculating a difference in a relative positional relationship between the two second images obtained by capturing an image of the second chart by the optical device in a state where the second chart is disposed at a position where the spatial frequency characteristic values of the two first images are equivalent; a third calculation step of calculating a difference in relative brightness ratio between two third images obtained by capturing images of the third chart illuminated by the light source using the optical device, for each RGB signal; a parameter calculation step of calculating a first correction parameter for correcting the difference in the positional relationship and a second correction parameter for correcting the difference in the brightness ratio; 13. An image calibration method comprising: (Additional note 2) An endoscope system comprising: a correction parameter storage unit that stores a first correction parameter for correcting the difference in the positional relationship described in appended claim 1; and a second correction parameter for correcting the difference in the brightness ratio. [Industrial Applicability]
[0105] As described above, the present invention is useful for an image calibration device that reduces degradation of image quality and artifacts (noise) in a synthesized image by obtaining optimal correction parameters in advance when synthesizing two images in an endoscopic system. [Explanation of symbols]
[0106] L1~L10 lenses AX optical axis I image plane OBL Objective Optical System 1 Endoscope system 4 Processor Unit 5 Image display device 100 Image Calibration Device 101 First Chart 102 Second Chart 103 Third Chart Stages 104 and 105 106, 107 Holder 108 Calibration Processor 109 Storage area 110 First calculation section 111 Second calculation section 112 Third Calculation Section 113 Parameter Calculation Unit 114 Drive unit 115 Light source 116,116a, 116b light guide 120 Optical path splitter 122 Image sensor 200 Optical equipment
Claims
1. An image calibration device for an optical device having, in order from an object side, an objective optical system, an optical path splitting unit that splits an object image obtained by the objective optical system into two optical images having different optical path lengths, and an image sensor that obtains two images from the two optical images, A first chart; A second chart; A third chart; a light source that irradiates illumination light onto the first chart, the second chart, and the third chart; a stage on which the first chart, the second chart, and the third chart are placed; A calibration processor; The calibration processor: a first calculation unit that calculates a spatial frequency characteristic value of each of first two images obtained by capturing an image of the first chart; a second calculation unit that calculates a difference in a relative positional relationship between two second images obtained by capturing an image of the second chart by the optical device in a state in which the second chart is disposed at a position of the first chart where the spatial frequency characteristic values of the two first images are equivalent; a third calculation unit that calculates a difference in relative brightness ratio between two third images obtained by capturing images of the third chart illuminated by the light source using the optical device, for each RGB signal; a parameter calculation unit that calculates a first correction parameter for correcting the difference in the positional relationship and a second correction parameter for correcting the difference in the brightness ratio; 1. An image calibration device comprising:
2. the calibration processor further includes a driving unit that drives the stage so as to change the observation distance of the first chart with respect to the optical device in any step; a storage unit in which a still image of the first chart is stored for each of the observation distances, the first calculation unit calculates a spatial frequency characteristic value of each of the two images of the first chart calculated from the still image of the first chart stored in the storage unit, and calculates an observation distance at which the spatial frequency characteristic value is the same for the two images of the first chart; the stage has a holder for placing the second chart at the observation distance calculated by the first calculation unit, the storage unit stores a still image of the second chart; 2. The image calibration device according to claim 1 , wherein the parameter calculation unit calculates the first correction parameter based on a still image of the second chart and the difference in the positional relationship calculated by the second calculation unit.
3. the first chart is a chart for determining a spatial frequency characteristic value, the first chart having a chart pattern including a black-and-white edge ridge line, 3. The image calibration device according to claim 2, wherein the first chart is disposed so that the centers of view of the two images intersect with the black and white edge ridgeline.
4. 2. The image calibration device according to claim 1, wherein the second chart is a grayscale checker pattern in which areas having at least two or more different gradations are randomly arranged.
5. 2 . The image calibration device according to claim 1 , wherein the parameter calculation section calculates the first correction parameter using one of the two images of the second chart as a reference.
6. 2. The image calibration device according to claim 1, wherein the third chart is an integrating sphere, and the spectral characteristics of the illumination light guided to the integrating sphere are the same as the spectral characteristics of a light source when the optical device is used.
7. the optical device is an endoscope system capable of switching between different observation modes, and illumination light from the light source is switched for each of the observation modes of the endoscope system; 2. The image calibration device according to claim 1, wherein the parameter calculation section calculates the second correction parameter for each of the observation modes.
Citation Information
Patent Citations
Calibration method and device
JP2011030698A
Calibration method and endoscope system
JP2017196430A
Image capture device and iamge capture device system
WO2013027459A1
Endoscopic system
WO2013061819A1
Endoscope system
WO2014002740A1
