Image processing device, endoscope device, image processing method, program, and storage medium

Abstract

To provide a technique advantageous for improvement in the effect of image processing.SOLUTION: An image processing device comprises: a gradation correction unit which generates a plurality of gradation correction signals obtained by converting the gradation characteristics of a plurality of image signals in mutually-different wavebands; a contour emphasis unit which generates a contour emphasis signal by performing prescribed processing on at least one image signal in the plurality of image signals; and a synthesis unit which synthesizes the signal based on the plurality of gradation correction signals and the contour emphasis signal. The gradation correction unit generates at least two gradation correction signals obtained by converting the gradation characteristics of at least two image signals while keeping a ratio of pixel values of corresponding pixels of the at least two image signals in the plurality of image signals.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device, an endoscope device, an image processing method, a program, and a storage medium. [Background technology]

[0002] In image processing devices, various signal processing operations are performed to improve image quality. Patent Document 1 discloses that in a video signal processing device that performs edge enhancement processing and gradation conversion processing, edge enhancement processing is performed according to the characteristics of the gradation conversion processing. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2005-252869 A Summary of the Invention [Problem to be solved by the invention]

[0004] For high-contrast images with blurred edges and blocked-up shadows or blown-out highlights, it is conceivable to apply both edge enhancement and gradation conversion intensively. When edge enhancement is performed according to the characteristics of the gradation conversion, it is affected by the characteristics of the gradation conversion, and therefore there is a possibility that appropriate edge enhancement cannot be performed.

[0005] An object of the present invention is to provide a technique that is advantageous for improving the effect of image processing. [Means for solving the problem]

[0006] In view of the above problems, an image processing device according to an embodiment of the present invention includes a gradation correction unit that generates a plurality of gradation correction signals by converting the gradation characteristics of a plurality of image signals having different wavelength bands, a contour enhancement unit that generates a contour enhancement signal by performing a predetermined processing on at least one of the plurality of image signals, and a synthesis unit that synthesizes the contour enhancement signal with a signal based on the plurality of gradation correction signals, wherein the gradation correction unit generates at least two gradation correction signals by converting the gradation characteristics of the at least two image signals while maintaining a ratio of pixel values ​​of corresponding pixels of at least two image signals among the plurality of image signals. Effect of the Invention

[0007] According to the present invention, it is possible to provide a technique that is advantageous in improving the effect of image processing. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an image processing apparatus according to an embodiment of the present invention. [Diagram 2] 2 is a diagram showing an example of the arrangement of a tone correction unit of the image processing device in FIG. 1; [Diagram 3] 1 is a diagram showing an example of the configuration of an endoscope apparatus using an image processing apparatus according to an embodiment of the present invention. [Figure 4] FIG. 4 is a diagram showing an example of the configuration of the image processing device shown in FIG. 3. [Diagram 5] 4 is a diagram showing an example of an LUT characteristic of a tone correction unit of the image processing device in FIG. 3. [Figure 6] 4 is a diagram showing an example of input / output characteristics of a tone correction unit of the image processing device in FIG. 3. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0010] An image processing device according to an embodiment of the present disclosure will be described with reference to FIGS. 1 to 6. FIG. 1 is a diagram showing an example of the configuration of an image processing device 100 according to the present embodiment. The image processing device 100 includes a gradation correction unit 152, a contour enhancement unit 153, and a synthesis unit 157. The gradation correction unit 152 generates a plurality of gradation correction signals 211, 212 by converting the gradation characteristics of a plurality of image signals 201, 202 having different wavelength bands. In the configuration shown in FIG. 1, two image signals 201, 202 are input to the gradation correction unit 152. The contour enhancement unit 153 generates a contour enhancement signal 221 by performing a predetermined process on at least one image signal 201 of the plurality of image signals 201, 202. The gradation correction signals 211, 212 generated by the gradation correction unit 152 and the contour enhancement signal 221 generated by the contour enhancement unit 153 are input to the synthesis unit 157. The synthesis unit 157 synthesizes the plurality of gradation correction signals 211 and 212 and the edge emphasis signal 221 to generate synthesis signals 231 and 232 .

[0011] 1, an image signal 201 is input in parallel to the gradation correction unit 152 and the contour enhancement unit 153. Therefore, the image signal 201 input to the image processing device 100 is subjected to gradation correction processing by the gradation correction unit 152. Furthermore, the image signal 201 input to the image processing device 100 is subjected to contour enhancement processing by the contour enhancement unit 153 without being affected by increased noise and reduced contrast due to the gradation correction processing. Furthermore, by combining a contour enhancement signal 221 with the gradation correction signals 211 and 212 after the gradation correction processing, it is possible to improve the local contrast of the portion where the contrast has been reduced by the compression processing during the gradation correction processing.

[0012] FIG. 2 is a diagram showing an example of the configuration of the gradation correction unit 152. The gradation correction unit 152 includes a gain multiplication unit 521 and a gain calculation unit 522. The two input image signals 201 and 202 are input in parallel to the gain calculation unit 522 and the gain multiplication unit 521. The gain calculation unit 522 calculates a gain value based on the two input image signals 201 and 202. Details of the calculation of the gain value will be described later. The calculated gain value is input to the gain multiplication unit 521. The gain multiplication unit 521 applies the gain value calculated by the gain calculation unit 522 to the two input image signals 201 and 202, respectively, and outputs gradation correction signals 211 and 212. Since the same gain value is applied to the image signals 201 and 202 in the gain multiplication unit 521, the two gradation correction signals 211 and 212 are generated while maintaining the ratio of the pixel values ​​of the corresponding pixels of the two image signals 201 and 202. It can also be said that the gradation correction unit 152 generates a plurality of gradation correction signals 211, 212 by converting the gradation characteristics of the two image signals 201, 202 while maintaining the ratio of the pixel values ​​of the corresponding pixels of the two image signals 201, 202. Since the gradation correction unit 152 converts the gradation characteristics while maintaining the ratio of the pixel values ​​of the corresponding pixels of the plurality of image signals 201, 202, the gradation correction process is performed without changing the hue or saturation.

[0013] Here, the image signals 201 and 202 input to the image processing device 100 are signals of different wavelength bands obtained by capturing an image of the same subject. The image signals 201 and 202 may be image signals of different wavelength bands obtained by capturing an image once. For example, the image signals 201 and 202 may be image signals captured continuously by switching between illuminations of different wavelength bands. For example, the image signals 201 and 202 may be image signals of different wavelength bands such as red, green, and blue. For example, the image signal 201 may be an image signal of visible light, and the image signal 202 may be an image signal of non-visible light. As a more specific example, an endoscope device including the image processing device 100 of this embodiment is given, and the image processing device 100 will be described in further detail below.

[0014] 3 is a diagram showing an example of the configuration of an endoscope apparatus 101 equipped with the image processing device 100 of this embodiment. The endoscope apparatus 101 includes a light source device 170, an imaging device 102 that generates a plurality of image signals by capturing an image of a subject illuminated by the light source device 170, a camera control unit 140 that includes the above-mentioned image processing device 100, and a display device 180. The imaging device 102 includes a scope 110, a coupler lens 120, and a camera head 130.

[0015] The scope 110 can be inserted into the body of a subject, for example. Light emitted from the light source device 170 is irradiated onto the subject via an illumination optical system 112 arranged in the scope 110. Reflected light of the light irradiated onto the subject and fluorescence generated by the irradiated light are imaged on the image pickup elements 132 to 135 via an imaging optical system 111 arranged in the scope 110, a coupler lens 120, and a color separation element 131 arranged in the camera head 130. Image signals output from each of the image pickup elements 132 to 135 are output from the camera head 130 via a transmission unit 136 and input to the camera control unit 140. In the camera control unit 140, the input image signal is restored by a receiving unit 141, and the restored image signals obtained by the image pickup elements 132 to 135 are input to the image processing device 100. The image signal processed by the image processing device 100 is transferred from the camera control unit 140 to the display device 180 via the transmission section 143, and the image (video) is displayed on the display device 180.

[0016] The settings of the operations of the camera head 130, the light source device 170, and the image processing device 100 can be performed by the user operating an input unit such as a graphical user interface (GUI) 144 arranged in the camera control unit 140. The settings input by the user via the GUI 144 are received by a control unit 160 arranged in the camera control unit 140, and the processing in the image processing device 100 and the control of the camera head 130 are changed as appropriate. For example, when the control content of the operation of the camera head 130 is changed, a control signal is transferred from the control unit 160 of the camera control unit 140 to a control unit 137 arranged in the camera head 130, and the control of the operations of the image pickup elements 132 to 135 is changed in response to the control signal.

[0017] Camera head 130 may be provided with flash memory 138. Flash memory 138 may store programs related to the operation and control of camera head 130, adjustment parameters for correcting individual differences in camera head 130, and the like.

[0018] The camera control unit 140 may be provided with a flash memory 142. Programs related to the operation and control of the camera control unit 140, values ​​set from the outside such as by a user, etc. may be stored in the flash memory 142. Furthermore, the flash memories 138 and 142 may be other storage devices such as HDDs.

[0019] In this embodiment, the light source device 170 includes three types of light emitting diodes (LEDs) 172 to 174 and a laser diode (LD) 175. The light source device 170 combines the emitted light from the LEDs 172 to 174 and the LD 175 in an illumination optical system 171 arranged in the light source device 170, and outputs the combined light to an illumination optical system 112 arranged in the scope 110. The control unit 160 arranged in the camera control unit 140 outputs a control signal to a control unit 176 arranged in the light source device 170 as necessary, and the light source device 170 adjusts the type, timing, and output level of the light source to be turned on in response to the control signal.

[0020] For example, the LED 172 may emit light in a red wavelength band, the LED 173 may emit light in a green wavelength band, and the LED 174 may emit light in a blue wavelength band. For example, the LD 175 may emit light of an infrared excitation wavelength of a fluorescent reagent. Hereinafter, the light emitted by the LD 175, which is not a general visible light, may be called special light. Furthermore, an image signal obtained by irradiating these special lights may be called an image signal of special light. Examples of the image signal of special light include image signals obtained by non-visible light, fluorescence due to irradiation with excitation light, and reflected light due to irradiation with narrow band light having a half-width of 20 μm or less. In this embodiment, the image signal of special light is an image signal obtained from fluorescence excited by infrared excitation light (LD 175). In this case, the reflected light of the infrared excitation light may be removed by a wavelength cut filter not shown.

[0021] The wavelengths separated by color separation element 131 correspond to the wavelengths emitted by LEDs 172-174 and LD 175. By color separation element 131, light in the red wavelength band is input to imaging element 132, light in the green wavelength band is input to imaging element 133, light in the blue wavelength band is input to imaging element 134, and light in the wavelength band of fluorescence excited by infrared excitation light is input to imaging element 135. The outputs of imaging elements 132-135 are transferred to image processing device 100 as signals R_in, G_in, B_in, and IR_in.

[0022] 4 shows an example of the configuration of the image processing device 100 disposed in the camera control unit 140 of this embodiment. The image processing device 100 includes the above-mentioned gradation correction unit 152, contour enhancement unit 153, and synthesis unit 157. In the configuration shown in FIG. 4, the image processing device 100 further includes a front-stage image processing unit 151, a luminance processing unit 154, a color processing unit 155, a special light processing unit 156, a color synthesis unit 158, and an image output unit 159.

[0023] The signals R_in, G_in, B_in, and IR_in input to the image processing device 100 are first processed by a front-stage image processing unit 151. The front-stage image processing unit 151 performs, for example, digital gain processing, linear matrix processing, defective pixel correction processing, shading correction processing, and the like on each of the signals R_in, G_in, B_in, and IR_in.

[0024] The output of the front-stage image processing unit 151 is transferred to the gradation correction unit 152 and the contour emphasis unit 153. Therefore, the image signals R_f, G_f, B_f, and IR_f that are the outputs of the front-stage image processing unit 151 correspond to the above-mentioned image signals 201 and 202. In the configuration shown in FIG. 4, the image signals R_f, G_f, B_f, and IR_f are transferred to the gradation correction unit 152. Moreover, the image signals R_f, G_f, and B_f are transferred in parallel not only to the gradation correction unit 152 but also to the contour emphasis unit 153. Therefore, in the configuration shown in FIG. 4, the image signals R_f, G_f, and B_f correspond to the above-mentioned image signal 201, and the image signal IR_f corresponds to the above-mentioned image signal 202.

[0025] The gradation correction unit 152 determines a gain value based on at least one of the multiple image signals R_f, G_f, B_f, and IR_f, and applies the gain value to each of the image signals R_f, G_f, B_f, and IR_f. As a result, the gradation correction unit 152 generates gradation correction signals R_g, G_g, B_g, and IR_g corresponding to the above-mentioned gradation correction signals 211 and 212. In the configuration shown in Fig. 4, three image signals R_f, G_f, and B_f, which are part of the signals input to the gradation correction unit 152, are input to a MAX circuit 523 arranged in a gain calculation unit 522 of the gradation correction unit 152. The MAX circuit 523 obtains the maximum pixel value among the pixel values ​​included in the input image signals R_f, G_f, and B_f. A signal max indicating the acquired maximum pixel value is input to the LUT 524 arranged in the gain calculation unit 522, and a signal gain indicating the gain value as an output is transferred to the gain multiplication unit 521. In this way, the gradation correction unit 152 determines the gain value based on the maximum pixel value among the pixel values ​​included in at least one of the multiple image signals R_f, G_f, B_f, and IR_f. At this time, unlike the configuration shown in FIG. 2, the multiple image signals R_f, G_f, B_f, and IR_f may include an image signal IR_f that is not used by the gradation correction unit 152 to determine the gain value. For example, the gradation correction unit 152 may determine the gain value based on the visible light image signals R_f, G_f, and B_f among the multiple image signals R_f, G_f, B_f, and IR_f, as in the configuration shown in FIG. 4. Furthermore, the gradation corrector 152 does not need to use the image signal IR_f of the special light among the multiple image signals R_f, G_f, B_f, and IR_f to determine the gain value.

[0026] The gain multiplication unit 521 applies the gain value calculated by the gain calculation unit 522 to the image signals R_f, G_f, B_f, and IR_f. For example, the gain multiplication unit 521 may apply the gain value to the visible light image signals R_f, G_f, and B_f. As a result, gradation correction signals R_g, G_g, and B_g are generated by converting the gradation characteristics of the image signals R_f, G_f, and B_f while maintaining the ratio of pixel values ​​of the corresponding pixels of the image signals R_f, G_f, and B_f. In this way, the gradation correction unit 152 may generate at least two gradation correction signals by converting the gradation characteristics of at least two image signals while maintaining the ratio of pixel values ​​of the corresponding pixels of at least two signals among the multiple image signals R_f, G_f, B_f.

[0027] Furthermore, the gain multiplication unit 521 may uniformly apply the gain value calculated by the gain calculation unit 522 to each of the image signals R_f, G_f, B_f, and IR_f, as shown in Fig. 4. In other words, the gradation correction unit 152 may generate a plurality of gradation correction signals R_g, G_g, B_g, and IR_g by converting the gradation characteristics of the plurality of image signals R_f, G_f, B_f, and IR_f while maintaining the ratio of pixel values ​​of corresponding pixels in all of the plurality of image signals R_f, G_f, B_f, and IR_f.

[0028] The gradation correction signals R_g, G_g, and B_g are transferred in parallel to a luminance processing unit 154 and a color processing unit 155. The gradation correction signal IR_g is transferred to a special light processing unit 156.

[0029] The image signals R_f, G_f, and B_f input to the contour enhancement unit 153 are transferred to a contour reference signal generation unit 531. For example, the contour reference signal generation unit 531 may generate a Y signal from the image signals R_f, G_f, and B_f (RGB signals) and use the Y signal as the reference signal Db. For another example, the contour reference signal generation unit 531 may generate an average signal of the image signals R_f and G_f and use the average signal as the reference signal Db. For another example, the contour reference signal generation unit 531 may use the image signal G_f of the input image signals R_f, G_f, and B_f as the reference signal. The reference signal Db, which is the output of the contour reference signal generation unit 531, is transferred to a noise suppression unit 532. The noise suppression unit 532 performs, for example, frame cyclic processing, filter processing, or a combination of these, on the reference signal Db. The reference signal Db, which has been noise-suppressed by the noise suppression unit 532, is transferred to the band extraction and amplification unit 533 as a reference signal Dn.

[0030] The band extraction / amplification unit 533 extracts one or more spatial frequency band components to be emphasized as a contour signal from the reference signal Dn. The band extraction / amplification unit 533 performs gain adjustment for each extracted spatial frequency band, synthesizes the components of each band as necessary, and outputs the result as a contour enhancement signal Dtl (corresponding to the above-mentioned contour enhancement signal 221). In this way, the contour enhancement unit 153 generates the reference signal Dn based on at least one image signal from the multiple image signals R_f, G_f, B_f, and IR_f, and generates the contour enhancement signal Dtl by extracting one or more spatial frequency band components from the reference signal.

[0031] The contour enhancing unit 153 may divide the reference signal Db, which is the output of the contour reference signal generating unit 531, into a plurality of signals (e.g., reference signals Db1, Db2) according to the signal amplitude, and perform noise suppression processing and band extraction / amplification processing with different characteristics on each of the signals. The contour enhancing unit 153 may combine the plurality of outputs after the band extraction / amplification processing, and output the combined result as the contour enhanced signal Dtl. In other words, the contour enhancing unit 153 may generate the contour enhanced signal Dtl by extracting components of a plurality of spatial frequency bands from the reference signal Db, adjusting the ratio between the components of the plurality of spatial frequency bands, and combining the components of the plurality of spatial frequency bands whose ratios have been adjusted.

[0032] The luminance processing unit 154 generates a Y signal (luminance signal Y_p) based on the input gradation correction signals R_g, G_g, B_g (RGB signals). The luminance processing unit 154 may perform, for example, noise suppression processing and band limiting processing before and after generating the luminance signal Y_p. The color processing unit 155 generates color difference signals Pb, Pr based on the input gradation correction signals R_g, G_g, B_g (RGB signals). The color processing unit 155 may perform, for example, color matrix processing, chroma gain processing, color deviation suppression processing, noise suppression processing, band limiting processing before and after generating the color difference signals Pb, Pr. In the configuration shown in FIG. 4, the luminance processing unit 154 and the color processing unit 155 are shown as separate configurations, but each processing may be performed as an integrated configuration. Each configuration is simply shown separately so that each processing to be performed is clear. The same applies to other configurations.

[0033] As described above, the gradation correction signal IR_g is transferred from the gradation correction unit 152 to the special light processing unit 156. The special light processing unit 156 performs processes such as noise suppression, band limitation, gradation correction, shading correction, and offset adjustment on the gradation correction signal IR_g, and outputs a signal IR_p.

[0034] The synthesis unit 157 receives the luminance signal Y_p output from the luminance processing unit 154 and the contour emphasis signal Dtl output from the contour emphasis unit 153. The synthesis unit 157 synthesizes the luminance signal Y_p generated by the luminance processing unit 154 based on the multiple gradation correction signals R_g, G_g, and B_g generated by the gradation correction unit 152, and the contour emphasis signal Dtl generated by the contour emphasis unit 153. As a synthesis method, for example, the luminance signal Y_p and the contour emphasis signal Dtl may be simply added. The synthesis unit 157 outputs a luminance signal Y_d obtained by contour-emphasizing the luminance signal Y_p.

[0035] To the color synthesis unit 158, the luminance signal Y_d from the synthesis unit 157, the color difference signals Pb_p and Pr_p from the color processing unit 155, and the signal IR_p from the special light processing unit are transferred. In a display mode related to normal visible light, the input luminance signal Y_d and color difference signals Pb_p and Pr_p can be output from the color synthesis unit 158 ​​as is. In a display mode using special light, the color synthesis unit 158 ​​performs synthesis processing so that an image (video) obtained by the special light is visualized or emphasized. For example, the color synthesis unit 158 ​​performs processing such as changing the color according to the ratio of the pixel values ​​of the luminance signal Y_p output from the luminance processing unit 154 and the gradation correction signal IR_p output from the gradation correction unit 152.

[0036] The image output unit 159 receives the signals Y_c, Pb_c, and Pr_c transferred from the color synthesis unit 158. The image output unit 159 may perform processing such as RGB conversion and signal value range restriction in accordance with the standard of the output signal of the camera control unit 140. As described above, the luminance signal Y_p based on the gradation correction signals R_g, G_g, and B_g and the edge emphasis signal Dtl are synthesized as a luminance signal Y_d by the synthesis unit 157. Based on this luminance signal Y_d, the color difference signals Pb_p and Pr_p generated by the color processing unit 155, and the signal Ir_p generated by the special light processing unit 156, the color synthesis unit 158 ​​generates the signals Y_c, Pb_c, and Pr_c. A nonlinear correction unit may be provided in the image output unit 159 for performing correction processing such as gamma correction according to the nonlinear characteristics of the display device 180 connected to the image processing device 100 (camera control unit 140) on the signals Y_c, Pb_c, and Pr_c based on the signal synthesized by the synthesis unit 157. After these processes are performed, signals R_out, G_out, and B_out are output from the image processing device 100. The signals R_out, G_out, and B_out are transferred to the display device 180 via the transmission unit 143, and an image (video) is displayed on the display device 180.

[0037] Fig. 5 shows an example of the characteristics of the LUT 524 of the gradation correction unit 152 of this embodiment. The black signal of the display device 180 is 0 IRE, the reference white signal of the display device 180 is 100 IRE, and the range of signals that can be input to the display device 180 is 0 to 109 IRE. Also, it is assumed that the outputs of the imaging elements 132 to 135 and the input to the image processing device 100 range from 0 to 425 IRE. In this case, in order to suppress blackout and whiteout and achieve a gradation display with good visibility, it is necessary to raise the dark areas, suppress the bright areas, and concentrate the signals in the intermediate range. Fig. 5 shows an example of the characteristics of the LUT 524 for achieving this.

[0038] 5, the horizontal axis represents the magnitude (IRE) of the input signal to the LUT 524. The vertical axis represents the gain value (dB) that is the output of the LUT 524. As described above, the input signal to the LUT 524 is the maximum pixel value (signal max) among the pixel values ​​included in each of the image signals R_f, G_f, and B_f.

[0039] In order to improve blackout, it is necessary to increase the gain in a region where the input signal is small. In the example shown in FIG. 5, the gain when 0IRE is input is set as the output peak. The gain may be somewhat higher or lower depending on the balance with noise, and the output peak may be up to 10% of the maximum input range (input of 0IRE to 42.5IRE in this embodiment). In other words, the pixel values ​​(signal max) included in each of the multiple image signals R_f, G_f, and B_f have a signal range from the minimum luminance value (0IRE) to the maximum luminance value (425IRE). In this case, the gain value may be such that the maximum pixel value (signal max) is in the range from the minimum luminance value (0IRE) to 10% (42.5IRE) of the signal range.

[0040] Furthermore, in order to concentrate signals in the middle range with a natural appearance, the output may be monotonically decreased in a range of at least up to 50% (input up to 212.5IRE in this embodiment) after the input corresponding to the output peak. In other words, when the maximum pixel value (signal max) is in the range from the pixel value where the gain value is maximum to 50% (212.5IRE) of the signal range, the gain value may be monotonically decreased from the pixel value where the gain value is maximum to 50% (212.5IRE) of the signal range. In the high luminance part (212.5IRE to 425IRE) where the signal input (signal max) exceeds 50%, as shown in FIG. 5, the gain value may continue to be monotonically decreased, or may be increased to prevent a decrease in the maximum value of the output of the gradation correction unit 152.

[0041] FIG. 6 shows an example of the input / output characteristics of the gradation correction unit 152 of this embodiment. FIG. 6 shows the characteristics of a portion of the output signal of the gradation correction unit 152 when the image signal on which the signal max is based is multiplied by the gain value of the LUT 524 having the characteristics shown in FIG. 5. With respect to the signal input, dark areas are amplified and bright areas are suppressed, with the signal concentrating in the intermediate range. Although the contrast in the intermediate range is reduced, the signal value output across the entire range increases monotonically (x <yならばf(x)≦f(y))である。

[0042] The gradation correction unit 152 includes a gain calculation unit 522 that outputs a gain value in response to an input of at least one image signal among a plurality of image signals. In this case, the input / output characteristics of the gain calculation unit 522 may be configured to be changeable in response to a user's setting. For example, the characteristics of the LUT 524 may be switched in response to a control signal from the control unit 160 in response to a user's input via the GUI 144. For example, the characteristics of the LUT 524 may be switched between a case where the image signal IR_f of special light is used and a case where it is not used. In addition, the degree of gradation correction may be adjusted in various ways, such as by receiving a control signal from the control unit 160 in response to a user's input via the GUI 144 and weighting and interpolating the gain value with 1.

[0043] In the endoscope device 101 configured as above, the input image signals R_f, G_f, B_f, and IR_f are subjected to gradation correction processing, but the edge emphasis processing is performed without being affected by noise enhancement and contrast reduction caused by the gradation correction processing. Furthermore, by synthesizing the edge emphasis processing signal Dtl that has been subjected to the edge emphasis processing with the luminance signal Y_p based on the gradation correction signals R_g, G_g, and B_g after the gradation correction, it is possible to improve the local contrast of the part with reduced contrast. As a result, it is possible to output an image that has been effectively processed against the blurred edges, display of high contrast scenes, noise in dark areas, and the like that are generally problems with the endoscope device 101. In addition, it is possible to maintain the ratio of pixel values ​​of the image signals R_f, G_f, and B_f to the image signal IR_f while performing the gradation correction processing and edge emphasis processing based on the image signals R_f, G_f, and B_f (RGB signals). In other words, the gradation correction unit 152 generates a plurality of corresponding gradation correction signals R_g, G_g, B_g, IR_g by converting the gradation characteristics of the visible light image signals R_f, G_f, B_f and the special light image signal IR_f while maintaining the ratio of pixel values ​​of pixels corresponding to each of the visible light image signals R_f, G_f, B_f and the special light image signal IR_f among the plurality of image signals R_f, G_f, B_f, IR_f. Therefore, in the signal processing at the subsequent stage, appropriate processing can be performed according to the respective signal ratios of the image obtained by visible light and the image obtained by special light such as invisible light.

[0044] The present disclosure enables tone correction processing independent of contour emphasis processing, and does not deny performing nonlinear processing or tone correction processing on a contour emphasis signal. For example, in order to save the number of bits in signal processing, a gamma correction curve or a Log curve may be applied to each signal generated by the contour emphasis unit 153, the luminance processing unit 154, the color processing unit 155, etc. In addition, since noise increase is not completely eliminated even in tone correction processing, noise reduction processing according to the degree of tone correction processing may be performed at an appropriate timing in the image processing device 100. In the above embodiment, a delay circuit for adjusting the timing of each signal is not described, but it goes without saying that a delay circuit is used appropriately according to the configuration of the circuit.

[0045] The disclosure of this specification includes the following image processing device, endoscope device, image processing method, program, and storage medium.

[0046] (Item 1) a gradation correction unit that generates a plurality of gradation correction signals by converting gradation characteristics of a plurality of image signals having mutually different wavelength bands; a contour emphasis unit that generates a contour emphasis signal by performing a predetermined process on at least one image signal among the plurality of image signals; and a synthesis unit that synthesizes a signal based on the plurality of gradation correction signals and the contour emphasis signal, The image processing device is characterized in that the gradation correction unit generates at least two gradation correction signals by converting the gradation characteristics of at least two image signals among the multiple image signals while maintaining the ratio of pixel values ​​of corresponding pixels of the at least two image signals.

[0047] (Item 2) The image processing device according to item 1, characterized in that the gradation correction unit generates the plurality of gradation correction signals by converting the gradation characteristics of the plurality of image signals while maintaining the ratio of pixel values ​​of corresponding pixels of all of the plurality of image signals.

[0048] (Item 3) The gradation correction unit is determining a gain value based on at least one image signal of the plurality of image signals; 3. The image processing device according to item 1 or 2, wherein the plurality of gradation correction signals are generated by applying the gain value to each of the plurality of image signals.

[0049] (Item 4) 4. The image processing device according to item 3, wherein the plurality of image signals includes an image signal that is not used by the gradation correction unit in determining the gain value.

[0050] (Item 5) 5. The image processing device according to item 3 or 4, wherein the gradation correction unit determines the gain value based on a maximum pixel value among pixel values ​​included in at least one of the plurality of image signals.

[0051] (Item 6) a pixel value included in each of the plurality of image signals has a signal range from a minimum luminance value to a maximum luminance value; The gain value is the maximum pixel value is maximum when it is within a range from the minimum luminance value to 10% of the signal range, 6. The image processing device according to item 5, characterized in that when the maximum pixel value is within a range from the pixel value at which the gain value is maximum to 50% of the signal range, the gain value monotonically decreases from the pixel value at which the gain value is maximum to 50% of the signal range.

[0052] (Item 7) the gradation correction unit includes a gain calculation unit that outputs the gain value in response to an input of at least one image signal among the plurality of image signals; 7. The image processing device according to any one of items 3 to 6, wherein the input / output characteristics of the gain calculation unit are configured to be changeable in response to a user setting.

[0053] (Item 8) 8. The image processing device according to any one of items 3 to 7, wherein the gradation correction unit determines the gain value based on a visible light image signal among the plurality of image signals.

[0054] (Item 9) The image processing device according to any one of items 3 to 8, characterized in that the gradation correction unit does not use an image signal of special light, which includes at least one of invisible light, fluorescence due to irradiation with excitation light, and reflected light due to irradiation with narrowband light having a half-width of 20 μm or less, among the plurality of image signals, in determining the gain value.

[0055] (Item 10) The image processing device described in item 9, wherein the gradation correction unit generates corresponding gradation correction signals among the plurality of gradation correction signals by converting the gradation characteristics of the visible light image signal and the special light image signal while maintaining a ratio of pixel values ​​of corresponding pixels of the visible light image signal and the special light image signal among the plurality of image signals.

[0056] (Item 11) The image processing device described in any one of items 1 to 10, characterized in that the contour enhancement unit generates a reference signal based on at least one image signal among the multiple image signals, and generates the contour enhancement signal by extracting components of one or more spatial frequency bands from the reference signal.

[0057] (Item 12) Item 12. The image processing device according to item 11, wherein the contour enhancement unit extracts components of a plurality of spatial frequency bands from the reference signal, adjusts a ratio between the components of the plurality of spatial frequency bands, and synthesizes the components of the plurality of spatial frequency bands whose ratios have been adjusted, thereby generating the contour enhancement signal.

[0058] (Item 13) An image processing device according to any one of items 1 to 12, further comprising a nonlinear correction unit that performs correction processing on the signal combined by the combining unit in accordance with the nonlinear characteristics of a display device connected to the image processing device.

[0059] (Item 14) A light source device; an imaging device that generates the plurality of image signals by capturing an image of a subject illuminated by the light source device; An image processing device according to any one of items 1 to 13, An endoscope apparatus comprising:

[0060] (Item 15) generating a plurality of gradation correction signals by converting the gradation characteristics of a plurality of image signals; generating an edge-enhanced signal by performing a predetermined processing on at least one image signal among the plurality of image signals; a step of combining a signal based on the plurality of gradation correction signals with the edge emphasis signal, an image processing method, characterized in that in the step of generating a plurality of gradation correction signals, at least two gradation correction signals are generated by converting the gradation characteristics of at least two image signals among the plurality of image signals while maintaining a ratio of pixel values ​​of corresponding pixels of at least two image signals.

[0061] (Item 16) the plurality of image signals include an image signal of visible light, and an image signal of special light including at least one of invisible light, fluorescent light caused by irradiation with excitation light, and reflected light caused by irradiation with narrowband light having a half-width of 20 μm or less; Item 16. The image processing method according to item 15, characterized in that in the step of generating the plurality of gradation correction signals, corresponding gradation correction signals are generated from among the plurality of gradation correction signals obtained by converting the gradation characteristics of the visible light image signal and the special light image signal while maintaining a ratio of pixel values ​​of corresponding pixels of the visible light image signal and the special light image signal.

[0062] (Item 17) 17. A program for causing a computer to execute each step of the image processing method according to item 15 or 16.

[0063] (Item 18) Item 18. A computer-readable storage medium storing the program according to item 17.

[0064] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0065] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0066] 100: image processing device, 152: tone correction section, 153: edge enhancement section, 157: synthesis section

Claims

1. At least one processor that performs tone correction processing for generating a plurality of tone correction signals obtained by converting the tone characteristics of a plurality of image signals in different wavelength bands, edge enhancement processing for generating an edge enhancement signal by performing a predetermined process on at least one of the plurality of image signals, and combining processing for combining a signal based on the plurality of tone correction signals and the edge enhancement signal. In the tone correction processing, at least two tone correction signals obtained by converting the tone characteristics of at least two of the plurality of image signals are generated while maintaining the ratio of the pixel values of the corresponding pixels of the at least two image signals. The image processing apparatus, wherein the tone correction processing and the edge enhancement processing are independently executed.

2. The image processing apparatus according to claim 1, wherein in the tone correction processing, the plurality of tone correction signals obtained by converting the tone characteristics of the plurality of image signals are generated while maintaining the ratio of the pixel values of the corresponding pixels of all the image signals of the plurality of image signals.

3. In the tone correction processing, A gain value is determined based on at least one of the plurality of image signals. The image processing apparatus according to claim 2, wherein the plurality of tone correction signals are generated by applying the gain value to each of the plurality of image signals.

4. The image processing apparatus according to claim 3, wherein the plurality of image signals include an image signal not used for determining the gain value in the tone correction processing.

5. The image processing apparatus according to claim 3, wherein in the tone correction processing, the gain value is determined based on the maximum pixel value among the pixel values included in at least one of the plurality of image signals.

6. The pixel values included in each of the plurality of image signals have a signal range from the lowest luminance value to the highest luminance value, The gain value is, maximum when the maximum pixel value is in the range from the lowest luminance value to 10% of the signal range, The image processing apparatus according to claim 5, wherein when the maximum pixel value is in the range from the pixel value at which the gain value becomes maximum to 50% of the signal range, the gain value monotonically decreases from the pixel value at which the gain value becomes maximum to 50% of the signal range.

7. In the gradation correction process, the gain value is calculated for an input of at least one of the plurality of image signals, The image processing apparatus according to claim 3, wherein input / output characteristics when calculating the gain value for an input of at least one of the plurality of image signals are configured to be changeable according to a user setting.

8. The image processing apparatus according to claim 3, wherein in the gradation correction process, the gain value is determined based on an image signal of visible light among the plurality of image signals.

9. The image processing apparatus according to claim 3, wherein in the gradation correction process, an image signal of special light including at least one of non-visible light, fluorescence by excitation light irradiation, and reflected light by narrow-band light irradiation with a half-value width of 20 μm or less among the plurality of image signals is not used for determination of the gain value.

10. The image processing apparatus according to claim 9, wherein in the gradation correction process, corresponding gradation correction signals are generated among the plurality of gradation correction signals in which the gradation characteristics of the visible light image signal and the special light image signal are converted while maintaining the ratio of the pixel values of the corresponding pixels of the visible light image signal and the special light image signal among the plurality of image signals.

11. In the contour enhancement process, a reference signal is generated based on at least one of the plurality of image signals, and the contour enhancement signal is generated by extracting components in one or more spatial frequency bands from the reference signal. The image processing apparatus according to claim 1, characterized in that.

12. In the contour enhancement process, components in a plurality of spatial frequency bands are extracted from the reference signal, the ratio between the components in the plurality of spatial frequency bands is adjusted, and the components in the plurality of spatial frequency bands with the adjusted ratio are synthesized. Thus, the contour enhancement signal is generated. The image processing apparatus according to claim 11, characterized in that.

13. The at least one processor further performs correction processing according to the non-linear characteristics of a display device connected to the image processing apparatus on the signal synthesized by the synthesis processing. The image processing apparatus according to claim 1, characterized in that.

14. The gradation correction process and the contour enhancement process are independently executed on one signal. The image processing apparatus according to claim 1, characterized in that.

15. The gradation correction process and the contour enhancement process are independently executed by inputting the at least one image signal in parallel for the gradation correction process and the contour enhancement process. The image processing apparatus according to claim 1, characterized in that.

16. A light source device, An imaging device that generates the plurality of image signals by imaging a subject illuminated by the light source device, The image processing apparatus according to any one of claims 1 to 15, An endoscope device characterized by comprising.

17. A step of generating a plurality of gradation correction signals obtained by converting the gradation characteristics of a plurality of image signals, A step of generating a contour enhancement signal by performing predetermined processing on at least one of the plurality of image signals, A step of synthesizing a signal based on the plurality of gradation correction signals and the contour enhancement signal, and In the step of generating the plurality of gradation correction signals, at least two gradation correction signals are generated by converting the gradation characteristics of at least two of the plurality of image signals while maintaining the ratio of the pixel values of the corresponding pixels of the at least two image signals. An image processing method, characterized in that the gradation correction processing in the step of generating the plurality of gradation correction signals and the edge enhancement processing in the step of generating the edge enhancement signal are executed independently.

18. The plurality of image signals include an image signal of visible light and an image signal of special light including at least one of non-visible light, fluorescence by excitation light irradiation, and reflected light by narrow-band light irradiation with a half-value width of 20 μm or less. The image processing method according to claim 17, characterized in that, in the step of generating the plurality of gradation correction signals, corresponding gradation correction signals among the plurality of gradation correction signals obtained by converting the gradation characteristics of the visible light image signal and the special light image signal are generated while maintaining the ratio of the pixel values of the corresponding pixels of the visible light image signal and the special light image signal.

19. A program for causing a computer to execute each step of the image processing method according to claim 17 or 18.

20. A computer-readable storage medium storing the program according to claim 19.

Images