Image processing device, endoscope device, image processing method, program and storage medium

Abstract

To provide a technique advantageous for improvement in the effect of image processing.SOLUTION: An image processing device comprises: a gradation correction unit which generates a plurality of gradation correction signals obtained by converting the gradation characteristics of a plurality of image signals in mutually-different wavebands; a contour emphasis unit which generates a contour emphasis signal by performing prescribed processing on at least one image signal in the plurality of image signals; and a synthesis unit which synthesizes the signal based on the plurality of gradation correction signals and the contour emphasis signal. The gradation correction unit generates at least two gradation correction signals obtained by converting the gradation characteristics of at least two image signals while keeping a ratio of pixel values of corresponding pixels of the at least two image signals in the plurality of image signals.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an endoscope apparatus, an image processing method, a program, and a storage medium.

Background Art

[0002] In an image processing apparatus, various signal processes are performed for the purpose of improving image quality. In Patent Document 1, it is shown that in a video signal processing apparatus that performs edge enhancement processing and gradation conversion processing, the edge enhancement processing is performed according to the characteristics of the gradation conversion processing.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] It is conceivable to strongly apply both edge enhancement processing and gradation conversion processing to a high-contrast video such as an image in which the edges are blurred and there are black crush or white clip. When performing edge enhancement processing according to the characteristics of the gradation conversion processing, since it is affected by the characteristics of the gradation conversion processing, it may not be possible to perform appropriate edge enhancement processing.

[0005] An object of the present invention is to provide a technique advantageous for improving the effect of image processing.

Means for Solving the Problems

[0006] In view of the above problems, an image processing apparatus according to an embodiment of the present invention comprises at least one processor that performs: a tone correction process that generates a plurality of tone correction signals obtained by converting the tone characteristics of a plurality of image signals in different wavelength bands; a contour enhancement process that generates a contour enhancement signal by performing a predetermined process on at least one of the plurality of image signals; and a synthesis process that synthesizes a signal based on the plurality of tone correction signals and the contour enhancement signal, wherein in the tone correction process, at least two tone correction signals are generated in which the tone characteristics of the at least two image signals are converted while maintaining the ratio of the pixel values ​​of corresponding pixels of at least two of the plurality of image signals. The at least one image signal is input in parallel for the tone correction process and the edge enhancement process, The gradation correction process and the edge enhancement process are characterized by being performed independently. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a technology that is advantageous in improving the effectiveness of image processing. [Brief explanation of the drawing]

[0008] [Figure 1] A diagram showing an example configuration of the image processing apparatus according to this embodiment. [Figure 2] Figure 1 shows an example of the configuration of the grayscale correction unit of the image processing device. [Figure 3] This figure shows an example configuration of an endoscope device using the image processing device according to this embodiment. [Figure 4] Figure 3 shows an example of the configuration of an image processing device. [Figure 5] Figure 3 shows an example of the LUT characteristics of the grayscale correction unit of the image processing device. [Figure 6] Figure 3 shows an example of the input / output characteristics of the grayscale correction unit of the image processing device. [Modes for carrying out the invention]

[0009] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0010] An image processing apparatus according to an embodiment of the present disclosure will be described with reference to Figures 1 to 6. Figure 1 is a diagram showing an example configuration of the image processing apparatus 100 of this embodiment. The image processing apparatus 100 includes a tone correction unit 152, an edge enhancement unit 153, and a synthesis unit 157. The tone correction unit 152 generates a plurality of tone correction signals 211 and 212 by converting the tone characteristics of a plurality of image signals 201 and 202 in different wavelength bands. In the configuration shown in Figure 1, two image signals 201 and 202 are input to the tone correction unit 152. The edge enhancement unit 153 generates an edge enhancement signal 221 by performing a predetermined process on at least one of the plurality of image signals 201 and 202. The tone correction signals 211 and 212 generated by the tone correction unit 152 and the edge enhancement signal 221 generated by the edge enhancement unit 153 are input to the synthesis unit 157. The combining unit 157 generates combined signals 231 and 232 by combining multiple gradation correction signals 211 and 212 with the edge enhancement signal 221.

[0011] As shown in Figure 1, the image signal 201 is input to the tone correction unit 152 and the edge enhancement unit 153 in parallel. Therefore, tone correction processing is performed on the image signal 201 input to the image processing device 100 by the tone correction unit 152. In addition, edge enhancement processing is performed on the image signal 201 input to the image processing device 100 by the edge enhancement unit 153 without being affected by the noise enhancement or contrast reduction caused by the tone correction processing. Furthermore, by combining the edge enhancement signal 221 with the tone correction signals 211 and 212 after tone correction processing, the local contrast in areas where the contrast has decreased due to the compression processing during tone correction can be improved.

[0012] Figure 2 shows an example configuration of the tone correction unit 152. The tone correction unit 152 includes a gain multiplication unit 521 and a gain calculation unit 522. Two input image signals 201 and 202 are input to the gain calculation unit 522 and the gain multiplication unit 521 in parallel. Based on the two input image signals 201 and 202, the gain calculation unit 522 calculates a gain value. Details of the gain value calculation will be described later. The calculated gain value is input to the gain multiplication unit 521. The gain multiplication unit 521 applies the gain value calculated by the gain calculation unit 522 to the two input image signals 201 and 202, respectively, and outputs tone correction signals 211 and 212. Since the same gain value is applied to the image signals 201 and 202 in the gain multiplication unit 521, the two tone correction signals 211 and 212 are generated while maintaining the ratio of the pixel values ​​of the corresponding pixels in the two image signals 201 and 202. It can also be said that the tone correction unit 152 generates multiple tone correction signals 211 and 212 by converting the tone characteristics of the two image signals 201 and 202 while maintaining the ratio of the pixel values ​​of the corresponding pixels in each of the two image signals 201 and 202. Because the tone correction unit 152 converts the tone characteristics while maintaining the ratio of the pixel values ​​of the corresponding pixels in the multiple image signals 201 and 202, tone correction processing is performed without changing the hue or saturation.

[0013] Here, the image signals 201 and 202 input to the image processing device 100 are signals in different wavelength bands that capture the same subject. Image signals 201 and 202 may be image signals in different wavelength bands obtained in a single image capture. Alternatively, for example, image signals 201 and 202 may be image signals continuously captured by irradiating with illumination in different wavelength bands while switching between them. For example, image signals 201 and 202 may be image signals in different wavelength bands such as red, green, and blue. Alternatively, for example, image signal 201 may be a visible light image signal, and image signal 202 may be a non-visible light image signal. As a more specific example, an endoscope device equipped with the image processing device 100 of this embodiment will be given, and further details of the image processing device 100 will be described below.

[0014] Figure 3 shows an example configuration of an endoscope device 101 equipped with the image processing device 100 of this embodiment. The endoscope device 101 includes a light source device 170, an imaging device 102 that generates multiple image signals by imaging a subject illuminated by the light source device 170, a camera control unit 140 including the image processing device 100 described above, and a display device 180. The imaging device 102 includes a scope 110, a coupler lens 120, and a camera head 130.

[0015] The scope 110 can be inserted, for example, into the body of a subject. Light emitted from the light source device 170 is irradiated onto the subject via the illumination optical system 112 arranged in the scope 110. The reflected light and fluorescence generated by the irradiated light are imaged onto image sensors 132-135 via the imaging optical system 111, coupler lens 120, and color separation element 131 arranged in the camera head 130. The image signals output from each of the image sensors 132-135 are output from the camera head 130 via the transmission unit 136 and input to the camera control unit 140. In the camera control unit 140, the input image signals are reconstructed by the receiving unit 141, and the reconstructed image signals obtained from the image sensors 132-135 are input to the image processing device 100. The image signal processed by the image processing device 100 is transferred from the camera control unit 140 to the display device 180 via the transmission unit 143, and the image (video) is displayed on the display device 180.

[0016] The operations of the camera head 130, the light source device 170, and the image processing device 100, such as settings, can be performed by the user operating an input unit such as a graphical user interface (GUI) 144 provided in the camera control unit 140. The control unit 160 provided in the camera control unit 140 receives the settings input by the user via the GUI 144, and the processing in the image processing device 100 and the control of the camera head 130 are appropriately changed. For example, when the control content of the operation of the camera head 130 is changed, a control signal is transferred from the control unit 160 of the camera control unit 140 to the control unit 137 provided in the camera head 130, and the control of the operations of the imaging elements 132 to 135 is changed according to the control signal.

[0017] A flash memory 138 may be provided in the camera head 130. The flash memory 138 may store programs related to the operation and control of the camera head 130, adjustment parameters for correcting individual differences of the camera head 130, and the like.

[0018] A flash memory 142 may be provided in the camera control unit 140. Programs related to the operation and control of the camera control unit 140, values set from the outside such as by the user, and the like may be stored. Also, the flash memories 138 and 142 may be other storage devices such as an HDD. <~

[0019] In the present embodiment, the light source device 170 includes three types of light emitting diodes (LEDs) 172 to 174 and a laser diode (LD) 175. The light source device 170 combines the emitted light of the LEDs 172 to 174 and the LD 175 with an illumination optical system 171 provided in the light source device 170, and outputs it to the illumination optical system 112 provided in the scope 110. The control unit 160 provided in the camera control unit 140 outputs a control signal to the control unit 176 provided in the light source device 170 as necessary, and the light source device 170 adjusts the type, timing, and output level of the light source to be lit according to the control signal.

[0020] For example, LED 172 may emit light in the red wavelength band, LED 173 may emit light in the green wavelength band, and LED 174 may emit light in the blue wavelength band. Also, for example, LD 175 may emit light at the infrared excitation wavelength of the fluorescent reagent. Hereinafter, the light emitted by LD 175, which is not general visible light, may be referred to as special light. Also, the image signal obtained by irradiating these special lights may be referred to as the image signal of special light. Examples of the image signal of special light include image signals obtained by non-visible light, fluorescence by excitation light irradiation, and reflected light by narrow-band light irradiation with a full width at half maximum of 20 μm or less. In the present embodiment, the image signal of special light is an image signal obtained from fluorescence excited by infrared excitation light (LD 175). In this case, the reflected light of the infrared excitation light can be removed by a wavelength cut filter (not shown).

[0021] Each wavelength decomposed by the color separation element 131 corresponds to the wavelengths emitted by LEDs 172 to 174 and LD 175. The color separation element 131 inputs light in the red wavelength band to the imaging element 132, light in the green wavelength band to the imaging element 133, light in the blue wavelength band to the imaging element 134, and light in the wavelength band of fluorescence excited by infrared excitation light to the imaging element 135. The outputs of the imaging elements 132 to 135 are transferred to the image processing apparatus 100 as signals R_in, G_in, B_in, and IR_in.

[0022] FIG. 4 shows a configuration example of the image processing apparatus 100 arranged in the camera control unit 140 of the present embodiment. The image processing apparatus 100 includes the above-described tone correction unit 152, edge enhancement unit 153, and synthesis unit 157. Also, in the configuration shown in FIG. 4, the image processing apparatus 100 further includes a pre-stage image processing unit 151, luminance processing unit 154, color processing unit 155, special light processing unit 156, color synthesis unit 158, and image output unit 159.

[0023] The signals R_in, G_in, B_in, and IR_in input to the image processing device 100 are first processed by the pre-image processing unit 151. The pre-image processing unit 151 performs, for example, digital gain processing, linear matrix processing, defective pixel correction processing, and shading correction processing on each of the signals R_in, G_in, B_in, and IR_in.

[0024] The output of the pre-processing unit 151 is transferred to the tone correction unit 152 and the edge enhancement unit 153. Therefore, the image signals R_f, G_f, B_f, and IR_f, which are the output of the pre-processing unit 151, correspond to the image signals 201 and 202 mentioned above. In the configuration shown in Figure 4, the image signals R_f, G_f, B_f, and IR_f are transferred to the tone correction unit 152. In addition, the image signals R_f, G_f, and B_f are transferred not only to the tone correction unit 152 but also to the edge enhancement unit 153 in parallel. Therefore, in the configuration shown in Figure 4, the image signals R_f, G_f, and B_f correspond to the image signal 201 mentioned above, and the image signal IR_f corresponds to the image signal 202 mentioned above.

[0025] The tone correction unit 152 determines a gain value based on at least one of the multiple image signals R_f, G_f, B_f, and IR_f, and applies the gain value to each of the image signals R_f, G_f, B_f, and IR_f. This allows the tone correction unit 152 to generate tone correction signals R_g, G_g, B_g, and IR_g, which correspond to the tone correction signals 211 and 212 mentioned above. In the configuration shown in Figure 4, three image signals, R_f, G_f, and B_f, which are part of the signals input to the tone correction unit 152, are input to the MAX circuit 523 located in the gain calculation unit 522 of the tone correction unit 152. The MAX circuit 523 obtains the largest pixel value among the pixel values ​​included in the input image signals R_f, G_f, and B_f. The signal max, which indicates the largest acquired pixel value, is input to the LUT 524 located in the gain calculation unit 522, and the signal gain, which indicates the gain value, is transferred as an output to the gain multiplication unit 521. In this way, the tone correction unit 152 determines the gain value based on the largest pixel value among the pixel values ​​contained in at least one of the multiple image signals R_f, G_f, B_f, and IR_f. In this case, unlike the configuration shown in Figure 2, the multiple image signals R_f, G_f, B_f, and IR_f may include the image signal IR_f, which is not used by the tone correction unit 152 to determine the gain value. For example, the tone correction unit 152 may determine the gain value based on the visible light image signals R_f, G_f, and B_f among the multiple image signals R_f, G_f, B_f, and IR_f, as shown in the configuration shown in Figure 4. Furthermore, the gradation correction unit 152 does not necessarily have to use the special light image signal IR_f, which is one of the multiple image signals R_f, G_f, B_f, and IR_f, in determining the gain value.

[0026] The gain multiplication unit 521 applies the gain value calculated by the gain calculation unit 522 to the image signals R_f, G_f, B_f, and IR_f. For example, the gain multiplication unit 521 may apply the gain value to the visible light image signals R_f, G_f, and B_f. This generates tone correction signals R_g, G_g, and B_g, which are obtained by converting the tone characteristics of the image signals R_f, G_f, and B_f while maintaining the ratio of the pixel values ​​of the corresponding pixels in each of the image signals R_f, G_f, and B_f. In this way, the tone correction unit 152 may generate at least two tone correction signals, which are obtained by converting the tone characteristics of at least two image signals while maintaining the ratio of the pixel values ​​of the corresponding pixels in at least two of the multiple image signals R_f, G_f, B_f, and IR_f.

[0027] Furthermore, the gain multiplication unit 521 may uniformly apply the gain value calculated by the gain calculation unit 522 to each of the image signals R_f, G_f, B_f, and IR_f, as shown in Figure 4. In other words, the tone correction unit 152 may generate multiple tone correction signals R_g, G_g, B_g, and IR_g by converting the tone characteristics of the multiple image signals R_f, G_f, B_f, and IR_f while maintaining the ratio of the pixel values ​​of the corresponding pixels for all of the multiple image signals R_f, G_f, B_f, and IR_f.

[0028] The gradation correction signals R_g, G_g, and B_g are transferred in parallel to the luminance processing unit 154 and the color processing unit 155, respectively. The gradation correction signal IR_g is transferred to the special light processing unit 156.

[0029] The image signals R_f, G_f, and B_f input to the contour enhancement unit 153 are transferred to the contour reference signal generation unit 531. For example, the contour reference signal generation unit 531 may generate a Y signal from the image signals R_f, G_f, and B_f (RGB signals) and use it as the reference signal Db. Alternatively, the contour reference signal generation unit 531 may generate an average signal of the image signals R_f and G_f and use it as the reference signal Db. Furthermore, the contour reference signal generation unit 531 may use the image signal G_f from the input image signals R_f, G_f, and B_f as the reference signal. The reference signal Db, which is the output of the contour reference signal generation unit 531, is transferred to the noise suppression unit 532. The noise suppression unit 532 performs processes on the reference signal Db, such as frame cyclic processing, filtering, or a combination thereof. The reference signal Db, which has been noise-suppressed by the noise suppression unit 532, is transferred to the band extraction and amplification unit 533 as the reference signal Dn.

[0030] The band extraction and amplification unit 533 extracts components of one or more spatial frequency bands from the reference signal Dn to be emphasized as contour signals. The band extraction and amplification unit 533 adjusts the gain for each extracted spatial frequency band, synthesizes the components of each band as needed, and outputs it as a contour enhancement signal Dtl (corresponding to the contour enhancement signal 221 described above). In this way, the contour enhancement unit 153 generates a reference signal Dn based on at least one image signal from among multiple image signals R_f, G_f, B_f, and IR_f, and generates a contour enhancement signal Dtl by extracting components of one or more spatial frequency bands from the reference signal.

[0031] The contour enhancement unit 153 may divide the reference signal Db, which is the output of the contour reference signal generation unit 531, into a plurality of signals (for example, reference signals Db1, Db2) according to the signal amplitude, and perform noise suppression processing and band extraction / amplification processing with different characteristics on each. The contour enhancement unit 153 may combine these plurality of outputs after the band extraction / amplification processing and output them as a contour enhancement signal Dtl. In other words, the contour enhancement unit 153 may generate a contour enhancement signal Dtl by extracting components of a plurality of spatial frequency bands from the reference signal Db, adjusting the ratio between the components of the plurality of spatial frequency bands, and combining the components of the plurality of spatial frequency bands with adjusted ratios.

[0032] The luminance processing unit 154 generates a Y signal (luminance signal Y_p) based on the input gradation correction signals R_g, G_g, B_g (RGB signals). Before and after generating the luminance signal Y_p, the luminance processing unit 154 may perform, for example, noise suppression processing or bandwidth limiting processing. The color processing unit 155 generates chrominance signals Pb and Pr based on the input gradation correction signals R_g, G_g, B_g (RGB signals). Before and after generating the chrominance signals Pb and Pr, the color processing unit 155 may perform, for example, color matrix processing, chroma gain processing, color shift suppression processing, noise suppression processing, bandwidth limiting processing, etc. In the configuration shown in Figure 4, the luminance processing unit 154 and the color processing unit 155 are shown as separate configurations, but their respective processes may be performed as an integrated configuration. The configurations are simply shown separately so that the respective processes being performed are clear. The same applies to other configurations.

[0033] As described above, the tone correction signal IR_g is transferred from the tone correction unit 152 to the special light processing unit 156. The special light processing unit 156 performs processes such as noise suppression, bandwidth limiting, tone correction, shading correction, and offset adjustment on the tone correction signal IR_g and outputs the signal IR_p.

[0034] The combining unit 157 receives the luminance signal Y_p output from the luminance processing unit 154 and the contour enhancement signal Dtl output from the contour enhancement unit 153 as inputs. The combining unit 157 combines the luminance signal Y_p generated by the luminance processing unit 154 based on multiple gradation correction signals R_g, G_g, and B_g generated by the gradation correction unit 152, with the contour enhancement signal Dtl generated by the contour enhancement unit 153. As a combining method, for example, the luminance signal Y_p and the contour enhancement signal Dtl may be simply added together. The combining unit 157 outputs a luminance signal Y_d obtained by contour enhancing the luminance signal Y_p.

[0035] The color synthesis unit 158 ​​receives the luminance signal Y_d from the synthesis unit 157, the color difference signals Pb_p and Pr_p from the color processing unit 155, and the signal IR_p from the special light processing unit. In normal visible light display modes, the input luminance signal Y_d and color difference signals Pb_p and Pr_p can be directly output by the color synthesis unit 158. In special light display modes, the color synthesis unit 158 ​​performs synthesis processing so that the image (video) obtained by the special light is visualized or enhanced. For example, the color synthesis unit 158 ​​performs processing such as changing the color based on the ratio of the luminance signal Y_p output from the luminance processing unit 154, the gradation correction signal IR_p output from the gradation correction unit 152, and the pixel values.

[0036] In the image output unit 159, signals Y_c, Pb_c, and Pr_c are transferred from the color synthesis unit 158. The image output unit 159 may perform processing such as RGB conversion and signal value range limitation to match the output signal specifications of the camera control unit 140. As described above, the luminance signal Y_p and the edge enhancement signal Dtl based on the gradation correction signals R_g, G_g, and B_g are synthesized as a luminance signal Y_d by the synthesis unit 157. Based on this luminance signal Y_d, the color difference signals Pb_p and Pr_p generated by the color processing unit 155, and the signal Ir_p generated by the special light processing unit 156, the color synthesis unit 158 ​​generates signals Y_c, Pb_c, and Pr_c. A nonlinear correction unit may be located in the image output unit 159 that performs correction processing, such as gamma correction, on the signals Y_c, Pb_c, and Pr_c based on the signals synthesized by the synthesis unit 157, according to the nonlinear characteristics of the display device 180 connected to the image processing device 100 (camera control unit 140). After these processes are performed, signals R_out, G_out, and B_out are output from the image processing device 100. Signals R_out, G_out, and B_out are transferred to the display device 180 via the transmission unit 143, and an image (video) is displayed on the display device 180.

[0037] Figure 5 shows an example of the characteristics of the LUT 524 of the gradation correction unit 152 in this embodiment. The black signal of the display device 180 is set to 0IRE, the reference white signal of the display device 180 is set to 100IRE, and the range of signals that can be input to the display device 180 is set to 0 to 109IRE. It is also assumed that the output of the image sensors 132 to 135 and the input of the image processing device 100 range from 0 to 425IRE. In this case, in order to suppress black crushing and white clipping and achieve a gradation display with good visibility, it is necessary to raise the dark areas, suppress the bright areas, and concentrate the signal in the intermediate range. Figure 5 shows an example of the characteristics of the LUT 524 to achieve this.

[0038] In Figure 5, the horizontal axis represents the magnitude (IRE) of the input signal to the LUT524. The vertical axis represents the gain value (dB) of the output of the LUT524. As described above, the input signal to the LUT524 is the maximum pixel value (signal max) among the pixel values ​​contained in each of the image signals R_f, G_f, and B_f.

[0039] To improve black clipping, it is necessary to increase the gain in the region where the input signal is small. In the example shown in Figure 5, the gain at an input of 0IRE is used as the output peak. It may fluctuate slightly depending on the balance with noise, but the output peak may be within 10% of the maximum input range (in this embodiment, inputs from 0IRE to 42.5IRE). In other words, the pixel values ​​(signal max) contained in each of the multiple image signals R_f, G_f, and B_f have a signal range from the lowest brightness value (0IRE) to the highest brightness value (425IRE). In this case, the gain value may be within the range from the lowest brightness value (0IRE) to 10% of the signal range (42.5IRE) where the maximum pixel value (signal max) is.

[0040] Furthermore, in order to concentrate the signal in the mid-range with a natural appearance, the output may monotonically decrease from the input corresponding to the output peak up to at least 50% (in this embodiment, input up to 212.5IRE). In other words, if the maximum pixel value (signal max) is in the range from the pixel value where the gain value is maximum to 50% of the signal range (212.5IRE), the gain value may monotonically decrease from the pixel value where the gain value is maximum down to 50% of the signal range (212.5IRE). In the high-luminance area (212.5IRE to 425IRE) where the signal input (signal max) exceeds 50%, as shown in Figure 5, the gain value may continue to monotonically decrease, or it may start to increase in order to prevent a decrease in the maximum output of the gradation correction unit 152.

[0041] Figure 6 shows an example of the input / output characteristics of the tone correction unit 152 of this embodiment. Figure 6 shows some characteristics of the output signal of the tone correction unit 152 when the gain value of the LUT 524 having the characteristics shown in Figure 5 is multiplied by the image signal that forms the basis of the signal max. With respect to the signal input, the dark areas are amplified, the bright areas are suppressed, and the signal is concentrated in the mid-range. Although the contrast in the mid-range is reduced, the signal value output across the entire range increases monotonically (x <yならばf(x)≦f(y))である。

[0042] The tone correction unit 152 includes a gain calculation unit 522 that outputs a gain value for the input of at least one image signal from among a plurality of image signals. In this case, the input / output characteristics of the gain calculation unit 522 may be configured to change according to user settings. For example, the characteristics of the LUT 524 may be switched in response to a control signal from the control unit 160 in response to user input via the GUI 144. For example, the characteristics of the LUT 524 may be switched depending on whether or not the special light image signal IR_f is used. In addition, the degree of tone correction may be adjusted in various ways, such as weighting the gain value to 1 and interpolating it in response to a control signal from the control unit 160 in response to user input via the GUI 144.

[0043] In an endoscope device 101 with this configuration, grayscale correction processing is applied to the input image signals R_f, G_f, B_f, and IR_f. However, edge enhancement processing is performed without being affected by noise enhancement or contrast reduction caused by the grayscale correction processing. Furthermore, by combining the edge-enhanced edge enhancement signal Dtl with the luminance signal Y_p based on the grayscale correction signals R_g, G_g, and B_g after grayscale correction, local contrast in areas with reduced contrast can be improved. As a result, it is possible to output images that have been effectively processed to address issues that are generally problematic with endoscope devices 101, such as blurred edges, display of high-contrast scenes, and noise in dark areas. In addition, while performing grayscale correction processing and edge enhancement processing based on the image signals R_f, G_f, and B_f (RGB signals), the ratio of pixel values ​​between the image signals R_f, G_f, and B_f and the image signal IR_f can be maintained. In other words, the tone correction unit 152 generates a number of corresponding tone correction signals R_g, G_g, B_g, and IR_g by converting the tone characteristics of the visible light image signals R_f, G_f, and B_f and the special light image signal IR_f, respectively, while maintaining the ratio of the pixel values ​​of the corresponding pixels of the visible light image signals R_f, G_f, and B_f and the special light image signal IR_f, respectively.Therefore, in subsequent signal processing, appropriate processing can be performed according to the respective signal ratios of the image obtained with visible light and the image obtained with special light such as invisible light.

[0044] This disclosure enables gradation correction processing independent of edge enhancement processing, but does not negate the possibility of performing nonlinear processing or gradation correction processing on edge enhancement signals. For example, to save bits in signal processing, gamma correction curves or log curves may be applied to each signal generated by the edge enhancement unit 153, luminance processing unit 154, color processing unit 155, etc. Also, since noise increase cannot be completely eliminated even in gradation correction processing, noise reduction processing according to the degree of gradation correction processing may be performed at an appropriate timing within the image processing device 100. In the above embodiments, delay circuits for synchronizing the timing of each signal are not described, but it goes without saying that delay circuits will be used as appropriate depending on the circuit configuration.

[0045] The disclosures herein include the following image processing devices, endoscope devices, image processing methods, programs, and storage media.

[0046] (Item 1) The system includes: a tone correction unit that generates multiple tone correction signals by converting the tone characteristics of multiple image signals in different wavelength bands; a contour enhancement unit that generates a contour enhancement signal by performing a predetermined process on at least one of the multiple image signals; and a synthesis unit that synthesizes a signal based on the multiple tone correction signals and the contour enhancement signal. The image processing apparatus is characterized in that the tone correction unit generates at least two tone correction signals by converting the tone characteristics of at least two image signals while maintaining the ratio of the pixel values ​​of the corresponding pixels of at least two of the plurality of image signals.

[0047] (Item 2) The image processing apparatus according to item 1, characterized in that the tone correction unit generates the plurality of tone correction signals by converting the tone characteristics of the plurality of image signals while maintaining the ratio of the pixel values ​​of the corresponding pixels of all of the plurality of image signals.

[0048] (Item 3) The aforementioned tone correction unit, The gain value is determined based on at least one image signal from the plurality of image signals. The image processing apparatus according to item 1 or 2, characterized in that it generates the plurality of grayscale correction signals by applying the gain value to each of the plurality of image signals.

[0049] (Item 4) The image processing apparatus according to item 3, characterized in that the plurality of image signals include image signals that the grayscale correction unit does not use to determine the gain value.

[0050] (Item 5) The image processing apparatus according to item 3 or 4, characterized in that the tone correction unit determines the gain value based on the largest pixel value among the pixel values ​​included in at least one of the plurality of image signals.

[0051] (Item 6) Each of the aforementioned plurality of image signals contains a pixel value having a signal range from the lowest brightness value to the highest brightness value. The aforementioned gain value is, The maximum pixel value is reached when it falls within the range from the minimum brightness value to 10% of the signal range. The image processing apparatus according to item 5, characterized in that when the maximum pixel value is within the range from the pixel value at which the gain value is maximum to 50% of the signal range, the gain value monotonically decreases from the pixel value at which the gain value is maximum to 50% of the signal range.

[0052] (Item 7) The grayscale correction unit includes a gain calculation unit that outputs the gain value for the input of at least one image signal among the plurality of image signals. The image processing apparatus according to any one of items 3 to 6, characterized in that the input / output characteristics of the gain calculation unit are configured to be changeable according to user settings.

[0053] (Item 8) The image processing apparatus according to any one of items 3 to 7, characterized in that the grayscale correction unit determines the gain value based on the visible light image signal among the plurality of image signals.

[0054] (Item 9) The image processing apparatus according to any one of items 3 to 8, characterized in that the grayscale correction unit does not use an image signal of special light, which includes at least one of the plurality of image signals, such as invisible light, fluorescence due to excitation light irradiation, and reflected light due to narrowband light irradiation with a half width of 20 μm or less, in determining the gain value.

[0055] (Item 10) The image processing apparatus according to item 9, characterized in that the tone correction unit generates corresponding tone correction signals from among the plurality of tone correction signals obtained by converting the tone characteristics of the visible light image signal and the special light image signal, while maintaining the ratio of the pixel values ​​of the corresponding pixels of the visible light image signal and the special light image signal among the plurality of image signals.

[0056] (Item 11) The image processing apparatus according to any one of items 1 to 10, characterized in that the contour enhancement unit generates a reference signal based on at least one image signal from the plurality of image signals, and generates the contour enhancement signal by extracting components of one or more spatial frequency bands from the reference signal.

[0057] (Item 12) The image processing apparatus according to item 11, characterized in that the contour enhancement unit generates the contour enhancement signal by extracting components of a plurality of spatial frequency bands from the reference signal, adjusting the ratios between the components of the plurality of spatial frequency bands, and synthesizing the ratios of the plurality of spatial frequency bands.

[0058] (Item 13) The image processing apparatus according to any one of items 1 to 12, further comprising a nonlinear correction unit that performs correction processing on the signal synthesized by the synthesis unit according to the nonlinear characteristics of a display device connected to the image processing apparatus.

[0059] (Item 14) Light source device, An imaging device that generates the plurality of image signals by imaging a subject illuminated by the light source device, An image processing device described in any one of items 1 through 13, An endoscope device characterized by being equipped with the following features.

[0060] (Item 15) A process for generating multiple tone correction signals by converting the tone characteristics of multiple image signals, A step of generating a contour enhancement signal by performing a predetermined process on at least one of the plurality of image signals, The process includes a step of synthesizing a signal based on the plurality of grayscale correction signals and the edge enhancement signal, An image processing method characterized in that, in the step of generating the plurality of tone correction signals, at least two tone correction signals are generated by converting the tone characteristics of at least two image signals while maintaining the ratio of the pixel values ​​of the corresponding pixels of at least two of the plurality of image signals.

[0061] (Item 16) The plurality of image signals include a visible light image signal and a special light image signal that includes at least one of invisible light, fluorescence due to excitation light irradiation, and reflected light from narrowband light irradiation with a half-width of 20 μm or less. The image processing method according to item 15, characterized in that, in the step of generating the plurality of tone correction signals, a corresponding tone correction signal is generated from the plurality of tone correction signals obtained by converting the tone characteristics of the visible light image signal and the special light image signal, while maintaining the ratio of the pixel values ​​of the corresponding pixels of the visible light image signal and the special light image signal, respectively.

[0062] (Item 17) A program that causes a computer to perform each step of the image processing method described in item 15 or 16.

[0063] (Item 18) A computer-readable storage medium containing the program described in item 17.

[0064] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0065] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0066] 100: Image processing unit, 152: Tone correction unit, 153: Edge enhancement unit, 157: Synthesis unit

Claims

1. The system includes at least one processor that performs: a tone correction process that generates multiple tone correction signals by converting the tone characteristics of multiple image signals in different wavelength bands; a contour enhancement process that generates a contour enhancement signal by performing a predetermined process on at least one of the multiple image signals; and a synthesis process that combines a signal based on the multiple tone correction signals with the contour enhancement signal. In the aforementioned tone correction process, at least two tone correction signals are generated in which the tone characteristics of at least two image signals are transformed while maintaining the ratio of the pixel values ​​of the corresponding pixels of at least two of the plurality of image signals. An image processing apparatus characterized in that the gradation correction process and the edge enhancement process are performed independently by inputting the at least one image signal in parallel for the gradation correction process and the edge enhancement process.

2. The image processing apparatus according to Claim 1, wherein the gradation correction process generates the at least two gradation correction signals by applying a common gain value to convert the gradation characteristics of the at least two image signals while maintaining the ratio of the pixel values ​​of the corresponding pixels of the at least two image signals.

3. The image processing apparatus according to claim 2, characterized in that the common gain value changes in accordance with at least one of the at least two image signals.

4. The image processing apparatus according to claim 1, characterized in that the gradation correction process generates a plurality of gradation correction signals in which the gradation characteristics of the plurality of image signals are converted while maintaining the ratio of the pixel values ​​of the corresponding pixels of all of the plurality of image signals.

5. In the aforementioned tone correction process, The gain value is determined based on at least one image signal from the aforementioned plurality of image signals. The image processing apparatus according to claim 4, characterized in that the plurality of grayscale correction signals are generated by applying the gain value to each of the plurality of image signals.

6. The image processing apparatus according to claim 5, characterized in that the plurality of image signals include image signals not used in determining the gain value in the grayscale correction process.

7. The image processing apparatus according to claim 5, characterized in that the gain value is determined based on the largest pixel value among the pixel values ​​included in at least one of the plurality of image signals in the grayscale correction process.

8. Each of the aforementioned plurality of image signals contains a pixel value having a signal range from the lowest brightness value to the highest brightness value. The aforementioned gain value is, The maximum pixel value is reached when it falls within the range from the minimum brightness value to 10% of the signal range. The image processing apparatus according to claim 7, characterized in that when the maximum pixel value is located within the range from the pixel value at which the gain value is maximum to 50% of the signal range, the gain value monotonically decreases from the pixel value at which the gain value is maximum to 50% of the signal range.

9. In the aforementioned gradation correction process, the gain value is calculated for the input of at least one image signal among the plurality of image signals. The image processing apparatus according to claim 5, characterized in that the input / output characteristics when calculating the gain value for at least one image signal among the plurality of image signals are configured to be changeable according to user settings.

10. The image processing apparatus according to claim 5, characterized in that the gain value is determined based on the visible light image signal among the plurality of image signals in the grayscale correction process.

11. The image processing apparatus according to claim 5, characterized in that, in the grayscale correction process, an image signal of special light including at least one of the plurality of image signals, such as invisible light, fluorescence due to excitation light irradiation, and reflected light from narrowband light irradiation with a half-width of 20 μm or less, is not used to determine the gain value.

12. The image processing apparatus according to claim 11, characterized in that, in the tone correction process, while maintaining the ratio of the pixel values ​​of the corresponding pixels of the visible light image signal and the special light image signal among the plurality of image signals, a corresponding tone correction signal is generated from the plurality of tone correction signals, which are obtained by converting the tone characteristics of the visible light image signal and the special light image signal.

13. The image processing apparatus according to claim 1, characterized in that, in the contour enhancement process, a reference signal is generated based on at least one image signal among the plurality of image signals, and the contour enhancement signal is generated by extracting components of one or more spatial frequency bands from the reference signal.

14. The image processing apparatus according to claim 13, characterized in that, in the contour enhancement process, components of a plurality of spatial frequency bands are extracted from the reference signal, the ratios between the components of the plurality of spatial frequency bands are adjusted, and the contour enhancement signal is generated by combining the components of the plurality of spatial frequency bands with adjusted ratios.

15. The image processing apparatus according to claim 1, wherein the at least one processor further performs a correction process on the signal synthesized by the synthesis process according to the nonlinear characteristics of a display device connected to the image processing apparatus.

16. The image processing apparatus according to claim 1, characterized in that the grayscale correction process and the edge enhancement process are performed independently for a single signal.

17. Light source device, An imaging device that generates the plurality of image signals by imaging a subject illuminated by the light source device, An image processing apparatus according to any one of claims 1 to 16, An endoscope device characterized by being equipped with the following features.

18. An image processing method performed by an image processing device, A process for generating multiple tone correction signals by converting the tone characteristics of multiple image signals, A step of generating a contour enhancement signal by performing a predetermined process on at least one of the plurality of image signals, The process includes a step of synthesizing a signal based on the plurality of grayscale correction signals and the edge enhancement signal, In the process of generating the plurality of tone correction signals, at least two tone correction signals are generated by converting the tone characteristics of at least two image signals while maintaining the ratio of the pixel values ​​of the corresponding pixels of at least two of the plurality of image signals. An image processing method characterized in that the tone correction processing in the step of generating the plurality of tone correction signals and the edge enhancement processing in the step of generating the edge enhancement signal are performed independently by inputting at least one image signal in parallel for the tone correction processing and the edge enhancement processing.

19. The plurality of image signals include a visible light image signal and a special light image signal that includes at least one of invisible light, fluorescence due to excitation light irradiation, and reflected light from narrowband light irradiation with a half-width of 20 μm or less. The image processing method according to claim 18, characterized in that, in the step of generating the plurality of tone correction signals, a corresponding tone correction signal is generated from the plurality of tone correction signals obtained by converting the tone characteristics of the visible light image signal and the special light image signal, while maintaining the ratio of the pixel values ​​of the corresponding pixels of the visible light image signal and the special light image signal, respectively.

20. A program for causing a computer to perform each step of the image processing method described in claim 18 or 19.

21. A computer-readable storage medium storing the program described in claim 20.

Images