Image processing apparatus and image processing method

Abstract

An image processing device is provided with a plurality of signal processing units (10), a second control signal generation unit that generates a second control signal on the basis of a first control signal generated by the signal processing units (10), and an image synthesis unit, the signal processing units (10) having a first digital processing unit (100) and a second digital processing unit (200) for image adjustment. The second digital processing unit (200) has a defect detection unit (210) that detects a defective pixel, and a defective pixel adaptive processing unit (240) that generates a first control signal that specifies a synthesis coefficient to the second control signal generation unit on the basis of the detection result of the defective pixel. A defective pixel adaptive processing unit (240) synthesizes a plurality of images subjected to image adjustment by a first digital processing unit (100) of each of a plurality of signal processing units (10), and when a defective pixel is detected, generates a first control signal such that an image in which the defective pixel remains is included in the plurality of images.

Description

Technical Field

[0001] The present disclosure relates to an image processing device and an image processing method. Background Art

[0002] As a means of expanding the dynamic range of an image signal, a means of synthesizing a plurality of images with different sensitivities to expand the dynamic range is known. However, when a defect occurs in an image signal during image synthesis, there is a problem that the image synthesis cannot be performed correctly due to the influence of the defect signal.

[0003] Patent document 1 discloses a method of arranging a processing unit that performs defect correction on each frame with different sensitivities before synthesis, performing defect correction before synthesis, and synthesizing images using image signals after the defect correction. In this case, synthesis can be performed in a state where the image signal has no defect, so synthesis can be performed correctly.

[0004] Prior Art Literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2015-80152 Summary of the invention

[0007] Problems to be solved by the invention

[0008] However, in the method proposed in the above patent document, since a processing unit is provided to perform defect correction individually for each frame, there is a problem that the circuit scale becomes large and the power consumption also increases accordingly.

[0009] Therefore, the present disclosure provides an image processing device or the like that can reduce the circuit scale and power consumption and perform defect (blemish) correction.

[0010] Means for solving problems

[0011] The image processing device disclosed in the present invention comprises: a plurality of signal processing units for performing signal processing on a plurality of image signals; a second control signal generating unit for generating a second control signal based on the first control signal generated by the signal processing unit; and an image synthesis unit, wherein the signal processing unit comprises a first digital processing unit for performing image adjustment, and a second digital processing unit, wherein the second digital processing unit comprises a defect detection unit for detecting defective pixels (bad pixels, defective pixels), and a defective pixel adaptive processing unit, wherein the defective pixel adaptive processing unit generates the first control signal for specifying a synthesis coefficient to the second control signal generating unit based on the detection result of the defective pixels, and the image synthesis unit synthesizes a plurality of images after the image adjustment is performed by the first digital processing unit of each of the plurality of signal processing units through the synthesis coefficient indicated by the second control signal, and when a defective pixel is detected, the plurality of images include an image in which the defective pixel remains.

[0012] The image processing method disclosed in the present invention is executed by an image processing device, and includes: a plurality of signal processing steps, performing signal processing on a plurality of image signals; a second control signal generating step, generating a second control signal based on the first control signal generated in the signal processing step; and an image synthesis step, wherein the signal processing step includes a first digital processing step for performing image adjustment, and a second digital processing step, wherein the second digital processing step includes a defect detection step for detecting defective pixels, and a defective pixel adaptive processing step, wherein the defective pixel adaptive processing step generates the first control signal for specifying a synthesis coefficient used in the second control signal generating step based on a detection result of the defective pixels, wherein in the image synthesis step, a plurality of images after the image adjustment is performed in the first digital processing step of each of the plurality of signal processing steps are synthesized by the synthesis coefficient represented by the second control signal, and when a defective pixel is detected, the plurality of images include an image in which a defective pixel remains.

[0013] In addition, these general or specific methods can also be implemented through systems, methods, integrated circuits, computer programs, or computer-readable recording media such as CD-ROMs, or through any combination of systems, methods, integrated circuits, computer programs, and recording media.

[0014] Effects of the Invention

[0015] According to an image processing device and the like according to one embodiment of the present disclosure, it is possible to perform defect correction while reducing circuit scale and power consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a block diagram showing a configuration example of an image processing device according to the first embodiment.

[0017] Figure 2 This is a diagram showing a configuration example of a signal processing unit according to the first embodiment.

[0018] Figure 3 This is a diagram showing a first video signal and a second video signal according to the first embodiment.

[0019] Figure 4 This is a diagram showing the relationship between the first video signal and the second video signal according to the first embodiment.

[0020] Figure 5 This is a diagram showing a synthesized signal according to the first embodiment.

[0021] Figure 6 This is a diagram for explaining the synthesis coefficient involved in the first embodiment.

[0022] Figure 7 This is a diagram showing the relationship between a defective pixel and a video signal level according to the first embodiment.

[0023] Figure 8 This is a diagram for explaining defective pixel detection and defective pixel removal according to the first embodiment.

[0024] Fig. 9 This is a diagram for explaining the defect adaptive control involved in the first embodiment.

[0025] Fig.10 This is a block diagram showing a configuration example of an image processing device according to the second embodiment.

[0026] Fig.11 This is a diagram showing a configuration example of a signal processing unit according to the second embodiment.

[0027] Fig.12 The diagram shows the video signal of each frame and the multiplexed video signal.

[0028] Fig.13 This is a diagram showing the first video signal and the second video signal after multiplexing according to the second embodiment.

[0029] Fig.14 This is a diagram showing the relationship between the first video signal and the second video signal after multiplexing according to the second embodiment.

[0030] Fig.15 This is a diagram for explaining the synthesis coefficient involved in the second embodiment.

[0031] Fig.16 This is a diagram showing the relationship among defective pixels, video signal levels, and determination signals according to the second embodiment.

[0032] Fig.17 This is a diagram for explaining defective pixel detection according to the second embodiment.

[0033] Fig.18 This is a diagram for explaining the defect adaptive control involved in the second embodiment.

[0034] Fig.19 is a flowchart showing an image processing method according to another embodiment. DETAILED DESCRIPTION

[0035] An image processing device involved in one embodiment of the present disclosure includes: multiple signal processing units that perform signal processing on multiple image signals; a second control signal generating unit that generates a second control signal based on the first control signal generated by the signal processing unit; and an image synthesis unit, wherein the signal processing unit has a first digital processing unit for performing image adjustment, and a second digital processing unit, the second digital processing unit has a defect detection unit that detects defective pixels, and a defective pixel adaptive processing unit, the defective pixel adaptive processing unit generates the first control signal that specifies a synthesis coefficient to the second control signal generating unit based on the detection result of the defective pixels, the image synthesis unit synthesizes a plurality of images that have been subjected to the image adjustment by the first digital processing units of the respective multiple signal processing units through the synthesis coefficient represented by the second control signal, and when a defective pixel is detected, the defective pixel adaptive processing unit generates the first control signal that specifies the synthesis coefficient to the second control signal generating unit so that the plurality of images include an image with residual defective pixels.

[0036] Thus, when a defective pixel is detected, after image adjustment, the image in which the defective pixel remains is synthesized, so there is no need to configure a processing unit that performs defect correction on each of the multiple image signals individually. For example, only one processing unit that performs defect correction on one synthesized image signal is required, so the circuit scale can be reduced, and power consumption can also be reduced accordingly. Therefore, according to the image processing device disclosed in the present invention, it is possible to reduce the circuit scale and power consumption and perform defect correction.

[0037] For example, the plurality of video signals may be a plurality of video signals having different sensitivities.

[0038] Thus, when a plurality of video signals having different sensitivities are synthesized in order to expand the dynamic range, it is possible to perform defect correction while reducing the circuit scale and power consumption.

[0039] For example, the second digital processing unit may include a defect removal unit for removing defective pixels, and a low-pass filter for flattening an image signal after the defective pixels are removed, and the defective pixel adaptive processing unit may generate the first control signal for pixels in which no defects are detected, using the image signal after the defective pixels are removed and flattened.

[0040] In image synthesis, the signal used to determine the synthesis coefficient may change due to the influence of noise or jitter, causing the synthesized image to degrade. In contrast, the change of the synthesis coefficient can be reduced by flattening the image using a low-pass filter, which can reduce the degradation of the synthesized image. However, in the case where a defective pixel is generated in the image signal, the signal of the defective pixel is extended to the surrounding pixels due to the flattening process of the low-pass filter, which may cause the synthesized image to degrade. Therefore, for pixels where no defects are detected, the synthesis coefficient is determined using the image signal after the defective pixel is removed and flattened. As a result, even in the case where a defective pixel is generated in the image signal, the degradation of the synthesized image can be reduced.

[0041] For example, the plurality of image signals may include a first image signal and a second image signal having a lower sensitivity than the first image signal, and when a defective pixel is detected in the first image signal, the defective pixel adaptive processing unit of the signal processing unit that performs signal processing on the first image signal generates the first control signal that specifies a synthesis coefficient higher than that of the second image signal to the second control signal generating unit as the synthesis coefficient of the first image signal. For example, the first control signal that specifies the synthesis coefficient of the first image signal as 100% may be generated.

[0042] Therefore, if a non-defective pixel is synthesized with a defective pixel, the synthesized image may be degraded. Therefore, when a defective pixel is detected in the first image signal, the synthesis coefficient of the first image signal becomes higher (for example, becomes 100%), which can suppress the defective pixel and the non-defective pixel from being synthesized, and can reduce the degradation of the synthesized image.

[0043] For example, when no defective pixel is detected in the first image signal, when the signal level of the first image signal is above a predetermined level and below a saturation level, the defective pixel adaptive processing unit of the signal processing unit that performs signal processing on the first image signal refers to the detection result of the defective pixel in the second image signal, and when a defective pixel is detected in the second image signal, generates the first control signal that specifies a synthesis coefficient higher than the first image signal as the synthesis coefficient of the second image signal to the second control signal generating unit. For example, the first control signal that specifies the synthesis coefficient of the second image signal as 100% may be generated.

[0044] When the signal level of the first image signal is above a predetermined level and below a saturation level, the first image signal and the second image signal are synthesized. At this time, even if a defective pixel is not detected in the first image signal, the synthesized image may be degraded if a defective pixel exists in the second image signal. Therefore, in this case, when a defective pixel is detected in the second image signal, the synthesis coefficient of the second image signal becomes higher (for example, becomes 100%), which can suppress the defective pixel from being synthesized with the pixel without defect, and can reduce the degradation of the synthesized image.

[0045] For example, each of the plurality of image signals may be a signal obtained by selecting and multiplexing a plurality of image signals having different sensitivities when outputting a pixel, and the defect detection unit may detect a defective pixel based on a signal indicating a state of the multiplexed signal.

[0046] Thus, when a plurality of image signals with different sensitivities are synthesized after multiplexing in order to expand the dynamic range, it is possible to reduce the circuit scale and power consumption and perform defect correction. In addition, the signal level of a defective pixel in the multiplexed signal is greatly changed compared with that of an adjacent pixel, and the state of the multiplexed signal changes. Therefore, it is possible to detect a defective pixel based on a signal indicating the state of such a multiplexed signal.

[0047] For example, the plurality of image signals may include a main image signal and a sub-image signal, the main image signal being a signal obtained by multiplexing at least a first image signal and a second image signal having a lower sensitivity than the first image signal, and when the main image signal is the first image signal, the sub-image signal is the second image signal having a lower sensitivity than the first image signal, and when a defective pixel is detected in the first image signal in the main image signal, the defective pixel adaptive processing unit of the signal processing unit that performs signal processing on the main image signal generates the first control signal that specifies a synthesis coefficient higher than the second image signal as the synthesis coefficient of the first image signal in the main image signal to the second control signal generating unit. For example, the first control signal that specifies the synthesis coefficient of the first image signal in the main image signal as 100% may be generated.

[0048] Therefore, if a non-defective pixel is synthesized with a defective pixel, the synthesized image may be degraded. Therefore, when a defective pixel is detected in the first image signal in the main image signal, the synthesis coefficient of the first image signal becomes higher (for example, becomes 100%), which can prevent the defective pixel from being synthesized with the non-defective pixel and reduce the degradation of the synthesized image.

[0049] An image processing method according to one embodiment of the present disclosure is performed by an image processing device, and includes: a plurality of signal processing steps for performing signal processing on a plurality of image signals; a second control signal generating step for generating a second control signal based on a first control signal generated in the signal processing step; and an image synthesis step, wherein the signal processing step includes a first digital processing step for performing image adjustment, and a second digital processing step, wherein the second digital processing step includes a defect detection step for detecting defective pixels, and a defective pixel adaptive processing step, wherein the defective pixel adaptive processing step generates the first control signal for specifying a synthesis coefficient used in the second control signal generating step based on a detection result of the defective pixels, wherein in the image synthesis step, a plurality of images after the image adjustment is performed in the first digital processing step of each of the plurality of signal processing steps are synthesized by the synthesis coefficient indicated by the second control signal, and in the defective pixel adaptive processing step, when a defective pixel is detected, the first control signal for specifying the synthesis coefficient used in the second control signal generating step is generated so that an image with residual defective pixels is included in the plurality of images.

[0050] Thus, it is possible to provide an image processing method that can reduce circuit scale and power consumption and perform defect correction.

[0051] Hereinafter, embodiments will be described in detail with reference to the drawings.

[0052] In addition, the embodiments described below are general or specific examples. The numerical values, shapes, materials, components, configuration positions and connection methods of components, steps, and the order of steps shown in the following embodiments are examples and are not intended to limit the present disclosure.

[0053] (Implementation Method 1)

[0054] First, use Figure 1 A configuration example of an image processing device according to this embodiment will be described.

[0055] Figure 1 This is a block diagram showing a configuration example of the image processing device 1 according to the first embodiment.

[0056] In the first embodiment, a configuration example related to a circuit for synthesizing an image by performing signal processing on a plurality of (two or more) video signals having different sensitivities is described in detail.

[0057] like Figure 1 As shown, the image processing device 1 includes a plurality of signal processing units 10, a second control signal generating unit 20 that generates a second control signal based on a first control signal generated by the signal processing unit 10, and an image synthesizing unit 30 that synthesizes the images after the signal processing using a synthesis coefficient indicated by the second control signal. In addition, no circuit for correcting defects is provided in any of the plurality of signal processing units 10.

[0058] The components constituting the image processing device 1 can be realized by, for example, a processor or the like that executes a program stored in a memory.

[0059] The plurality of signal processing units 10 are processing units for performing signal processing on a plurality of image signals. Figure 1 As shown, a signal processing unit 10 is provided one-to-one for each of the first to n-th image signals (n is an integer greater than 2). In Embodiment 1, the plurality of image signals (the first to n-th image signals) are a plurality of image signals with different sensitivities. Figure 2 The details of the signal processing unit 10 will be described.

[0060] Figure 2 This is a diagram showing a configuration example of the signal processing unit 10 according to the first embodiment.

[0061] like Figure 2 As shown, the signal processing unit 10 includes a first digital processing unit 100 and a second digital processing unit 200. The first digital processing unit 100 processes the image signal for synthesizing the image signal, and the second digital processing unit 200 performs processing for controlling the synthesis conditions.

[0062] The first digital processing unit 100 includes a level correction unit 110 for adjusting the level of a video signal as image adjustment.

[0063] In image synthesis, if the signal used to determine the synthesis conditions changes due to the influence of noise or jitter, the synthesis conditions will change accordingly, and the synthesized image may be degraded. For example, in "Japanese Patent Publication No. 2002-190983", the following structure is proposed as a part about synthesis calculation coefficients: before calculating the coefficients, low-pass filtering is performed for the purpose of suppressing frequency changes. The flattening effect obtained by the low-pass filter has the effect of reducing the frequency changes of the synthesis conditions, and the degradation of the synthesized image can be reduced. However, in the case of defects in the image signal, due to the processing of the low-pass filter, the defective signal extends to the surrounding pixels, which may cause the degradation of the synthesized image.

[0064] On the other hand, in the present disclosure, this problem can be solved by the second digital processing unit 200 .

[0065] The second digital processing unit 200 includes a defect detection unit 210 for detecting defective pixels, a defect removal unit 220 for removing defective pixels, a low pass filter (LPF) 230 for flattening the image signal after the defective pixels are removed, and a defective pixel adaptive processing unit 240 for performing adaptive processing. The second digital processing unit 200 performs adaptive processing by the defective pixel adaptive processing unit 240 based on the defective pixel detection result of the defect detection unit 210 and the image signal after the defect is removed by the defect removal unit 220 and flattened by the low pass filter 230, and outputs a first control signal as a result of the adaptive processing to the second control signal generation unit 20.

[0066] The second control signal generating section 20 refers to the adaptive processing result (in other words, the first control signal) based on the defective pixel detection result to generate a second control signal for image synthesis, and outputs the second control signal to the image synthesizing section 30 .

[0067] The image synthesis unit 30 synthesizes the plurality of images that have been image-adjusted by the first digital processing unit 100 of each of the plurality of signal processing units 10, using the synthesis coefficient indicated by the second control signal. In addition, when a defective pixel is detected, the plurality of images include an image in which the defective pixel remains. In other words, when a defective pixel is detected, the defective pixel adaptive processing unit 240 generates a first control signal that specifies a synthesis coefficient to the second control signal generating unit 20 so that the plurality of images include an image in which the defective pixel remains. Thus, the image synthesis unit 30 can synthesize the image in which the defective pixel remains under the best conditions.

[0068] A plurality of image signals with different sensitivities are outputted as respective image signals and inputted to the corresponding signal processing units 10. Here, an example is described in which the plurality of image signals include a first image signal and a second image signal with a lower sensitivity than the first image signal. Specifically, the first image signal and the second image signal are two 10-bit image signals with a sensitivity ratio of 4:1.

[0069] Figure 3 This is a diagram showing a first video signal and a second video signal according to the first embodiment. Figure 3 (a) represents a first image signal, Figure 3 (b) represents the second image signal. Figure 3 3-1 of (a) represents the output characteristics of the first image signal, Figure 3 3-2 of (b) represents the output characteristics of the second image signal.

[0070] The first image signal (3-1) and the second image signal (3-2) have different sensitivities, and therefore the image signals have different signal levels. When the signal level of the first image signal is 1023 LSB, which is a saturated brightness, the signal level of the second image signal is 256 LSB, which is 1 / 4 of the brightness, and becomes 1023 LSB and saturated at 4 times the brightness.

[0071] The first video signal (3-1) is input to the signal processing unit 10 corresponding to the first video signal, and the second video signal (3-2) is input to the signal processing unit 10 corresponding to the second video signal. In each signal processing unit 10, the first digital processing unit 100 performs level adjustment on the corresponding video signal, and outputs the video signal after the level adjustment to the image synthesis unit 30. The synthesis condition is generated as a second control signal by the second control signal generation unit 20. The image synthesis unit 30 performs synthesis based on the generated second control signal.

[0072] Here, use Figures 4 to 6 Describes image synthesis.

[0073] Figure 4 The diagrams show the relationship between the first video signal and the second video signal according to Embodiment 1. 4-1 shows the output characteristic of the first video signal, 4-2 shows the output characteristic of the second video signal, and 4-3 shows the output characteristic of the second video signal quadrupled.

[0074] Figure 5 5-1 is a diagram showing a composite signal according to Embodiment 1. 5-1 shows a portion of the composite signal to which the first video signal is applied, and 5-2 shows a portion of the composite signal to which the second video signal is applied.

[0075] Figure 6This is a diagram for explaining the synthesis coefficient involved in the first embodiment.

[0076] like Figure 4 As shown, the first image signal (4-1) is a 10-bit signal. When the brightness is above L-1, it becomes saturated, that is, limited to 1023 LSB, and cannot express brightness above L-1. On the other hand, the second image signal (4-2) becomes 1 / 4 of the data volume of the first image signal (4-1) at the brightness of L-1, and specifically, the signal level becomes 256 LSB, which can express brightness above L-1. The second image signal (4-2) has a signal level of 1 / 4 of the first image signal (4-1), so the first image signal (4-1) is adjusted in level by 1 times, and the second image signal (4-2) is adjusted in level by 4 times. The 4-fold second image signal (4-3) becomes a 12-bit signal, which can express up to 4095 LSB. However, the second image signal (4-3) is a signal generated by quadrupling the signal level of the second image signal (4-2) whose signal level is lower than that of the first image signal (4-1) and 1 / 4 of that of the first image signal (4-1), so there is a tendency for the S / N to deteriorate. Therefore, if Figure 5 As shown, by applying the first image signal (5-1) with good S / N to the period of brightness represented by 5-3 with a low signal level up to 1023LSB, and applying the second image signal (5-2) to the signal above 1023LSB where the first image signal (5-1) is saturated, an image signal with a large dynamic range can be synthesized.

[0077] At this time, the first image signal is switched to the second image signal at a saturated level, but if there is an error between the signal level of the first image signal and the signal level of the second image signal after horizontal correction, a step difference in signal level is generated. In addition, even if the signal level of the first image signal is the same as the signal level of the second image signal after horizontal correction, a step difference in S / N is generated at the boundary of the switching due to the S / N difference of the second image signal obtained by quadrupling the horizontal. Therefore, there is a tendency for the image to have a sense of incongruity.

[0078] In order to reduce the discomfort caused by the difference in signal level and S / N, Figure 6 As shown, the synthesis ratio of the first image signal and the second image signal is gradually changed from an arbitrary signal level of the first image signal (for example, a level corresponding to brightness L-1). Specifically, the synthesis ratio of the second image signal is gradually increased, and control is performed to switch to 100% of the second image signal at a saturated signal level of 1023LSB. By gradually changing the synthesis coefficient, the step difference in signal level and the step difference in S / N can be made flat, and the sense of disharmony caused by these step differences can be reduced.

[0079] The synthesis process Figure 2 The processing is performed as follows in the illustrated configuration. The first image signal and the second image signal of the synthesis object are each horizontally adjusted by the horizontal correction unit 110 of the corresponding first digital processing unit 100. The first image signal is horizontally adjusted by 1 times, and the second image signal is horizontally adjusted by 4 times, and are input to the image synthesis unit 30. The synthesis condition of the synthesis processing in the image synthesis unit 30 is generated as a second control signal by the second control signal generation unit 20 with reference to the first control signal, and the second control signal is input to the image synthesis unit 30. The image synthesis unit 30 synthesizes the first image signal that has been horizontally adjusted by 1 times and the second image signal that has been horizontally adjusted by 4 times based on the second control signal indicating the synthesis condition input from the second control signal generation unit 20.

[0080] Here, consider the case where the first image signal has a defect (defective pixel). In the case where the first image signal has a defect, it is impossible to correctly determine the pixel used for calculation. Figure 6 The synthesis coefficient level shown is not correct, so the synthesis condition (second control signal) cannot be correctly generated. As a result, the synthesized image is degraded. As a phenomenon of image degradation, local S / N degradation, level fluctuation of defect signals, and expansion of defects to surrounding pixels can be considered.

[0081] In this embodiment, the second digital processing unit 200 performs defective pixel detection, defective pixel removal, and defective pixel adaptive processing operations, and controls the synthesis coefficient represented by the second control signal generated by the second control signal generating unit 20 based on the adaptive processing result, thereby performing image synthesis in a manner that is not affected by the defect even when a defect occurs. Even when a defect occurs, local S / N degradation and the expansion of the defect to the surrounding pixels are suppressed, and a synthetic image is generated in a state where the defect is retained as it is. In this case, the best image can be generated by performing appropriate defect correction by a subsequent defect correction circuit or the like. The detailed operation of the image processing device 1 is described below.

[0082] The defect detection unit 210 detects defective pixels with respect to the first image signal input to the second digital processing unit 200 .

[0083] Figure 7 This is a diagram showing the relationship between a defective pixel and a video signal level according to the first embodiment. Figure 7 (a) indicates the case of bright spots (white defects, white spots), Figure 7 (b) shows the case of dark spots (black defects, black spots).

[0084] As a defect, there are Figure 7As shown in 7-1 of (a), only the defective pixel is overexposed (white spot), and Figure 7 As shown in 7-3 of (b), the dark spot of the image becomes a black spot (black spot) of only the defective pixel. The image signal has redundancy and the amount of change of adjacent data is small. Therefore, when there is a defect in the image signal, it shows a characteristic that only the data of one pixel changes greatly. The image signal of the bright spot is shown in 7-2, and only one pixel becomes a bright signal level. The dark spot is shown in 7-4, and only one pixel becomes a dark signal level. Using this characteristic, the defect removal unit 220 removes the defective pixel.

[0085] Figure 8 This is a diagram for explaining defective pixel detection and defective pixel removal according to the first embodiment. Figure 8 (a) shows the bright spot situation. Figure 8 (b) shows the case of dark spots.

[0086] For example, considering Figure 8 In the case of a bright spot like 8-1 in (a), the image signal is represented by only the bright spot as shown in 8-2 with an extremely large value. The signal is processed by a median filter of n pixels in the horizontal direction. For example, if a median filter is processed for 3 pixels, the data becomes 8-3, and the data of the bright spot is removed. Considering the case of a dark spot like 8-11, the image signal is represented by only the dark spot as shown in 8-12 with an extremely small value. If the signal is similarly processed by a median filter of 3 pixels in the horizontal direction, the data becomes 8-13, and the data of the dark spot is removed. Here, an example of using a median filter of n pixels is described as a method for removing defective pixels, but any removal method that utilizes the characteristics of the defect may be implemented using a different method.

[0087] The defect detection unit 210 detects defective pixels. Specifically, the defect detection unit 210 compares the signal level of the pixel of interest with the signal level of the pixels surrounding it. For example, when the difference between the difference of the pixel of interest and the average value of the two surrounding pixels is obtained, the difference result is as shown in 8-4, and only for the defective pixel, a larger value in the positive direction is calculated as the difference. When the signal level of the difference determined as a defective pixel is set to 60LSB, if the difference result is above 60LSB, it can be determined as a defect. In addition, similarly for dark spots, the defect detection unit 210 compares the signal level of the pixel of interest with the signal level of the pixels surrounding it. For example, when the difference between the difference of the pixel of interest and the average value of the two surrounding pixels is obtained, the difference result is as shown in 8-14, and only for the defective pixel, a larger value in the negative direction is calculated as the difference. When the signal level of the difference determined as a defective pixel is set to -60LSB, if the difference result is below -60LSB, it can be determined as a defect. Here, an example using a difference with surrounding pixels as a defective pixel detection method is described, but any detection method using the characteristics of a defect may be used, and implementation may be performed using a different method.

[0088] Next, the low-pass filter 230 will be described.

[0089] Usually, noise components are superimposed on the image signal, so data deviations are caused by the influence of noise. When the defective pixel adaptive processing unit 240 uses only one pixel to generate the first control signal for the second control signal generating unit 20 to determine the synthesis condition, there is a tendency for the first control signal to be deviated. Therefore, flattening the noise component by the low-pass filter 230 is an effective means for stabilizing the synthesis condition. However, when there are defects in the image signal, not only when there are defects in the pixel of interest, but also when there are defects in the surrounding pixels, the defective pixel affects the processing result of the low-pass filter 230, so the first control signal generates an error. When the second control signal generating unit 20 uses the first control signal that generates an error, it is impossible to calculate the correct synthesis condition (second control signal). As a result, the synthesized signal deteriorates in the form of the influence of the defective pixel extending to the surrounding pixels.

[0090] In view of such a problem, by providing the defect removal unit 220, the defective pixels of the image signal input to the low-pass filter 230 are removed, which has the effect of reducing the image quality degradation caused by the extension of the defect to the surrounding pixels. Even if there is a defect in the image signal, by inputting the image signal processed by the median filter performed by the defect removal unit 220 to the low-pass filter 230, the low-pass filter 230 can also perform a flattening process on the image signal without the defective signal, so that the influence of the defective pixel will not extend to the surrounding pixels, and the degradation of the synthesized signal can be reduced.

[0091] As described above, the defective pixel adaptive processing unit 240 generates the first control signal using the image signal after the defective pixels are removed and the flattening process is performed for the pixels in which the defects are not detected.

[0092] Next, the defective pixel adaptive processing unit 240 will be described.

[0093] The defect detection signal detected by the defect detection unit 210 and the image signal without defect signal obtained by the processing of the defect removal unit 220 and the low-pass filter 230 and after the flattening process are input to the defect pixel adaptive processing unit 240. Adaptive processing is performed on the pixels where the defect is detected. Fig. 9 Provide explanation.

[0094] Fig. 9 This is a diagram for explaining the defect adaptive control involved in the first embodiment.

[0095] First, consider the case where a bright spot is generated. Here, the result of image synthesis varies due to the signal level of the defect, so Fig. 9 The synthesis coefficient when the first image signal and the second image signal are synthesized is explained.

[0096] 9-1 indicates an area where the first image signal is used at 100%, 9-2 indicates an area where the first image signal and the second image signal are respectively multiplied by a synthesis coefficient and synthesized, and 9-3 indicates an area where the second image signal is used at 100%. When the signal level of the first image signal is smaller than the brightness L-1, the synthesis coefficient of the area 9-1 is applied. When the signal level of the first image signal is greater than the brightness L-1 and smaller than the saturation level, the synthesis coefficient of the area 9-2 is applied. When the signal level of the first image signal is saturated, the synthesis coefficient of the area 9-3 is applied.

[0097] In the case where a bright spot is generated in the first image signal, when the signal level of the bright spot is at a saturated level, even if the image signal is not saturated if the bright spot is not generated, it is determined to be a saturated signal due to the influence of the bright spot. Therefore, it is mistakenly recognized as a state where the synthesis coefficient of 9-3 is applied (i.e., a state where the second image signal is applied at 100%), and the second image signal is applied at 100%. At this time, when the signal level of the original image signal is at a level corresponding to the brightness of 9-1, the second image signal is in a state of poor SN due to the low signal level, and the signal with the poor SN is applied at 100%. Therefore, based on the synthesized signal, an image with poor SN locally is synthesized.

[0098] In addition, the highlights are Fig. 9 In the case where the brightness is above the level corresponding to the brightness L-1 and below the saturation level, it is mistakenly recognized as a state where the synthesis coefficient of 9-2 is applied, so the defect signal of the first image signal is synthesized with the second image signal at a certain ratio. As a result, the synthesized image becomes a synthesized image obtained by multiplying the defect signal by a certain synthesis coefficient. Therefore, the synthesized signal changes to a signal level affected by the defect, and a correct synthesized signal cannot be obtained.

[0099] In order to avoid synthesis based on the misrecognition, when a defective pixel is detected by the defect detection unit 210, the defective pixel adaptive processing unit 240 sends a first control signal to the second control signal generation unit 20 to adaptively change the synthesis coefficient unconditionally to a synthesis coefficient of 9-1, and change the synthesis coefficient.

[0100] For pixels for which no defect is detected by the defect detection unit 210, the image signal after the flattening process is used for normal processing. However, when the first image signal is above the level corresponding to the brightness L-1 and below the saturation level, it is synthesized with the second image signal at a certain synthesis ratio, but when a defect occurs in the second image signal, the defect signal is used for synthesis. Therefore, in this case, the detection result of the defective pixel of the second image signal is referred to, and when a defect occurs, the second image signal is applied at 100%.

[0101] Next, consider the case where a dark spot occurs.

[0102] In the case where the first image signal generates a dark spot below the level corresponding to the brightness L-1, regardless of which area of ​​9-1, 9-2, and 9-3 the signal level of the image signal that does not generate the dark spot is, it is determined that the synthesis coefficient of 9-1 is applicable. In this case, as a result, the first image signal is output as 100%, and the signal of the dark spot is output as it is, so the signal level of the dark spot does not change. However, in the case where the first image signal generates a dark spot above the level corresponding to the brightness L-1, it is mistakenly recognized as a state where the synthesis coefficient of 9-2 is applicable, and the defect signal of the first image signal is synthesized with the second image signal at a certain ratio, similar to the case of the bright spot. As a result, the synthesized image becomes a synthesized image obtained by multiplying the defective signal by a certain synthesis coefficient. Therefore, the synthesized signal changes to a signal level affected by the defect, and a correct synthesized signal cannot be obtained.

[0103] In order to avoid synthesis based on this misrecognition, as in the case of the bright spot, the defective pixel adaptive processing unit 240 sends a first control signal to the second control signal generating unit 20 to adaptively and unconditionally change the synthesis coefficient to a synthesis coefficient of 9-1, and changes the synthesis coefficient.

[0104] In this way, when a defective pixel is detected in the first image signal, the defective pixel adaptive processing unit 240 of the signal processing unit 10 that performs signal processing on the first image signal generates a first control signal that specifies a synthesis coefficient higher than that of the second image signal as the synthesis coefficient of the first image signal to the second control signal generating unit 20. For example, the defective pixel adaptive processing unit 240 can reduce the degradation of the synthesized image by generating a first control signal that specifies the synthesis coefficient of the first image signal to be 75% or more. In addition, for example, the defective pixel adaptive processing unit 240 can sufficiently reduce the degradation of the synthesized image by generating a first control signal that specifies the synthesis coefficient of the first image signal to be 90% or more. In addition, for example, the defective pixel adaptive processing unit 240 can further reduce the degradation of the synthesized image by generating a first control signal that specifies the synthesis coefficient of the first image signal to be 100%.

[0105] When no defective pixel is detected in the first image signal, the defective pixel adaptive processing unit 240 detects the defective pixel according to the level of the first image signal, such as Fig. 9 The composite ratio of the first image signal and the second image signal is calculated in this manner, and a first control signal for specifying the calculated composite ratio is generated and output to the second control signal generating unit 20. On the other hand, when the first control signal for specifying the composite coefficient of the first image signal to be 100% is sent from the defective pixel adaptive processing unit 240, the second control signal generating unit 20 generates a second control signal for unconditionally changing the composite coefficient of the first image signal to 100%.

[0106] In addition, when no defective pixel is detected in the first image signal, when the signal level of the first image signal is above a predetermined level (specifically, a level corresponding to brightness L-1) and below a saturation level, the defective pixel adaptive processing unit 240 of the signal processing unit 10 that performs signal processing on the first image signal refers to the detection result of the defective pixel in the second image signal, and when a defective pixel in the second image signal is detected, generates a first control signal that specifies a synthesis coefficient higher than the first image signal as the synthesis coefficient of the second image signal to the second control signal generating unit 20. For example, the defective pixel adaptive processing unit 240 can reduce the degradation of the synthesized image by generating the first control signal that specifies the synthesis coefficient of the second image signal to be 75% or more. In addition, for example, the defective pixel adaptive processing unit 240 can sufficiently reduce the degradation of the synthesized image by generating the first control signal that specifies the synthesis coefficient of the second image signal to be 90% or more. In addition, for example, the defective pixel adaptive processing unit 240 can further reduce the degradation of the synthesized image by generating the first control signal that specifies the synthesis coefficient of the second image signal to be 100%.

[0107] The second control signal generating unit 20 generates a second control signal based on the synthesis coefficient specified by the first control signal generated by the process in the defective pixel adaptive processing unit 240, and outputs it to the image synthesizing unit 30. The image synthesizing unit 30 synthesizes the first image signal and the second image signal after the level adjustment based on the synthesis ratio indicated by the second control signal, thereby generating a synthesized signal in a form that is not affected by the defect. In addition, the synthesized signal generated by the image synthesizing unit 30 can be defect-corrected without any problem in the subsequent processing.

[0108] This configuration can form the defect detection unit 210 by a median filter of n pixels, and can form the defect removal unit 220 by a comparison circuit. Therefore, compared with installing a circuit in each signal processing unit 10 that corrects defective pixels by supplementing pixels from the periphery, the circuit scale can be reduced.

[0109] (Implementation Method 2)

[0110] Next, Embodiment 2 will be described.

[0111] First, use Fig.10 A configuration example of an image processing device according to this embodiment will be described.

[0112] Fig.10 This is a block diagram showing a configuration example of an image processing device 2 according to the second embodiment.

[0113] In Embodiment 2, the plurality of image signals processed by the plurality of signal processing units are signals obtained by selecting and multiplexing a plurality of (two or more) image signals having different sensitivities when outputting pixels. Hereinafter, a configuration example of a circuit for synthesizing an image by signal processing the multiplexed signals will be described in detail.

[0114] like Fig.10 As shown in the figure, the image processing device 2 includes: a plurality of signal processing units (signal processing units 10a and 10b in the second embodiment), a second control signal generating unit 20 that generates a second control signal based on a first control signal generated by the signal processing unit 10a, and an image synthesizing unit 30 that synthesizes the images after the signal processing using a synthesis coefficient indicated by the second control signal. In addition, a defect correction circuit is not provided in any of the plurality of signal processing units. The functions of the second control signal generating unit 20 and the image synthesizing unit 30 are basically the same as those in the first embodiment, and therefore the description thereof is omitted.

[0115] The components constituting the image processing device 2 can be realized by, for example, a processor or the like that executes a program stored in a memory.

[0116] The plurality of signal processing units (the signal processing units 10a and 10b in the second embodiment) are processing units that perform signal processing on the plurality of video signals (the main video signal and the sub-video signal as the multiplexed signal in the second embodiment). Fig.10 As shown in FIG. 1 , a signal processing unit 10a is provided for the main image signal, and a signal processing unit 10b is provided for the sub-image signal. Fig.11 The signal processing unit 10 a will be described in detail.

[0117] Fig.11 This is a diagram showing a configuration example of a signal processing unit 10 a according to the second embodiment.

[0118] like Fig.11 As shown, the signal processing unit 10a includes a first digital processing unit 100 for performing image adjustment and a second digital processing unit 200a. The purpose of the first digital processing unit 100 is to process the image signal for synthesizing the image signal, and the purpose of the second digital processing unit 200a is to control the synthesis conditions. The function of the first digital processing unit 100 is basically the same as that in the first embodiment, so the description is omitted.

[0119] The second digital processing unit 200a includes a defect detection unit 210a for detecting defective pixels, and a defective pixel adaptive processing unit 240a for performing adaptive processing. The second digital processing unit 200a performs adaptive processing by the defective pixel adaptive processing unit 240a based on the detection result of the defective pixel by the defect detection unit 210a, and outputs a first control signal to the second control signal generating unit 20 as the result of the adaptive processing.

[0120] Furthermore, the signal processing unit 10b includes the first digital processing unit 100 but does not include the second digital processing unit 200a. That is, the signal processing unit 10b only has a function of adjusting the level of the video signal.

[0121] Next, use Fig.12 Describe the multiplexed image signal.

[0122] Fig.12 The diagram shows the video signal of each frame and the multiplexed video signal. Fig.12 (a) represents the image signal of each frame, Fig.12 (b) represents the multiplexed image signal.

[0123] Multiple image signals with different sensitivities such as Fig.12 The image signal selected and multiplexed when the image is read out as in (b) is obtained from Fig.12 The main image signal and the sub-image signal are multiplexed by selecting one exposure image signal from the multiple signals from the first image signal to the nth image signal shown in (a) when reading out the pixel, and a discrimination signal indicating which signal is selected for each pixel.

[0124] The main image signal is a signal obtained by multiplexing at least a first image signal and a second image signal having a lower sensitivity than the first image signal. The sub-image signal is a signal that is to be synthesized with the main image signal. For example, when the main image signal is the first image signal, the sub-image signal is the second image signal having a lower sensitivity than the first image signal. For example, when the main image signal is the second image signal, the sub-image signal to be synthesized is the third image signal having a lower sensitivity than the second image signal. For example, when the main image signal is the nth image signal, the sub-image signal becomes invalid data because there is no synthesis object.

[0125] Here, a description will be given of a process performed using a first image signal and a second image signal which are two 10-bit image signals having a sensitivity ratio of 4:1.

[0126] Fig.1313 - 1 is a diagram showing a first video signal and a second video signal after multiplexing according to Embodiment 2. 13 - 1 shows an output characteristic of the first video signal, and 13 - 2 shows an output characteristic of the second video signal.

[0127] Fig.14 14 - 1 is a diagram showing the relationship between the first video signal and the second video signal after multiplexing according to Embodiment 2. 14 - 1 shows the output characteristic of the first video signal, and 14 - 2 shows the output characteristic of the second video signal which is quadrupled.

[0128] The signal processing unit 10a performs signal processing on the multiplexed main video signal based on the determination signal (see FIG. 1 ) indicating which signal is selected for each pixel by the first digital processing unit 100. Fig.12 (b)) performs horizontal correction. The discrimination signal is an example of a signal indicating the state of the multiplexed signal. Fig.13 As shown, the multiplexed main image signal becomes the first image signal (13-1) up to the brightness L-1, and becomes the second image signal (13-2) when the brightness is above L-1. At this time, the second image signal (13-2) becomes a signal level of 1 / 4 of the first image signal (13-1). The discrimination signal is a signal indicating which signal is selected, and the discrimination signal during the selection period (13-3) of the first image signal (13-1) is set to "00", for example, and the discrimination signal during the selection period (13-4) of the second image signal (13-2) is set to "01", for example. For example, it is set to perform a 1-fold horizontal adjustment when the discrimination signal is "00", and a 4-fold horizontal adjustment when the discrimination signal is "01". Thus, as Fig.14 As shown, in the area 14-4 where the brightness is darker than L-1, the main image signal becomes the first image signal (14-1) that can be displayed with 1023LSB, and in the area 14-5 where the brightness is greater than L-1, the main image signal becomes the second image signal (14-2) after a 4-fold horizontal adjustment, thereby obtaining an image signal with a large dynamic range.

[0129] At this time, the first image signal is switched to the second image signal at a saturated level, but if there is an error between the signal level of the first image signal and the signal level of the second image signal after horizontal correction, a step difference in signal level is generated. In addition, even if the signal level of the first image signal is the same as the signal level of the second image signal after horizontal correction, a step difference in S / N is generated at the boundary of the switching due to the S / N difference of the second image signal obtained by quadrupling the horizontal. Therefore, there is a tendency for the image to have a sense of incongruity.

[0130] In order to reduce the discomfort caused by the difference in signal level and S / N, Fig.15 The sub-image signal is shown.

[0131] Fig.15 This is a diagram for explaining the synthesis coefficient involved in the second embodiment. Fig.15 (a) is a diagram showing a synthesis ratio of a main image signal as a first image signal and a sub-image signal as a second image signal, Fig.15 (b) is a diagram showing the relationship between the brightness and signal level of a main image signal as a first image signal and a sub-image signal as a second image signal. 15-1 shows the output characteristics of the main image signal, and 15-2 shows the output characteristics of the sub-image signal.

[0132] like Fig.15 As shown in (a) and (b), the synthesis ratio of the main image signal (15-1) and the sub-image signal (15-2) is gradually changed from any signal level (for example, a level corresponding to brightness L-1) of the main image signal (15-1) as the first image signal. Specifically, control is performed to gradually increase the synthesis ratio of the sub-image signal and switch to 100% of the sub-image signal at a saturated signal level of 1023LSB. At a signal level of 1024LSB or more, the main image signal is switched to the second image signal. Therefore, by gradually changing the synthesis coefficient, the step difference in signal level and the change in the step difference in S / N can be made flat, and the sense of disharmony caused by these step differences can be reduced.

[0133] When the main video signal has a defect, the second control signal generating unit 20 cannot correctly determine the Fig.15 The synthesis coefficient shown in (a) is at a level that is not sufficient, and therefore the synthesis condition (second control signal) cannot be correctly generated. As a result, the synthesized image is degraded. Examples of image degradation phenomena include changes in the level of defect signals and the extension of defects to surrounding pixels.

[0134] In this embodiment, the second digital processing unit 200a performs defective pixel detection and adaptive processing operations on the defective pixels, and controls the synthesis coefficient represented by the second control signal generated by the second control signal generating unit 20 based on the adaptive processing result, thereby performing image synthesis in a manner that is not affected by the defect even when a defect occurs. Even when a defect occurs, local S / N degradation and the expansion of the defect to the surrounding pixels are suppressed, and a synthetic image is generated in a state where the defect is retained as it is. In this case, the best image can be generated by performing appropriate defect correction by a subsequent defect correction circuit or the like. The following describes the detailed operation of the image processing device 2.

[0135] The defect detection unit 210 a detects defective pixels with respect to the first video signal input to the second digital processing unit 200 a .

[0136] Fig.16 This is a diagram showing the relationship among defective pixels, video signal levels, and determination signals according to the second embodiment. Fig.16 (a) shows the bright spot situation. Fig.16 (b) shows the case of dark spots.

[0137] Defective images exist such as Fig.16 As shown in 16-1 of (a), only the defective pixel is overexposed (white spot), and Fig.16 As shown in 16-4 of (b), a dark spot in the image is formed when only the defective pixel is black. The image signal has redundancy and the amount of change in adjacent data is small. Therefore, when there is a defect in the image signal, there is a tendency for the data of only one pixel to change greatly.

[0138] At this time, the bright spot is represented as an image signal as shown in 16-2, where only one pixel becomes bright signal level, and the dark spot is represented as an image as shown in 16-5, where only one pixel is dark.

[0139] The first video signal in the multiplexed main video signal has the following characteristics: as shown in 16-3, the surrounding pixels are "00" as a discrimination signal, and only the defective pixel shows a different signal "01". Using this characteristic, the defect detection unit 210a detects defective pixels.

[0140] Fig.17 This is a diagram for explaining defective pixel detection according to the second embodiment. Fig.17 (a) shows the bright spot situation. Fig.17 (b) shows the case of dark spots.

[0141] Considering the Fig.17In the case of the bright spot shown in 17-1. In embodiment 2, the main image signal becomes a multiplexed signal, so when the defect signal is less than the saturation level of 1024LSB of the first image signal as in 17-2, the discrimination signals are all the same as shown in 17-3. The detection of defective pixels at this time is performed, for example, by comparing the signal levels of the pixel of interest and the pixels surrounding it. As shown in 17-4, only for the defective pixel, a larger value is calculated as the difference. At this time, when the signal level of the difference determined as a defective pixel is set to 60LSB, if the difference (17-4) is greater than 60LSB, it can be determined as a defective pixel. For example, the sensitivity ratio of the signal when the discrimination signal is "00" and when it is "01" is set to 16:1. When the defect signal is greater than the saturation level of 1024LSB of the first image signal, only the defective pixel becomes a different discrimination signal as shown in 17-6. In 17-5, the signal level of the pixel of interest becomes "200", and when the discrimination signal is "00", the signal with a saturation level of 1024LSB or more becomes a non-saturated signal when the discrimination signal is "01" as shown in 17-6. At this time, the defective pixel is detected by comparing the discrimination signals of the pixel of interest with the pixels surrounding it. If the discrimination signal of the pixel of interest is different from the discrimination signals of the surrounding pixels, the pixel of interest can be determined as a defective pixel.

[0142] Regarding the dark spots shown in 17-11, defective pixels can also be detected by the same processing as the bright spots. When the surrounding pixels are less than the saturation level of 1024LSB of the first image signal as shown in 17-12, the discrimination signals are all the same as shown in 17-13. The detection of defective pixels at this time is performed by comparing the signal levels of the pixel of interest and the pixels surrounding it. As shown in 17-14, only for the defective pixels, a larger value in the negative direction is calculated as a difference. At this time, when the signal level of the difference determined as a defective pixel is set to -60LSB, if the difference (17-14) is less than -60LSB, it can be determined as a defective pixel. For example, the sensitivity ratio of the signal when the discrimination signal is "00" and when it is "01" is set to 16:1. When the surrounding pixels are greater than the saturation level of 1024LSB of the first image signal, as shown in 17-16, only the defective pixels become different discrimination signals relative to the surrounding pixels. In 17-15, the signal levels of the surrounding pixels are "500", "496", "502", "497", "499", and "508". When the discrimination signal is "00", the signal with a saturation level of 1024LSB or more is as shown in 17-16. When the discrimination signal is "01", the signal level becomes 1 / 16 and becomes an unsaturated signal. At this time, the defective pixel is detected by comparing the discrimination signals of the pixel of interest with the pixels surrounding it. If the discrimination signal of the pixel of interest is different from the discrimination signals of the surrounding pixels, the pixel of interest can be determined as a defective pixel.

[0143] Next, the defective pixel adaptive processing unit 240 a will be described.

[0144] The defect detection signal detected by the defect detection unit 210a is input to the defect pixel adaptive processing unit 240a. For pixels that are not determined to be defective pixels, normal processing is performed using the main image signal and the auxiliary image signal. For pixels that are determined to be defective pixels, adaptive processing is performed according to the situation. Here, the pixel detection signal is used. Fig.18 Adaptive processing for defective pixels is described.

[0145] Fig.18 This is a diagram for explaining the defect adaptive control involved in the second embodiment.

[0146] First, consider the case where a bright spot is generated. Here, the result of image synthesis varies due to the signal level of the defect, so Fig.18 The synthesis coefficient when the first image signal and the second image signal are synthesized is described.

[0147] 18-1 indicates an area where the main image signal of the first image signal is used 100%, 18-2 indicates an area where the main image signal of the first image signal and the sub-image signal of the second image signal are multiplied by a synthesis coefficient and synthesized, and 18-3 indicates an area where the main image signal of the second image signal is used 100%.

[0148] In the case where a bright spot is generated in the main image signal of the first image signal, when the signal level of the bright spot is at a saturated level, even if the image signal is not saturated if the bright spot is not generated, the discrimination signal becomes a value corresponding to the main image signal of the second image signal due to the influence of the bright spot, and is determined to be the second image signal. Therefore, it is mistakenly recognized as a state where the main image signal of the second image signal of 18-3 is used 100%, and the second image signal is applied 100%. At this time, when the signal level of the original image signal is at a level corresponding to the brightness of the area of ​​18-1, the second image signal is in a state of poor SN due to the low signal level, and as a result, the signal with poor SN is applied 100%. Therefore, according to the synthesized signal, an image with poor SN locally is synthesized.

[0149] In addition, the highlights are as follows Fig.18 When the brightness is above the level corresponding to the brightness L-1 and below the saturation level, it is mistakenly recognized as a state where the synthesis coefficient of 18-2 is applied, so the defect signal of the main image signal and the auxiliary image signal are synthesized at a certain ratio. As a result, the synthesized image becomes a synthesized image obtained by multiplying the defect signal by a certain synthesis coefficient. Therefore, the synthesized signal changes to a signal level affected by the defect, and a correct synthesized signal cannot be obtained.

[0150] In order to avoid synthesis based on this misrecognition, when a defective pixel is detected by the defect detection unit 210a, the defect detection unit 210a outputs the defect detection result obtained by comparing the focus pixel of the image signal with the surrounding pixels (referred to as the first defect detection result) or the defect detection result based on the discrimination signal (referred to as the second defect detection result) to the defective pixel adaptive processing unit 240a.

[0151] When the first defect detection result is input, the defective pixel adaptive processing unit 240a unconditionally changes the synthesis coefficient to 18-1 to output the main image signal at 100%. At the synthesis coefficient of 18-1, the main image signal is output at 100% as it is, so it is synthesized in a form that retains the defect signal as it is regardless of the signal level of the defect. When the second defect detection result is input, the defective pixel adaptive processing unit 240a does not exist in the main image signal. In this case, the first image signal should become a saturated signal of 1023LSB due to the defect signal, so the signal level is unconditionally replaced with 1023LSB. By being replaced with 1023LSB, a saturated signal of the signal level of the defect signal is output, so the synthesis is performed in a form that retains the defect signal as it is.

[0152] Next, consider the case where a dark spot occurs.

[0153] In the case where the main image signal generates a dark spot below the level corresponding to the brightness L-1, regardless of which area of ​​the signal level of the image signal that does not generate the dark spot is 18-1, 18-2, or 18-3, it is determined to be in a state where the synthesis coefficient of 18-1 is applicable. In this case, as a result, the first image signal is output as 100%, and the signal of the dark spot is output as it is, so the signal level of the dark spot does not change. However, in the case where the main image signal generates a dark spot above the level corresponding to the brightness L-1, it is mistakenly recognized as a state where the synthesis coefficient of 18-2 is applicable, so the defect signal of the main image signal is synthesized with the second image signal at a certain ratio, as in the case of the bright spot. As a result, the synthesized image becomes a synthesized image obtained by multiplying the defective signal by a certain synthesis coefficient. Therefore, the synthesized signal changes to a signal level affected by the defect, and a correct synthesized signal cannot be obtained.

[0154] In order to avoid synthesis based on the misrecognition, as in the case of bright spots, when a defective pixel is detected by the defect detection unit 210a, the defect detection unit 210a outputs the first defect detection result obtained by comparing the focus pixel of the image signal with the surrounding pixels, or the second defect detection result based on the discrimination signal to the defective pixel adaptive processing unit 240a.

[0155] Regardless of whether the first defect detection result or the second defect detection result is input, the defective pixel adaptive processing unit 240a unconditionally changes to a synthesis coefficient of 18-1 when a defective pixel is detected by the defect detection unit 210a. At the synthesis coefficient of 18-1, the main image signal is outputted at 100% as it is. In the case of a dark spot, the first image signal is always present in the main image signal, and synthesis can be performed in a form in which the defective signal is retained as it is, regardless of the signal level of the defect.

[0156] Based on the defect detection result of the main image signal, the defective pixel adaptive processing unit 240a unconditionally changes the synthesis coefficient to 18-1 using the first image signal at 100% for both bright spots and dark spots. In addition, when the bright spot is detected according to the discrimination signal, the defective pixel adaptive processing unit 240a sends a first control signal to the second control signal generating unit 20 to adaptively fix the main image signal to the saturation level 1023LSB. When the first control signal is sent from the defective pixel adaptive processing unit 240a, the second control signal generating unit 20 generates a second control signal for changing the synthesis coefficient so that the main image signal is fixed to the saturation level 1023LSB. The second control signal generating unit 20 usually generates a second control signal based on the level of the first image signal, such as Fig.18 The synthesis ratio (second control signal) of the main image signal and the sub-image signal is generated in this way, but when a first control signal specifying the synthesis coefficient of the first image signal to be 100% is sent from the defective pixel adaptive processing unit 240a, a second control signal is generated to unconditionally change the synthesis coefficient of the first image signal to 100%.

[0157] In this way, when a defective pixel is detected in the first image signal in the main image signal, the defective pixel adaptive processing unit 240a of the signal processing unit 10a that performs signal processing on the main image signal generates a first control signal that specifies a synthesis coefficient higher than the second image signal as the synthesis coefficient of the first image signal in the main image signal to the second control signal generating unit 20. For example, the defective pixel adaptive processing unit 240a can reduce the degradation of the synthesized image by generating the first control signal that specifies the synthesis coefficient of the first image signal to be 75% or more. In addition, for example, the defective pixel adaptive processing unit 240a can sufficiently reduce the degradation of the synthesized image by generating the first control signal that specifies the synthesis coefficient of the first image signal to be 90% or more. In addition, for example, the defective pixel adaptive processing unit 240a can further reduce the degradation of the synthesized image by generating the first control signal that specifies the synthesis coefficient of the first image signal to be 100%.

[0158] The second control signal generating unit 20 generates a second control signal based on the synthesis coefficient specified by the first control signal generated by the process in the defective pixel adaptive processing unit 240a, and outputs it to the image synthesizing unit 30. The image synthesizing unit 30 synthesizes the first image signal and the second image signal after the level adjustment based on the synthesis ratio indicated by the second control signal, thereby generating a synthesized signal in a form that is not affected by the defect. In addition, the synthesized signal generated by the image synthesizing unit 30 can be defect-corrected without any problem in the subsequent processing.

[0159] As described above, when processing a signal obtained by selecting and multiplexing a plurality of image signals having different sensitivities at the time of pixel output, the signals of each frame are not uniform for the multiplexed signal, so defect correction may not be correctly performed and may be affected by the defect at the time of synthesis. In contrast, according to the present embodiment, it is possible to provide an image processing device 2 that can minimize the influence of defects even when a plurality of image signals having different sensitivities are selected and multiplexed at the time of pixel output.

[0160] (Other embodiments)

[0161] The above describes the image processing device involved in one or more embodiments of the present disclosure based on the embodiments, but the present disclosure is not limited to the embodiments. As long as it does not deviate from the main purpose of the present disclosure, the embodiments obtained by applying various modifications that can be thought of by those skilled in the art, or the embodiments constructed by combining the constituent elements in different embodiments are all included in the scope of one or more embodiments of the present disclosure.

[0162] For example, in Embodiment 1, an example is described in which the second digital processing unit 200 includes the defect removal unit 220 and the low-pass filter 230 . However, the second digital processing unit 200 may not include the defect removal unit 220 and the low-pass filter 230 .

[0163] For example, the present disclosure can be realized not only as an image processing device but also as an image processing method including steps (processing) performed by constituent elements constituting the image processing device.

[0164] Fig.19 is a flowchart showing an image processing method according to another embodiment.

[0165] The image processing method is performed by an image processing device, such as Fig.19As shown, it includes: multiple signal processing steps (step S1), signal processing is performed on multiple image signals; a second control signal generating step (step S2), based on the first control signal generated in the signal processing step, a second control signal is generated; and an image synthesis step (step S3), the signal processing step includes: a first digital processing step (step S11) for image adjustment, and a second digital processing step (step S12), the second digital processing step includes: a defect detection step (step S101) for detecting defective pixels, and a defective pixel adaptive processing step (step S102) for generating a first control signal for specifying a synthesis coefficient used in the second control signal generating step based on the detection result of the defective pixels, in the image synthesis step, multiple images that have been image-adjusted in the first digital processing steps of the multiple signal processing steps are synthesized by the synthesis coefficient represented by the second control signal, in the defective pixel adaptive processing step, when a defective pixel is detected, a first control signal for specifying the synthesis coefficient used in the second control signal generating step is generated so that an image with residual defective pixels is included in the multiple images.

[0166] For example, the present disclosure can be implemented as a program for causing a processor to execute the steps included in the image processing method. Furthermore, the present disclosure can be implemented as a non-volatile computer-readable recording medium such as a CD-ROM on which the program is recorded.

[0167] For example, when the present disclosure is implemented by a program (software), each step is executed by executing the program using hardware resources such as a CPU, a memory, and an input / output circuit of a computer. In other words, each step is executed by the CPU obtaining data from a memory or an input / output circuit and performing calculations, or outputting calculation results to a memory or an input / output circuit.

[0168] In addition, in the above-mentioned embodiment, each component included in the image processing device may be formed by dedicated hardware, or implemented by executing a software program suitable for each component. Each component may also be implemented by a program execution unit such as a CPU or a processor reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory.

[0169] Part or all of the functions of the image processing device involved in the above-mentioned embodiment are typically implemented by LSI as an integrated circuit. It can be used as a single chip individually or as a single chip in a manner that includes part or all of it. In addition, the formation of the integrated circuit is not limited to LSI, and it can also be implemented by a dedicated circuit or a general-purpose processor. It is also possible to use an FPGA (Field Programmable Gate Array) that can be programmed after manufacturing the LSI, or a reconfigurable processor that can reconfigure the connection and setting of the circuit unit inside the LSI.

[0170] Industrial Applicability

[0171] The present disclosure is applicable to an image processing apparatus, and a photographing apparatus or a distance measuring photographing apparatus using the image processing apparatus as an imaging device, for example, a video camera, a digital camera, or a distance measuring system.

[0172] Description of reference numerals:

[0173] 1.2 Image processing device

[0174] 10.10a Signal processing unit

[0175] 20 Second control signal generating unit

[0176] 30 Image synthesis unit

[0177] 100 First Digital Processing Unit

[0178] 200, 200a second digital processing unit

[0179] 210, 210a Defect detection unit

[0180] 220 Defect Removal Department

[0181] 230 Low pass filter

[0182] 240, 240a defective pixel adaptive processing unit

Claims

1. An image processing device comprising: A plurality of signal processing units for performing signal processing on a plurality of image signals; a second control signal generating unit that generates a second control signal based on the first control signal generated by the signal processing unit; and Image synthesis unit, The signal processing unit includes a first digital processing unit and a second digital processing unit for performing image adjustment. The second digital processing unit includes a defect detection unit for detecting defective pixels, and a defective pixel adaptive processing unit for generating the first control signal for specifying a synthesis coefficient to the second control signal generating unit based on a detection result of the defective pixels. The image synthesis unit synthesizes the plurality of images that have been subjected to the image adjustment by the first digital processing unit of each of the plurality of signal processing units, using a synthesis coefficient indicated by the second control signal. When a defective pixel is detected, the defective pixel adaptive processing unit generates the first control signal for specifying a synthesis coefficient to the second control signal generating unit so that an image in which the defective pixel remains is included in the plurality of images.

2. The image processing device according to claim 1, The plurality of image signals are a plurality of image signals having different sensitivities.

3. The image processing device according to claim 2, The second digital processing unit includes a defect removal unit for removing defective pixels, and a low-pass filter for flattening the image signal after the defective pixels are removed. The defective pixel adaptive processing unit generates the first control signal using an image signal obtained by removing defective pixels and performing a flattening process for pixels in which no defect is detected.

4. The image processing device according to claim 2 or 3, The plurality of image signals include a first image signal and a second image signal having a lower sensitivity than the first image signal. When a defective pixel is detected in the first image signal, the defective pixel adaptive processing unit of the signal processing unit that performs signal processing on the first image signal generates the first control signal to the second control signal generating unit, which specifies a synthesis coefficient higher than that of the second image signal as the synthesis coefficient of the first image signal.

5. The image processing device according to claim 4, When no defective pixels are detected in the first image signal, and when the signal level of the first image signal is above a specified level and below a saturation level, the defective pixel adaptive processing unit of the signal processing unit that performs signal processing on the first image signal refers to the detection result of the defective pixels in the second image signal, and when a defective pixel is detected in the second image signal, generates the first control signal that specifies a synthesis coefficient higher than the first image signal as a synthesis coefficient of the second image signal to the second control signal generating unit.

6. The image processing device according to claim 1, Each of the plurality of image signals is a signal obtained by selecting and multiplexing a plurality of image signals with different sensitivities when outputting pixels, The defect detection unit detects a defective pixel based on a signal indicating a state of the multiplexed signal.

7. The image processing device according to claim 6, The plurality of image signals include a main image signal and a sub-image signal, The main image signal is a signal obtained by multiplexing at least a first image signal and a second image signal having a lower sensitivity than the first image signal. When the main image signal is a first image signal, the sub-image signal is a second image signal with a lower sensitivity than the first image signal. When a defective pixel is detected in the first image signal in the main image signal, the defective pixel adaptive processing unit of the signal processing unit that performs signal processing on the main image signal generates the first control signal to the second control signal generating unit, which specifies a synthesis coefficient higher than the second image signal as the synthesis coefficient of the first image signal in the main image signal.

8. An image processing method, performed by an image processing device, comprising: A plurality of signal processing steps are used to perform signal processing on a plurality of image signals; a second control signal generating step of generating a second control signal based on the first control signal generated in the signal processing step; and Image synthesis steps, The signal processing step includes a first digital processing step for image adjustment and a second digital processing step, The second digital processing step includes a defect detection step for detecting defective pixels, and a defective pixel adaptive processing step for generating the first control signal for specifying a synthesis coefficient used in the second control signal generating step based on a detection result of the defective pixels. In the image synthesis step, the plurality of images subjected to the image adjustment in the first digital processing step in each of the plurality of signal processing steps are synthesized by using the synthesis coefficient indicated by the second control signal. In the defective pixel adaptive processing step, when a defective pixel is detected, the first control signal for specifying a synthesis coefficient used in the second control signal generating step is generated so that an image in which the defective pixel remains is included in the plurality of images.

Images