Signal processing device and imaging device

Abstract

A signal processing device according to the present disclosure comprises: an exposure control unit that divides a plurality of pixels for respective one or more colors in an image sensor having the plurality of pixels into a plurality of pixel groups for each of the colors, and exposes the plurality of pixel groups for different exposure durations; an extraction unit that acquires a difference signal between a pixel signal from one pixel group among the plurality of pixel groups and a signal obtained by multiplying a pixel signal from a pixel group different from the one pixel group among the plurality of pixel groups by the reciprocal of an exposure ratio to thereby extract an optical noise component; and a subtraction unit that performs a process for subtracting the optical noise component from pixel signals from the plurality of pixel groups on the basis of the difference signal and the exposure ratio.

Description

Signal processing device and imaging device

[0001] The present disclosure relates to a signal processing device and an imaging device.

[0002] In a global shutter type image sensor having a charge storage unit (memory) in a pixel, due to stray light or the like in the charge storage unit, charges may be generated due to causes other than exposure and may be superimposed on the charge storage unit. The superimposed charges are called optical noise. When there is optical noise, for example, an afterimage may occur as an imaging result. There is PLS (Parasitic Light Sensitivity), which is an index indicating the ratio of unnecessary signals generated by optical noise. Patent Document 1 proposes a technique for calculating optical noise based on the difference between signals between a plurality of fields and correcting image data so as to suppress optical noise.

[0003] International Publication No. 2020 / 021887

[0004] In the above technique, in order to obtain optical noise from the difference between signals between fields, optical noise generated during the signal readout period can be measured, but it is difficult to measure optical noise mixed in before reading the signal of the first field. Therefore, it is difficult to calculate optical noise with high accuracy and suppress optical noise with high accuracy.

[0005] For this reason, it is desirable to provide a signal processing device and an imaging device that can improve the calculation accuracy of optical noise and suppress optical noise with high accuracy.

[0006] The signal processing device according to an embodiment of the present disclosure divides a plurality of pixels in an image sensor having a plurality of pixels for each of one or more colors into a plurality of pixel groups for each color, and exposes each of the plurality of pixel groups with different exposure times. An exposure control unit, a pixel signal from one of the plurality of pixel groups, and an extraction unit that extracts an optical noise component by obtaining a difference signal between the pixel signal and a signal obtained by multiplying the pixel signal from a pixel group different from one of the plurality of pixel groups by the reciprocal of the exposure ratio, and a subtraction unit that performs a process of subtracting the optical noise component from the pixel signal from each of the plurality of pixel groups based on the difference signal and the exposure ratio.

[0007] An imaging device according to one embodiment of the present disclosure includes: an image sensor having multiple pixels for one or more colors; an exposure control unit that divides the multiple pixels in the image sensor into multiple pixel groups for each color and exposes each of the multiple pixel groups with different exposure times; an extraction unit that extracts optical noise components by obtaining a difference signal between a pixel signal from one of the multiple pixel groups and a signal obtained by multiplying the pixel signal from a different pixel group by the reciprocal of the exposure ratio; and a subtraction unit that performs a process of subtracting the optical noise component from the pixel signals from each of the multiple pixel groups based on the difference signal and the exposure ratio.

[0008] In a signal processing device or imaging device according to one embodiment of the present disclosure, optical noise components are extracted by obtaining a difference signal between the pixel signal from one of a plurality of pixel groups exposed at different exposure times and a signal obtained by multiplying the pixel signal from a pixel group different from the one pixel group by the reciprocal of the exposure ratio. Then, based on the difference signal and the exposure ratio, a process is performed to subtract the optical noise component from the pixel signals from each of the plurality of pixel groups.

[0009] Figure 1 is a schematic block diagram showing one example configuration of an imaging device including a signal processing device according to one embodiment of the present disclosure. Figure 2 is a schematic explanatory diagram showing one example configuration of a plurality of pixel groups. Figure 3 is a schematic explanatory diagram showing one example configuration of a plurality of pixel groups. Figure 4 is a schematic explanatory diagram showing one example configuration of a plurality of pixel groups. Figure 5 is a schematic explanatory diagram showing one example configuration of a plurality of pixel groups. Figure 6 is a schematic explanatory diagram showing one example configuration of a plurality of pixel groups. Figure 7 is a schematic explanatory diagram showing one example configuration of a plurality of pixel groups. Figure 8 is a schematic explanatory diagram showing a modified example of the pixel arrangement. Figure 9 is an explanatory diagram showing an overview of the operation of an imaging device according to one embodiment. Figure 10 is a schematic block diagram showing one example configuration of an exposure control unit when there are two types of exposure times. Figure 11 is a schematic block diagram showing one example configuration of an exposure control unit when there are three types of exposure times. Figure 12 is a schematic block diagram showing one example configuration of an optical noise component extraction unit when there are two types of exposure times. Figure 13 is a schematic block diagram showing one example configuration of the optical noise component extraction unit when there are three types of exposure times. Figure 14 is a schematic block diagram showing one example configuration of the optical noise component correction unit when there are two types of exposure times. Figure 15 is a schematic block diagram showing one example configuration of the optical noise component correction unit when there are three types of exposure times. Figure 16 is a schematic block diagram showing one example configuration of the optical noise component subtraction unit when there are two types of exposure times. Figure 17 is a schematic block diagram showing one example configuration of the optical noise component subtraction unit when there are three types of exposure times. Figure 18 is a schematic explanatory diagram showing one specific example of the processing of the optical noise component subtraction unit. Figure 19 is a schematic explanatory diagram showing one specific example of the processing of the optical noise component subtraction unit. Figure 20 is a schematic explanatory diagram showing one specific example of the processing of the optical noise component subtraction unit. Figure 21 is a schematic explanatory diagram showing one specific example of the processing of the optical noise component subtraction unit. Figure 22 is a schematic explanatory diagram showing one specific example of the exposure ratio and one specific example of various values ​​calculated from the exposure ratio. Figure 23 is a schematic diagram illustrating an example of setting the exposure time using the parameter setting unit when there are two types of exposure times. Figure 24 is a schematic diagram illustrating an example of setting the exposure time using the parameter setting unit when there are three types of exposure times.

[0010] The embodiments of this disclosure will be described in detail below with reference to the drawings. The description will be in the following order: 1. One Embodiment 1.1 Overview 1.2 Specific Examples of Each Part 1.3 Modifications 1.4 Effects 2. Other Embodiments

[0011] <1. One Embodiment> [1.1 Overview] Figure 1 is a schematic block diagram showing one example of the configuration of an imaging device including a signal processing device according to one embodiment of the present disclosure.

[0012] An imaging device according to one embodiment includes an exposure control unit 10, an optical noise component extraction unit 20, an optical noise component correction unit 30, an optical noise component subtraction unit 40, a parameter setting unit 50, and an image sensor 60. A signal processing device according to one embodiment includes at least an exposure control unit 10, an optical noise component extraction unit 20 (extraction unit), an optical noise component correction unit 30 (correction unit), and an optical noise component subtraction unit 40 (subtraction unit).

[0013] An imaging device according to one embodiment includes a global shutter type image sensor 60. The image sensor 60 has a plurality of pixels arranged in two dimensions. The image sensor 60 has a plurality of pixels for one or more colors. Each of the plurality of pixels has a charge storage unit (memory). A signal processing device according to one embodiment relates to a technique for suppressing optical noise leaking into the charge storage unit within the pixels of the image sensor 60. In the signal processing device according to one embodiment, a plurality of pixels of the image sensor 60 are photographed with a plurality of exposure times in a single shot, optical noise is separated from the obtained plurality of pixel signals, and an image with suppressed optical noise is obtained.

[0014] The exposure control unit 10 divides each pixel of the image sensor 60 into multiple pixel groups for each color and exposes each of the multiple pixel groups with a different exposure time. If the image sensor 60 is a color sensor, the exposure control unit 10 divides each color (for each RGB) into multiple pixel groups and exposes each pixel group with a different exposure time. If the image sensor 60 is a monochrome sensor, the exposure control unit 10 divides each color into multiple pixel groups and exposes each pixel group with a different exposure time. The exposure control unit 10 sets the exposure time of the first pixel group based on the appropriate exposure value of AE (Auto Exposure), etc. The exposure control unit 10 sets the exposure time of the other (second) pixel group, which is different from the first pixel group, to an exposure ratio that is not 1x to the exposure time of the first pixel group.

[0015] The optical noise component extraction unit 20 extracts optical noise components by obtaining a difference signal between the pixel signal from one of the multiple pixel groups and the pixel signal from a different pixel group from the multiple pixel groups multiplied by the reciprocal of the exposure ratio. The optical noise component extraction unit 20 obtains the difference signal based on each pixel signal after LPF (low-pass filter) processing has been performed on the pixel signals from each of the multiple pixel groups.

[0016] The optical noise component subtraction unit 40 performs a process of subtracting the optical noise component from the pixel signal from each of the multiple pixel groups based on the difference signal and the exposure ratio.

[0017] The optical noise component correction unit 30 performs corrections on the difference signal from the optical noise component extraction unit 20 according to the luminance value of the difference signal.

[0018] The parameter setting unit 50 sets various parameters required for the processing of each unit.

[0019] Figures 2 to 7 are schematic diagrams illustrating one example configuration of multiple pixel groups. Figure 8 is a schematic diagram illustrating a modified version of the pixel array.

[0020] Figures 2 to 4 show an example configuration in which multiple pixels are divided into two pixel groups for each color, and each color is exposed with two different exposure times (bright and dark). By correcting the pixel signals based on the two types of pixel signals obtained by exposing each color with two different exposure times, an image with suppressed optical noise can be obtained.

[0021] Figure 2 shows an example of a configuration in which the image sensor 60 is an RGB color sensor with a pixel configuration in which pixels of each color, R (red), G (green), and B (blue), are arranged in a Bayer array, and the pixels of each color are divided into two types of pixel groups (R1, R2), (G1, G2), and (B1, B2). For example, R1 represents a first group of red pixels set as a bright exposure, and R2 represents a second group of red pixels set as a dark exposure. Figure 3 shows an example of a configuration in which the image sensor 60 is an RGB color sensor with a pixel configuration in which pixels of each color, R, G, and B, are arranged in a Quad Bayer array (a Bayer array in which four adjacent pixels are the same color), and the pixels of each color are divided into two types of pixel groups (R1, R2), (G1, G2), and (B1, B2). Figure 4 shows an example configuration in which the image sensor 60 is a monochrome sensor, and in a pixel configuration in which multiple pixels of one color are arranged, the pixels of one color are divided into two types of pixel groups (a first monochrome pixel group set as a bright exposure, and a second monochrome pixel group set as a dark exposure).

[0022] Figures 5 to 7 show an example configuration where multiple pixels are divided into three pixel groups for each color, and each color is exposed for three different exposure times (bright, dark, and intermediate). By correcting the pixel signals based on the three types of pixel signals obtained by exposing each color for three different exposure times, an HDR (High Dynamic Range) image with suppressed optical noise can be obtained.

[0023] Figure 5 shows an example of a configuration in which the image sensor 60 is an RGB color sensor, and the pixels of each R, G, and B color are arranged in a Bayer array, and the pixels of each color are divided into three types of pixel groups (R1, R2, R3), (G1, G2, G3), and (B1, B2, B3). For example, R1 represents the first group of red pixels set as a bright exposure, R2 represents the second group of red pixels set as an intermediate exposure, and R3 represents the third group of red pixels set as a dark exposure. Figure 6 shows an example of a configuration in which the image sensor 60 is an RGB color sensor, and the pixels of each R, G, and B color are arranged in a Quad Bayer array (a Bayer array in which four adjacent pixels are the same color), and the pixels of each color are divided into three types of pixel groups (R1, R2, R3), (G1, G2, G3), and (B1, B2, B3). Figure 7 shows an example of a configuration in which the image sensor 60 is a monochrome sensor, and in a pixel configuration in which multiple pixels of one color are arranged, the pixels of one color are divided into three types of pixel groups. The three types of monochrome pixel groups can be divided, for example, into a first monochrome pixel group set as a bright exposure, a second monochrome pixel group set as an intermediate exposure, and a third monochrome pixel group set as a dark exposure.

[0024] Note that the arrangement of multiple pixels is not limited to the examples described above. For example, a Bayer array in which two or more pixels other than four adjacent pixels are the same color is also acceptable. For example, as shown in Figure 8, a Bayer array in which nine adjacent pixels are the same color is also acceptable.

[0025] Figure 9 is an explanatory diagram showing an overview of the operation of an imaging device according to one embodiment.

[0026] Figure 9 shows an example of operation when there are two types of exposure times. When there are two types of exposure times, the optical noise component extraction unit 20 calculates the difference between the pixel signal S1 from the first pixel group and the signal S2' obtained by multiplying the pixel signal S2 from the second pixel group by the reciprocal of the exposure ratio Ea (step S102). Then, the optical noise component extraction unit 20 calculates (extracts) the optical noise components leaking into the first pixel group and the second pixel group based on the pixel signals S1 and S2' (step S103). The optical noise component correction unit 30 performs a correction on the difference signal Na from the optical noise component extraction unit 20 (step S104). The correction by the optical noise component correction unit 30 may include, for example, a correction according to at least one of the luminance value, color, and magnitude of the difference signal Na. The optical noise component subtraction unit 40 performs a process to subtract the optical noise component from the pixel signals from each of the multiple pixel groups (step S105). Subtraction is performed based on the corrected difference signal Na' and the exposure ratio between the first pixel group and the second pixel group.

[0027] In the optical noise component extraction unit 20, when calculating the difference between the pixel signal S1 from the first pixel group and the signal S2' obtained by multiplying the pixel signal S2 from the second pixel group by the reciprocal of the exposure ratio Ea, the difference is calculated from decimated signals with different coordinates. For this reason, the optical noise component extraction unit 20 removes the high-frequency components present in the subject and calculates the noise amount by considering the noise due to light leakage as a low-frequency component.

[0028] In the optical noise component extraction unit 20, LPF processing is performed when calculating the optical noise component (step S101). By performing low-pass filter processing, overcorrection when subtracting the optical noise component in the optical noise component subtraction unit 40 can be reduced.

[0029] In the optical noise component extraction unit 20, when calculating the difference between the pixel signal S1 from the first pixel group and the signal S2' obtained by multiplying the pixel signal S2 from the second pixel group by the reciprocal of the exposure ratio Ea, it becomes difficult to correctly calculate the difference amount when the exposure is saturated. Therefore, the optical noise component correction unit 30 may detect the high-brightness portion of the pixel signal and reduce the color fringing of the high-brightness portion due to overcorrection when subtracting optical noise. For example, in the optical noise component correction unit 30, the calculated optical noise component is attenuated in the high-brightness portion. In the case of a monochrome sensor, the optical noise component correction unit 30 reduces the gradation change in the high-brightness portion.

[0030] Furthermore, when calculating the difference between the pixel signal S1 from the first pixel group and the signal S2' obtained by multiplying the pixel signal S2 from the second pixel group by the reciprocal of the exposure ratio Ea, the difference is calculated from decimated signals with different coordinates. For this reason, high-frequency components present in the subject are removed, and noise due to light leakage is considered as a low-frequency component to calculate the noise amount, but errors remain. Therefore, the optical noise component correction unit 30 may use a correction gain corresponding to the magnitude of the difference signal Na and a correction gain corresponding to the color of the difference signal Na to reduce overcorrection when subtracting optical noise. For example, the optical noise component correction unit 30 utilizes the fact that noise due to light leakage is achromatic due to color mixing. Also, the optical noise component correction unit 30 utilizes the fact that when the difference is small, the afterimage blends in with the subject and is not noticeable. If the calculated optical noise component is chromatic, the optical noise component correction unit 30 attenuates the calculated optical noise component. In the optical noise component correction unit 30, if the calculated optical noise component is small, the calculated optical noise component is attenuated.

[0031] [1.2 Specific Examples of Each Part] (Exposure Control Unit 10) Figure 10 is a schematic block diagram showing one example configuration of the exposure control unit 10 when there are two types of exposure times.

[0032] The exposure control unit 10 divides each pixel of the image sensor 60 into multiple pixel groups for each color and exposes each group with a different exposure time. In the case of a color sensor, the exposure control unit 10 divides each color (for each RGB) into multiple pixel groups and exposes each group with a different exposure time. In the case of a monochrome sensor, the exposure control unit 10 divides each color into multiple pixel groups and exposes each group with a different exposure time.

[0033] The exposure control unit 10 sets the exposure time (bright exposure) for the first pixel group based on the appropriate exposure value of the AE, etc. The exposure control unit 10 also sets the exposure time (dark exposure) for the second pixel group to an exposure ratio Ea that is not 1 times the exposure time of the first pixel group. The exposure ratio Ea is set in the parameter setting unit 50.

[0034] Figure 11 is a schematic block diagram showing one example configuration of the exposure control unit 10 when there are three different exposure times.

[0035] The exposure control unit 10 sets the exposure time for the first pixel group (bright exposure) based on the appropriate exposure value of the AE, etc. The exposure control unit 10 sets the exposure time for the second pixel group (intermediate exposure) to an exposure ratio Eb that is not 1 times the exposure time of the first pixel group. The exposure control unit 10 sets the exposure time for the third pixel group (dark exposure) to an exposure ratio Ea that is not 1 times the exposure time of the second pixel group. Furthermore, the exposure control unit 10 sets the exposure time for the third pixel group (dark exposure) so that it is not equal to the exposure time of the first pixel group. The exposure ratios Ea and Eb are set in the parameter setting unit 50.

[0036] Alternatively, the exposure control unit 10 may set the exposure time of the second pixel group (intermediate exposure) based on the appropriate exposure value of AE, etc. The exposure control unit 10 may also set the exposure time of the first pixel group (bright exposure) to an exposure time that is not equal to 1x the exposure time of the second pixel group, but is the reciprocal of the exposure ratio Eb. Furthermore, the exposure control unit 10 may set the exposure time of the third pixel group (dark exposure) to an exposure time that is not equal to the exposure time of the second pixel group, but is an exposure ratio Ea. The exposure control unit 10 may also set the exposure time of the third pixel group (dark exposure) so that it is not equal to the exposure time of the first pixel group. The exposure ratios Ea and Eb are set in the parameter setting unit 50.

[0037] (Optical noise component extraction unit 20) Figure 12 is a schematic block diagram showing one example configuration of the optical noise component extraction unit 20 when there are two different exposure times.

[0038] When there are two types of exposure times, the optical noise component extraction unit 20 includes an image data sorting unit 21, an LPF (low-pass filter) 22A, an LPF 22B, an exposure ratio reciprocal multiplication unit 23, and a difference calculation unit 24.

[0039] The image data distribution unit 21 distributes the input signal (RAW data) for each color into pixel signals from the first pixel group (bright exposure) and pixel signals from the second pixel group (dark exposure).

[0040] The LPF 22A performs LPF processing on the pixel signals from the first pixel group that has been allocated. The pixel signals S1 from the first pixel group after LPF processing are output to the optical noise component subtraction unit 40 and the optical noise component correction unit 30 as an RGB signal Sca (in the case of a color sensor) or a B / W signal Swa (in the case of a monochrome sensor).

[0041] The LPF 22B performs LPF processing on the pixel signals from the allocated second pixel group. The exposure ratio reciprocal multiplication unit 23A multiplies the pixel signal S2 after LPF processing from the second pixel group by the reciprocal of the exposure ratio Ea. The difference calculation unit 24A obtains the difference between the pixel signal S1 (the pixel signal after LPF processing from the first pixel group) and the signal S2' (the signal obtained by multiplying the pixel signal S2 after LPF processing from the second pixel group by the reciprocal of the exposure ratio Ea), and outputs it as the difference signal Na to the optical noise component correction unit 30.

[0042] FIG. 13 is a block diagram schematically showing a configuration example of the optical noise component extraction unit 20 when there are three types of exposure times.

[0043] When there are three types of exposure times, the optical noise component extraction unit 20 includes an image data allocation unit 21, LPFs 22A, 22B, and 22C, exposure ratio reciprocal multiplication units 23A and 23B, difference calculation units 24A and 24B.

[0044] The image data allocation unit 21 allocates the input signal (RAW data) for each color to the pixel signals from the first pixel group (bright exposure), the second pixel group (medium exposure), and the third pixel group (dark exposure).

[0045] The LPF 22A performs LPF processing on the pixel signals from the allocated first pixel group. The pixel signal S11 after LPF processing from the first pixel group is output as the RGB signal Scb (in the case of a color sensor) or the B / W signal Swb (in the case of a black-and-white sensor) to the optical noise component subtraction unit 40 and the optical noise component correction unit 30.

[0046] The LPF 22B performs LPF processing on the pixel signals from the allocated second pixel group. The pixel signal S12 after LPF processing from the second pixel group is output as the RGB signal Sca (in the case of a color sensor) or the B / W signal Swa (in the case of a black-and-white sensor) to the optical noise component subtraction unit 40 and the optical noise component correction unit 30.

[0047] The exposure ratio inverse multiplication unit 23B multiplies the pixel signal S12 after LPF processing from the second pixel group by the inverse of the exposure ratio Ea. The difference calculation unit 24B obtains the difference between the pixel signal S11 (the pixel signal after LPF processing from the first pixel group) and the signal S12' (the signal obtained by multiplying the pixel signal S12 after LPF processing from the second pixel group by the inverse of the exposure ratio Ea), and outputs it as the difference signal Nb to the optical noise component correction unit 30.

[0048] The LPF 22C performs LPF processing on the pixel signal from the assigned third pixel group. The exposure ratio inverse multiplication unit 23A multiplies the pixel signal S13 after LPF processing from the third pixel group by the inverse of the exposure ratio Eb. The difference calculation unit 24A obtains the difference between the pixel signal S12 (the pixel signal after LPF processing from the second pixel group) and the signal S13' (the signal obtained by multiplying the pixel signal S13 after LPF processing from the third pixel group by the inverse of the exposure ratio Eb), and outputs it as the difference signal Na to the optical noise component correction unit 30.

[0049] (Optical Noise Component Correction Unit 30) FIG. 14 is a block diagram schematically showing a configuration example of the optical noise component correction unit 30 when there are two types of exposure times.

[0050] When there are two types of exposure times, the optical noise component correction unit 30 includes a luminance conversion unit 31A, a difference signal correction unit 32A based on luminance values, a difference signal correction unit 33A based on color tones, a difference signal correction unit 34A based on sizes, and a synthesis unit 35A.

[0051] In the optical noise component correction unit 30, the difference signal Na for each color and the pixel signal (RGB signal Sca from the color sensor or B / W signal Swa from the black and white sensor) for creating luminance information are input, and correction is performed on the difference signal Na for each color.

[0052] The luminance conversion unit 31A calculates luminance information from the RGB signal Sca or the B / W signal Swa, and outputs the calculated luminance information to the difference signal correction unit 32A based on luminance values and the synthesis unit 35A.

[0053] In the first pass (PATH-1), the luminance value-based difference signal correction unit 32A corrects the difference signal Na based on the luminance value, based on the luminance information output from the luminance conversion unit 31A. The luminance value-based difference signal correction unit 32A, for example, performs a correction by changing the correction gain when correcting the difference signal Na according to the luminance value of the difference signal Na. As a result, the optical noise component extraction unit 20 reduces the coloration of high-luminance areas due to overcorrection when subtracting optical noise.

[0054] In the second pass (PATH-2), the color-based difference signal correction unit 33A performs correction based on the color of the difference signal Na. The color-based difference signal correction unit 33A performs correction by changing the correction gain when correcting the difference signal Na according to the color of the difference signal Na. Subsequently, the magnitude-based difference signal correction unit 34A performs correction based on the magnitude of the difference signal Na. The magnitude-based difference signal correction unit 34A performs correction by changing the correction gain when correcting the difference signal Na according to the magnitude of the difference signal Na. In the second pass (PATH-2), in the case of a monochrome sensor, the color-based correction is skipped, and only the magnitude-based correction of the difference signal Na is performed. This reduces overcorrection when the optical noise component extraction unit 20 subtracts optical noise.

[0055] The synthesis unit 35A synthesizes the PATH-1 signal and the PATH-2 signal. The synthesis ratio is based on the luminance information output from the luminance conversion unit 31A. The result of the synthesis is output to the optical noise component subtraction unit 40 as a corrected difference signal Na'.

[0056] The color / monochrome switching signal in the brightness conversion unit 31A and the composite setting value in the composite unit 35A are input from the parameter setting unit 50. In addition, the respective correction setting values ​​(correction gains) in the difference signal correction unit 32A based on brightness value, the difference signal correction unit 33A based on color tone, and the difference signal correction unit 34A based on size are input from the parameter setting unit 50.

[0057] Figure 15 is a schematic block diagram showing one example configuration of the optical noise component correction unit 30 when the exposure time is different for three different cases.

[0058] The optical noise component correction unit 30 comprises a first correction unit 30A and a second correction unit 30B. The first correction unit 30A performs correction processing on the difference signal Na calculated in the optical noise component extraction unit 20 based on the pixel signal S12 from the second pixel group and the pixel signal S13 from the third pixel group. The second correction unit 30B performs correction processing on the difference signal Nb calculated in the optical noise component extraction unit 20 based on the pixel signal S11 from the first pixel group and the pixel signal S12 from the second pixel group.

[0059] The first correction unit 30A, as in the case of two types of exposure times (Figure 14), includes a luminance conversion unit 31A, a difference signal correction unit 32A based on luminance value, a difference signal correction unit 33A based on color, a difference signal correction unit 34A based on size, and a synthesis unit 35A. The first correction unit 30A, as in the case of two types of exposure times, takes the difference signal Na for each color and the pixel signal for creating luminance information (RGB signal Sca from the color sensor or B / W signal Swa from the monochrome sensor) as input and performs correction on the difference signal Na for each color.

[0060] The second correction unit 30B includes a luminance conversion unit 31B, a difference signal correction unit 32B based on luminance value, a difference signal correction unit 33B based on color, a difference signal correction unit 34B based on size, and a synthesis unit 35B. The second correction unit 30B takes the difference signal Nb for each color and the pixel signal for creating luminance information (RGB signal Scb from the color sensor or B / W signal Swb from the monochrome sensor) as input and performs correction on the difference signal Nb for each color.

[0061] The luminance conversion unit 31B calculates luminance information from the RGB signal Scob or the B / W signal Swb, and outputs the calculated luminance information to the difference signal correction unit 32B and the synthesis unit 35B, which are based on luminance values.

[0062] In the first pass (PATH-1), the luminance value-based difference signal correction unit 32B corrects the difference signal Nb based on the luminance value, based on the luminance information output from the luminance conversion unit 31B. The luminance value-based difference signal correction unit 32B performs a correction by changing the correction gain when correcting the difference signal Nb according to the luminance value of the difference signal Nb. As a result, the optical noise component extraction unit 20 reduces the coloration of high-luminance areas due to overcorrection when subtracting optical noise.

[0063] In the second pass (PATH-2), the color-based difference signal correction unit 33B performs correction based on the color of the difference signal Nb. The color-based difference signal correction unit 33B performs correction by changing the correction gain when correcting the difference signal Nb according to the color of the difference signal Nb. Subsequently, the magnitude-based difference signal correction unit 34B performs correction based on the magnitude of the difference signal Nb. The magnitude-based difference signal correction unit 34B performs correction by changing the correction gain when correcting the difference signal Nb according to the magnitude of the difference signal Nb. In the second pass (PATH-2), in the case of a monochrome sensor, the color-based correction is skipped, and only correction based on the magnitude of the difference signal Nb is performed. This reduces overcorrection when the optical noise component extraction unit 20 subtracts optical noise.

[0064] The synthesis unit 35B synthesizes the PATH-1 signal and the PATH-2 signal. The synthesis ratio is based on the luminance information output from the luminance conversion unit 31B. The result of the synthesis is output to the optical noise component subtraction unit 40 as a corrected difference signal Nb'.

[0065] The color / monochrome switching signal in the luminance conversion unit 31B and the composite setting value in the composite unit 35B are input from the parameter setting unit 50. In addition, the respective correction setting values ​​(correction gains) in the luminance value-based difference signal correction unit 32B, the color-based difference signal correction unit 33B, and the size-based difference signal correction unit 34B are input from the parameter setting unit 50.

[0066] (Optical Noise Component Subtraction Unit 40) Figure 16 is a schematic block diagram showing one example configuration of the optical noise component subtraction unit 40 when there are two different exposure times.

[0067] When there are two types of exposure times, the optical noise component subtraction unit 40 includes an image data distribution unit 41, an exposure ratio reciprocal multiplication unit 42A, an optical noise subtraction unit 43A, an optical noise subtraction unit 43B, and an image data merging unit 44.

[0068] The image data distribution unit 41 distributes the input signal (RAW data) for each color into pixel signals from the first pixel group (bright exposure) and pixel signals from the second pixel group (dark exposure).

[0069] The optical noise subtraction unit 43A performs subtraction processing on the pixel signals from the first pixel group that have been allocated, subtracting the optical noise component based on the corrected difference signal Na'. The subtraction gain when performing subtraction processing by the optical noise subtraction unit 43A is set in the parameter setting unit 50.

[0070] The exposure ratio reciprocal multiplication unit 42A multiplies the pixel signals from the allocated second pixel group by the reciprocal of the exposure ratio Ea. The optical noise subtraction unit 43B subtracts the optical noise component based on the corrected difference signal Na' from the pixel signals from the second pixel group that have been multiplied by the reciprocal of the exposure ratio Ea. The subtraction gain when performing the subtraction processing by the optical noise subtraction unit 43B is set in the parameter setting unit 50.

[0071] The image data merging unit 44 combines the pixel signal data from the first pixel group after subtraction processing by the optical noise subtraction unit 43A with the pixel signal data from the second pixel group after subtraction processing by the optical noise subtraction unit 43B, and outputs it as RAW data with suppressed afterimages. The image data merging unit 44 also rearranges the pixel signals, which have been sorted into two systems, the pixel signals from the first pixel group and the pixel signals from the second pixel group, by the image data sorting unit 41, back to the original input signal (RAW data) format.

[0072] Figure 17 is a schematic block diagram showing one example configuration of the optical noise component subtraction unit 40 when the exposure time is different for three different cases.

[0073] When there are three types of exposure times, the optical noise component subtraction unit 40 includes an image data distribution unit 41, an exposure ratio reciprocal multiplication unit 42A, an exposure ratio reciprocal multiplication unit 42B, an optical noise subtraction unit 43A, an optical noise subtraction unit 43B, an optical noise subtraction unit 43C, an optical noise subtraction unit 43D, an image data merging unit 44, an exposure ratio multiplication unit 45B, and an intermediate pixel merging unit 46.

[0074] The image data distribution unit 41 distributes the input signal (RAW data) for each color into pixel signals from the first pixel group (bright exposure), pixel signals from the second pixel group (intermediate exposure), and pixel signals from the third pixel group (dark exposure).

[0075] The optical noise subtraction unit 43A performs subtraction processing on the pixel signals from the second pixel group that have been allocated, subtracting the optical noise component based on the corrected difference signal Na'. The subtraction gain when performing subtraction processing by the optical noise subtraction unit 43A is set in the parameter setting unit 50.

[0076] The exposure ratio reciprocal multiplication unit 42A multiplies the pixel signals from the allocated third pixel group by the reciprocal of the exposure ratio Ea. The optical noise subtraction unit 43B subtracts the optical noise component based on the corrected difference signal Na' from the pixel signals from the third pixel group that have been multiplied by the reciprocal of the exposure ratio Ea. The subtraction gain when performing the subtraction processing by the optical noise subtraction unit 43B is set in the parameter setting unit 50.

[0077] The optical noise subtraction unit 43C performs subtraction processing of optical noise components based on the corrected difference signal Nb' on the pixel signals from the first pixel group that have been allocated. The subtraction gain when performing subtraction processing by the optical noise subtraction unit 43C is set in the parameter setting unit 50.

[0078] The exposure ratio reciprocal multiplication unit 42B multiplies the pixel signals from the allocated second pixel group by the reciprocal of the exposure ratio Eb. The optical noise subtraction unit 43D subtracts the optical noise component based on the corrected difference signal Nb' from the pixel signals from the second pixel group that have been multiplied by the reciprocal of the exposure ratio Eb. The subtraction gain when performing the subtraction processing by the optical noise subtraction unit 43D is set in the parameter setting unit 50.

[0079] The exposure ratio multiplication unit 45B multiplies the signal from the optical noise subtraction unit 43D by the exposure ratio Eb. The intermediate pixel synthesis unit 46 synthesizes the signal from the exposure ratio multiplication unit 45B and the signal from the optical noise subtraction unit 43A. The intermediate pixel synthesis unit 46 performs a synthesis process that changes the synthesis ratio according to the brightness thresholds K and L based on the brightness information. The thresholds K and L used when the intermediate pixel synthesis unit 46 performs the synthesis process are set in the parameter setting unit 50.

[0080] The image data merging unit 44 combines the pixel signal data from the first pixel group after subtraction processing by the optical noise subtraction unit 43C, the pixel signal data from the second pixel group after synthesis processing by the intermediate pixel synthesis unit 46, and the pixel signal data from the third pixel group after subtraction processing by the optical noise subtraction unit 43B, and outputs it as RAW data with suppressed afterimages. The image data merging unit 44 then rearranges the pixel signals, which have been distributed into three systems—the pixel signals from the first pixel group, the pixel signals from the second pixel group, and the pixel signals from the third pixel group—in the image data distribution unit 41, back to the original input signal (RAW data) format.

[0081] Figures 18 to 21 are schematic diagrams illustrating a specific example of the processing performed by the optical noise component subtraction unit 40. Figure 22 is a schematic diagram illustrating a specific example of the exposure ratio and specific examples of various values ​​calculated from the exposure ratio.

[0082] In the case of the two types of exposure times described above, the amount of noise in the optical noise component subtracted by the optical noise subtraction unit 43A and the optical noise subtraction unit 43B is determined according to the exposure ratio Ea between the exposure time of the first pixel group and the exposure time of the second pixel group. Furthermore, in the case of the three types of exposure times described above, the amount of noise in the optical noise component subtracted by the optical noise subtraction unit 43A, the optical noise subtraction unit 43B, the optical noise subtraction unit 43C, and the optical noise subtraction unit 43D is determined according to the exposure ratio Ea or the exposure ratio Eb.

[0083] Here, when the exposure ratio is m:n, if we let A be the difference in the PLS component as the observable optical noise component, then the values ​​B and C to be subtracted from the signal value can be determined as follows. Figures 18 to 21 show specific examples of the subtraction process for optical noise components for various exposure ratios. Figure 22 also shows a table summarizing specific examples of the subtraction process for optical noise components for various exposure ratios. A = (m - n) / m If B is the value to be subtracted from the signal value of a short exposure, multiplied by the reciprocal of the exposure ratio, then B = observed difference × 1 / A If C is the value to be subtracted from the signal value of a long exposure, then C = B × n / m

[0084] (Parameter setting unit 50) Figure 23 is a schematic diagram illustrating an example of setting the exposure time using the parameter setting unit 50 when there are two types of exposure times.

[0085] When there are two types of exposure times, the exposure time for the first pixel group (bright exposure) is set based on the appropriate exposure value of the AE, etc. The exposure time for the second pixel group (dark exposure) is set to an exposure time with an exposure ratio Ea that is not 1 times that of the exposure time for the first pixel group. The process for suppressing optical noise components according to the technology of one embodiment may be applied only to areas where noise due to light leakage occurs, and the exposure ratio Ea may be changed from 1:1 to m:n according to the ambient brightness, as shown in Figure 23.

[0086] Noise due to light leakage increases as the ambient brightness increases and the exposure time decreases. Therefore, if the exposure time determined based on the appropriate exposure value calculated after evaluating the ambient brightness is sufficiently long, noise due to light leakage can be ignored. For this reason, it is not necessary to perform the processing to suppress the light noise component according to the technology of one embodiment (in the case to the left of the boundary in Figure 23).

[0087] Figure 24 is a schematic diagram illustrating an example of how the parameter setting unit 50 sets the exposure time in three different exposure time cases.

[0088] When there are three exposure times, one of the exposure times for the first pixel group (bright exposure) and the second pixel group (intermediate exposure) is set based on the appropriate exposure value for AE, etc. The other is set to an exposure ratio Eb that is not 1x. Furthermore, the exposure ratio Eb does not need to be 1x, and is set based on the exposure ratio required to achieve HDR.

[0089] The exposure time for the third pixel group (dark exposure) is set to an exposure ratio Ea that is not 1x the exposure time for the second pixel group (intermediate exposure).

[0090] The process for suppressing optical noise components according to one embodiment of the technology may be applied only to areas where noise occurs due to light leakage, and the exposure ratio Ea may be changed from 1:1 to m:n according to the ambient brightness, as shown in Figure 24.

[0091] Otherwise, the process is the same as when there are two different exposure times.

[0092] (Regarding pre-white balance) When shooting outdoors, afterimages caused by light leakage noise tend to be colored magenta due to the spectral sensitivity characteristics of the image sensor 60 and the optical elements. In this case, pre-white balance processing can make the afterimages colorless, making them less noticeable. Another advantage is that the afterimages after suppression are less noticeable.

[0093] The exposure time for the first pixel group is set based on the appropriate exposure value of the AE, but the R:G:B exposure ratio relationship may also be adjusted to a ratio corresponding to the white balance, rather than being 1:1:1. It should be noted that this assumes an image sensor 60 that allows independent exposure time settings for RGB. The application of a ratio according to the white balance is performed, for example, for Bayer with 2x2 adjacent pixel drive, and Bayer and Quad Bayer with 4x4 adjacent pixel drive.

[0094] The parameter setting unit 50 calculates the white balance from past images taken during flight, for example, in the case of aerial photography by a drone, and outputs the relationship between the R:G:B exposure ratio to the exposure control unit 10, which is the appropriate exposure value for AE. The exposure control unit 10 sets the exposure time for each pixel group to an optimal value for each color so that the relationship between the exposure ratio for each pixel group for each color is based on the white balance of each color.

[0095] If there are two types of exposure times, the exposure control unit 10 may set the exposure time for the first pixel group (bright exposure) based on the appropriate exposure value of AE and the R:G:B exposure ratio. The exposure time for the second pixel group (dark exposure) is set to an exposure time with an exposure ratio Ea that is not 1 times that of the exposure time for the first pixel group. The exposure ratio Ea is set by the parameter setting unit 50.

[0096] If there are three types of exposure times, the exposure time for the first pixel group (bright exposure) may be set based on the appropriate exposure value of the AE and the R:G:B exposure ratio mentioned above. The exposure time for the second pixel group (intermediate exposure) is set based on the appropriate exposure value of the AE and the R:G:B exposure ratio mentioned above.

[0097] [1.3 Modifications] In the above explanation, examples were given of cases where multiple pixels are divided into two or three pixel groups for each color and exposed with two or three different exposure times. However, the scope of application of this technology is not limited to these examples. It can also be applied when multiple pixels are divided into four or more pixel groups for each color and exposed with four or more different exposure times for each color.

[0098] Furthermore, although the above description has given an example in which the image sensor 60 and the signal processing device according to one embodiment are configured separately, some of the component blocks of the signal processing device according to one embodiment may be built into the image sensor 60.

[0099] [1.4 Effects] As described above, according to the imaging device of one embodiment, the optical noise component is extracted by obtaining a difference signal between the pixel signal from one of a plurality of pixel groups exposed at different exposure times and the signal obtained by multiplying the pixel signal from a pixel group different from the one of the plurality of pixel groups by the reciprocal of the exposure ratio. Then, based on the difference signal and the exposure ratio, a process is performed to subtract the optical noise component from the pixel signal from each of the plurality of pixel groups. This improves the accuracy of optical noise calculation and makes it possible to suppress optical noise with high precision.

[0100] According to one embodiment of the imaging device, shutter efficiency, which is a key issue in a global shutter type image sensor 60, can be improved. According to one embodiment of the imaging device, a practical improvement effect of suppressing optical noise through correction can be obtained without significantly changing the pixel structure. Furthermore, according to one embodiment of the signal processing device, since there is no need to use frame memory, it can be built into the image sensor 60.

[0101] (Example of applicable application) When capturing high-resolution still images with a moving object such as a drone, afterimages may occur around high-brightness subjects. Afterimages occur when the camera moves relative to the high-brightness subject, or when the high-brightness subject moves relative to the camera. According to one embodiment of the imaging device, afterimages can be suitably suppressed by extracting the afterimage component and subtracting it from the original pixel signal.

[0102] Furthermore, according to one embodiment of the imaging device, afterimages can be suitably suppressed when photographing a workpiece moving at high speed using an industrial inspection camera. Afterimages can also be suitably suppressed when photographing moving subjects in backlit conditions.

[0103] The effects described herein are merely illustrative and not limiting, and other effects may also exist. The same applies to the effects of other embodiments described later.

[0104] <2. Other Embodiments> The technology described herein is not limited to the above-described embodiment and can be implemented in various modified forms.

[0105] For example, this technology can also take the following configuration. According to this configuration, the optical noise component is extracted by calculating a difference signal between the pixel signal from one of a group of pixels exposed at different exposure times and the pixel signal from a different pixel group, multiplied by the reciprocal of the exposure ratio. Then, based on the difference signal and the exposure ratio, the optical noise component is subtracted from the pixel signals from each of the multiple pixel groups. This makes it possible to provide a signal processing device and an imaging device that can improve the accuracy of optical noise calculation and suppress optical noise with high precision.

[0106] (1) An image sensor having multiple pixels for each of one or more colors, comprising: an exposure control unit that divides the multiple pixels into multiple pixel groups for each color and exposes each of the multiple pixel groups with different exposure times; an extraction unit that extracts optical noise components by obtaining a difference signal between a pixel signal from one of the multiple pixel groups and a signal obtained by multiplying the pixel signal from a different pixel group from the one pixel group by the reciprocal of the exposure ratio; and a subtraction unit that performs a process of subtracting the optical noise components from the pixel signals from each of the multiple pixel groups based on the difference signal and the exposure ratio. (2) The signal processing device according to (1) above, wherein the extraction unit obtains the difference signal based on each pixel signal after low-pass filter processing has been performed on the pixel signals from each of the multiple pixel groups. (3) The signal processing device according to (1) or (2) above, further comprising: a correction unit that corrects the difference signal. (4) The signal processing device according to (3) above, wherein the correction unit performs a correction on the difference signal according to the brightness value of the difference signal. (5) The signal processing device according to (3) or (4) above, wherein the correction unit performs a correction on the difference signal according to the color of the difference signal. (6) The signal processing device according to any one of (3) to (5) above, wherein the correction unit performs a correction on the difference signal according to the magnitude of the difference signal. (7) The signal processing device according to any one of (1) to (6) above, wherein the exposure control unit sets the exposure time of one of the plurality of pixel groups based on the appropriate exposure value of AE (Auto Exposure), and sets the exposure time of a pixel group different from the one pixel group to an exposure time with an exposure ratio that is not 1x to the exposure time of the one pixel group. (8) The signal processing device according to any one of (1) to (7) above, wherein the image sensor has a plurality of pixels for each of a plurality of colors, and the exposure control unit sets the exposure time of the different pixel groups for each color to an optimal value so that the relationship of the exposure ratio for each of the different pixel groups for each color becomes a value based on the white balance of each color. (9) The signal processing apparatus according to any one of (1) to (8) above, wherein the pixel array of the plurality of pixels is a Bayer array.(10) The signal processing device according to any one of (1) to (9) above, wherein the pixel array of the plurality of pixels is a Bayer array in which two or more adjacent pixels are of the same color. (11) An imaging device comprising: an image sensor having a plurality of pixels for one or more colors; an exposure control unit that divides the plurality of pixels in the image sensor into a plurality of pixel groups for each color and exposes each of the plurality of pixel groups with different exposure times; an extraction unit that extracts optical noise components by obtaining a difference signal between a pixel signal from one of the plurality of pixel groups and a signal obtained by multiplying the pixel signal from a pixel group different from the one of the plurality of pixel groups by the reciprocal of the exposure ratio; and a subtraction unit that performs a process of subtracting the optical noise components from the pixel signals from each of the plurality of pixel groups based on the difference signal and the exposure ratio.

[0107] This application claims priority based on Japanese Patent Application No. 2024-163176, filed with the Japan Patent Office on 19 September 2024, and all contents of that application are incorporated herein by reference.

[0108] Those skilled in the art will understand that various modifications, combinations, subcombinations, and changes can be conceived depending on design requirements and other factors, and that these fall within the scope of the attached claims and their equivalents.

Claims

1. An exposure control unit that divides the multiple pixels in an image sensor having multiple pixels for each of one or more colors into multiple pixel groups for each color and exposes each of the multiple pixel groups with different exposure times; an extraction unit that extracts optical noise components by obtaining a difference signal between a pixel signal from one of the multiple pixel groups and a signal obtained by multiplying the pixel signal from a pixel group different from the one pixel group by the reciprocal of the exposure ratio; and a subtraction unit that performs a process of subtracting the optical noise components from the pixel signals from each of the multiple pixel groups based on the difference signal and the exposure ratio.

2. The signal processing apparatus according to claim 1, wherein the extraction unit obtains the difference signal based on each pixel signal after applying a low-pass filter to the pixel signals from each of the plurality of pixel groups.

3. The signal processing apparatus according to claim 1, further comprising a correction unit for correcting the difference signal.

4. The signal processing apparatus according to claim 3, wherein the correction unit performs a correction on the difference signal according to the brightness value of the difference signal.

5. The signal processing apparatus according to claim 3, wherein the correction unit performs a correction on the difference signal according to the color of the difference signal.

6. The signal processing apparatus according to claim 3, wherein the correction unit performs a correction on the difference signal according to the magnitude of the difference signal.

7. The signal processing apparatus according to claim 1, wherein the exposure control unit sets the exposure time of one of the plurality of pixel groups based on the appropriate exposure value of AE (Auto Exposure), and sets the exposure time of a pixel group different from the one pixel group to an exposure time with an exposure ratio that is not 1x to the exposure time of the one pixel group.

8. The signal processing apparatus according to claim 7, wherein the image sensor has a plurality of pixels for each of a plurality of colors, and the exposure control unit sets the exposure time of each of the different pixel groups for each color to an optimal value such that the relationship of the exposure ratio for each of the different pixel groups for each color becomes a value based on the white balance of each color.

9. The signal processing apparatus according to claim 1, wherein the pixel array of the plurality of pixels is a Bayer array.

10. The signal processing apparatus according to claim 1, wherein the pixel array of the plurality of pixels is a Bayer array in which two or more adjacent pixels are of the same color.

11. An imaging device comprising: an image sensor having multiple pixels for one or more colors; an exposure control unit that divides the multiple pixels in the image sensor into multiple pixel groups for each color and exposes each of the multiple pixel groups with different exposure times; an extraction unit that extracts optical noise components by obtaining a difference signal between a pixel signal from one of the multiple pixel groups and a signal obtained by multiplying the pixel signal from a different pixel group from the multiple pixel groups by the reciprocal of the exposure ratio; and a subtraction unit that performs a process of subtracting the optical noise components from the pixel signals from each of the multiple pixel groups based on the difference signal and the exposure ratio.

Images