Image processing device, image processing method, and program

Abstract

The present invention improves image quality in an image processing device for processing an image.　This image processing device is provided with a color basis transforming unit and an image quality adjustment processing unit. In the image processing device, the color basis transforming unit out of the color basis transforming unit and the image quality adjustment processing unit performs, on an image, color basis transformation processing for transforming a color space from a first color space to a second color space. The image quality adjustment processing unit out of the color basis transforming unit and the image quality adjustment processing unit performs predetermined image quality adjustment processing on the image subjected to the color basis transformation processing by color basis transformation.

Description

Image processing device, image processing method, and program

[0001] The present technology relates to an image processing device, and more particularly to an image processing device that processes images in various color spaces, an image processing method in the image processing device, and a program for causing a computer to execute the method.

[0002] Conventionally, various types of image processing, such as white balance correction, have been performed on images to improve image quality. For example, an image processing device has been proposed that performs processing to correct sensitivity differences between pixels before converting a Bayer array RAW image into a demosaic image (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2015-154307

[0004] The above-mentioned conventional technology aims to improve the quality of Bayer-array images by correcting sensitivity differences. However, the above-mentioned image processing device has difficulty improving image quality when performing processes such as white balance correction on images in a color space other than RGB. For example, when the color space is not RGB, color reproducibility cannot be maintained unless the gain during white balance correction is increased, but a large gain increases noise. On the other hand, reducing the gain reduces noise but degrades color reproducibility. As such, the above-mentioned conventional technology encounters problems such as a trade-off between improved color reproducibility and noise reduction, making it difficult to improve image quality.

[0005] The present technology was developed in view of such circumstances, and aims to improve image quality in an image processing device that processes images.

[0006] The present technology has been made to solve the above-mentioned problems, and a first aspect thereof is an image processing device including a color base conversion unit that performs color base conversion processing on an image to convert the color space from a first color space to a second color space, and an image quality adjustment processing unit that performs predetermined image quality adjustment processing on the image after the color base conversion processing, an image processing method for the image processing device, and a program for causing a computer to execute the method, thereby improving image quality.

[0007] In addition, in the first aspect, the image processing apparatus may further include a demosaic processing unit that converts a RAW image into a demosaic image, and the image may be the RAW image or the demosaic image. This brings about an effect of improving the image quality of the demosaic image.

[0008] In addition, in the first aspect, the image processing apparatus may further include a photometry / AWB (Auto White Balance) unit that detects the demosaic image after the color base conversion processing and estimates a white balance gain, and an amplifier that amplifies color signals in the demosaic image after the color base conversion processing by the white balance gain and supplies the amplified color signals to the image quality adjustment processing unit, wherein the color base conversion unit performs the color base conversion processing on the demosaic image, and the image quality adjustment processing unit includes a color reproduction unit that corrects the amplified color signals using a color reproduction matrix, and a white clip processing unit that limits the corrected color signals by a predetermined threshold. This provides the effects of improving color reproducibility and reducing noise.

[0009] In addition, in the first aspect, the image processing apparatus may further include an AWB unit that estimates a white balance gain, the color basis conversion unit performs the color basis conversion process on the detection value of the RAW image and supplies the result to the AWB unit, and the image quality adjustment processing unit may include a color reproduction unit that amplifies color signals in the demosaic image using the white balance gain and corrects them using a color reproduction matrix. This improves color reproducibility and reduces noise.

[0010] In this first aspect, the first color space may be a color space different from the RGB space, and the second color space may be the RGB space, thereby providing an effect of converting from a space other than the RGB space to the RGB space.

[0011] In addition, in the first aspect, a pixel array unit that generates the image may be further provided, thereby providing an effect that processing is performed within the image sensor.

[0012] In the first aspect, the pixel array unit may include an array of pixels including clear pixels, and the clear pixels may be provided with transparent filters, thereby converting the color space of an image including the clear pixels.

[0013] In this first aspect, the image quality adjustment processing unit may perform at least one of gamma correction processing, edge enhancement processing, YUV conversion processing, and color conversion processing, thereby providing an effect of adjusting image quality.

[0014] 1 is a block diagram showing an example configuration of an imaging device according to a first embodiment of the present technology. FIG. 2 is a block diagram showing an example configuration of an image sensor according to the first embodiment of the present technology. FIG. 3 is a block diagram showing an example configuration of an image sensor according to the prior art. FIG. 4 is a diagram showing an example of processing up to color reproduction according to the prior art. FIG. 5 is a diagram showing details of processing up to color reproduction according to the prior art. FIG. 6 is a diagram showing an example of a Bayer array according to the prior art. FIG. 7 is a graph showing an example of spectral characteristics according to the prior art. FIG. 8 is an example of a factor graph according to the prior art. FIG. 9 is an example of a constrained color reproduction matrix according to the prior art. FIG. 10 is an example of white clip processing according to the first embodiment of the present technology. FIG. 11 is a diagram showing an example of an RGB space according to the prior art. FIG. 12 is a diagram showing an example of an RCB space according to the prior art. FIG. 13 is a diagram showing an example of a white balance gain and a color reproduction matrix according to the prior art. FIG. 14 is a diagram showing an example of a color space according to the first embodiment of the present technology. FIG. 15 is a diagram showing an example of a transformation matrix according to the first embodiment of the present technology. FIG. 16 is a diagram showing an example of processing within a photometry / AWB unit according to the first embodiment of the present technology. FIG. 17 is a diagram showing an example of a plot of detection values ​​according to the first embodiment of the present technology. FIG. 18 is a flowchart showing an example of a calibration method according to the first embodiment of the present technology. FIG. 19 is a diagram showing an example of a method of calculating correction matrix coefficients according to the first embodiment of the present technology. Fig. 1 is a diagram showing an example of a white balance gain and a color correction matrix according to a first embodiment of the present technology. Fig. 2 is a block diagram showing an example of a configuration of an image sensor according to a second embodiment of the present technology. Fig. 3 is a diagram showing an example of a color basis conversion process according to the second embodiment of the present technology. Fig. 4 is a block diagram showing a schematic example of a configuration of a vehicle control system. Fig. 5 is an explanatory diagram showing an example of an installation position of an imaging unit.

[0015] Hereinafter, modes for carrying out the present technology (hereinafter referred to as embodiments) will be described. The description will be given in the following order: 1. First embodiment (an example of performing color basis conversion processing on a demosaic image) 2. Second embodiment (an example of performing color basis conversion processing on detected values ​​of a RAW image) 3. Example of application to a moving body

[0016] 1. First Embodiment [Configuration Example of Imaging Device] Fig. 1 is a block diagram showing a configuration example of an imaging device 100 according to a first embodiment of the present technology. The imaging device 100 is a device for capturing image data and includes an optical unit 110, an image sensor 200, and a DSP (Digital Signal Processing) circuit 120. The imaging device 100 further includes a display unit 130, an operation unit 140, a bus 150, a frame memory 160, a storage unit 170, and a power supply unit 180. The imaging device 100 may be a digital camera or an electronic device with an imaging function (such as a smartphone or a personal computer). The imaging device 100 is an example of an image processing device as defined in the claims.

[0017] The optical unit 110 collects light from a subject and guides it to the image sensor 200. The image sensor 200 generates image data by photoelectric conversion. The image sensor 200 processes the generated image data and supplies the processed data to the DSP circuit 120 via a signal line 209.

[0018] The DSP circuit 120 performs predetermined signal processing on the data from the image sensor 200. The DSP circuit 120 outputs the processed data via a bus 150 to a frame memory 160 or the like.

[0019] The display unit 130 displays various types of data. The display unit 130 may be, for example, a liquid crystal panel or an organic EL (Electro Luminescence) panel. The operation unit 140 generates operation signals in accordance with user operations.

[0020] The bus 150 is a common path for the optical unit 110, image sensor 200, DSP circuit 120, display unit 130, operation unit 140, frame memory 160, storage unit 170, and power supply unit 180 to exchange data with one another.

[0021] The frame memory 160 holds image data. The storage unit 170 stores various data such as image data. The power supply unit 180 supplies power to the image sensor 200, the DSP circuit 120, the display unit 130, and the like.

[0022] 2 is a block diagram showing an example of the configuration of an image sensor 200 according to the first embodiment of the present technology. The image sensor 200 includes a pixel array unit 210, a demosaic processing unit 220, a color basis conversion unit 230, a photometry and AWB unit 241, an amplifier unit 242, a color reproduction unit 250, and a white clip processing unit 260. The image sensor 200 further includes a gamma correction unit 271, an edge enhancement unit 272, a YUV conversion unit 273, a color conversion unit 274, and a DNN (Deep Neural Network) monitor 275.

[0023] Note that some or all of the downstream processing (such as demosaic processing) of the pixel array unit 210 can be performed by a circuit external to the image sensor 200 (such as the DSP circuit 120).

[0024] A plurality of pixels 211 are arranged in a two-dimensional grid pattern within the pixel array unit 210. Each of the pixels 211 generates a pixel signal through photoelectric conversion. The pixel array unit 210 generates a RAW image in a color space different from RGB. A RAW image is an image in which each pixel has one piece of color information and is also called a mosaic image. For example, the pixel array unit 210 includes a plurality of pixels, including clear pixels, arranged in an array conforming to a Bayer array.

[0025] Here, a clear pixel is a pixel provided with a transparent filter having a transmittance of a predetermined value (e.g., 0.5) or more across the entire visible light range. A clear pixel is also called a clarity pixel. A clear pixel is referred to as a C1 pixel, and pixels of different colors other than the clear pixel are referred to as a C0 pixel and a C2 pixel. For example, an R (Red) pixel and a B (Blue) pixel are used as the C0 pixel and the C2 pixel. An R pixel is a pixel provided with a color filter that selectively transmits light in the red wavelength range, and a B pixel is a pixel provided with a color filter that selectively transmits light in the blue wavelength range.

[0026] Note that R and G (Green) pixels, or Y (Yellow) and Cy (Cyan) pixels can also be used as the C0 and C2 pixels. The G pixel is a pixel provided with a color filter that selectively transmits light in the green wavelength range, and the Y pixel is a pixel provided with a color filter that selectively transmits light in the yellow wavelength range. The Cy pixel is a pixel provided with a color filter that selectively transmits light in the cyan wavelength range. The C0 to C2 pixels are arranged in a Bayer array in which the position of the G pixel is replaced with the C1 pixel. The color space resulting from the arrangement of these pixels is hereinafter referred to as a "C0C1C2 space." Within the C0C1C2 space, a space in which the C0 and C2 pixels are R and B pixels is referred to as an "RCB space."

[0027] The pixel array unit 210 supplies the RAW image in the C0C1C2 space to the demosaic processing unit 220 via various circuits such as an ADC (Analog to Digital Converter). In the figure, the circuits between the pixel array unit 210 and the demosaic processing unit 220 are omitted.

[0028] The demosaic processing unit 220 converts the RAW image into a demosaic image. The demosaic image is an image in which missing color information among the three colors is complemented for each pixel. The demosaic processing unit 220 supplies the demosaic image to the color basis conversion unit 230.

[0029] The color basis conversion unit 230 performs color basis conversion processing on the demosaic image, converting the color space from the C0C1C2 space to the RGB space using a basis conversion matrix. Details of the basis conversion matrix will be described later. The color basis conversion unit 230 supplies the demosaic image after the color basis conversion processing to the photometry / AWB unit 241 and the amplifier 242. The converted demosaic image includes three color signals of R, G, and B for each pixel. Note that although the color basis conversion unit 230 converts the C0C1C2 space to the RGB space, the color space to be converted is not limited to the C0C1C2 space.

[0030] The C0C1C2 space is an example of a first color space described in the claims, and the RGB space is an example of a second color space described in the claims.

[0031] In accordance with the processing delay of the photometry and AWB unit 241, the output timing to the amplification unit 242 is delayed relative to the output timing to the photometry and AWB unit 241. A buffer can also be arranged between the color base conversion unit 230 and the amplification unit 242.

[0032] The photometry and AWB unit 241 measures (in other words, detects) the demosaic image from the color basis conversion unit 230 to estimate a white balance gain and a correlated color temperature. The white balance gain includes gains for each of R, G, and B. The photometry and AWB unit 241 supplies the white balance gain to the amplification unit 242 and the correlated color temperature to the color reproduction unit 250.

[0033] The amplifier 242 amplifies the color signals in the demosaic image by a white balance gain. An amplifier that amplifies the corresponding color signal for each color is arranged in the amplifier 242. Because the demosaic image includes color signals of three colors for each pixel, three amplifiers are arranged in the amplifier 242. The amplifier 242 supplies the amplified color signals to the color reproduction unit 250.

[0034] The color reproduction unit 250 corrects the color signals from the amplifier unit 242 using a color correction matrix (CCM). This CCM is also called a linear matrix. The color reproduction unit 250 stores in advance a set of correction matrix coefficients in the CCM for each of a plurality of color temperatures. The color reproduction unit 250 then performs correction using the CCM of the correction matrix coefficients corresponding to the correlated color temperature estimated by the photometry and AWB unit 241, and supplies the corrected demosaic image to the white clip processing unit 260. Details of the processing performed by the color reproduction unit 250 will be described later. Note that the circuits subsequent to the color reproduction unit 250 are an example of the image quality adjustment processing unit described in the claims.

[0035] The white clip processing unit 260 limits the corrected color signals using a predetermined threshold. If the R, G, or B color signals are greater than the upper threshold, the white clip processing unit 260 outputs the threshold. If the color signals are less than the threshold, the white clip processing unit 260 outputs the color signals as is. The white clip processing unit 260 supplies the processed image to the gamma correction unit 271.

[0036] The gamma correction unit 271 performs gamma correction on the image from the white clip processing unit 260 and supplies the image to the edge enhancement unit 272. The edge enhancement unit 272 performs edge enhancement on the image from the gamma correction unit 271 and supplies the image to the YUV conversion unit 273. The YUV conversion unit 273 converts the color space of the image from the edge enhancement unit 272 from RGB to YUV and supplies the image to the color conversion unit 274. The color conversion unit 274 converts the color space of the image from the YUV conversion unit 273 to match the input data format of the DNN monitor 275. For example, YUV is converted to a color space such as Lab, RGB, or CMYK.

[0037] The DNN monitor 275 uses DNN to perform image recognition on the image from the color conversion unit 274. The DNN monitor 275 outputs the recognition results and image data to the DSP circuit 120.

[0038] The processes from the gamma correction unit 271 to the DNN monitor 275 are an example of post-stage processing of the white clip processing unit 260, and these processes are executed as necessary. The image sensor 200 can execute only some of the processes from the gamma correction unit 271 to the DNN monitor 275, or can execute processes different from these.

[0039] Of the above-described configuration, the parts enclosed by dotted lines differ from those of the prior art, namely, the color base conversion unit 230, the photometry and AWB unit 241, the amplifier unit 242, the color reproduction unit 250, and the white clip processing unit 260. To explain the effects of the present invention, a prior art image sensor 200 in which the parts enclosed by dotted lines are replaced with a general configuration will be considered as a comparative example.

[0040] 3 is a block diagram showing an example of the configuration of a conventional image sensor 200. In the conventional technology, the color base conversion unit 230 and the white clip processing unit 260 are not provided.

[0041] The pixel array unit 210 also generates a RAW image in RGB space and supplies it to a photometry / AWB unit 241 and an amplifier unit 242. The photometry / AWB unit 241 detects the RAW image, estimates a white balance gain and a correlated color temperature, and supplies them to the amplifier unit 242 and a color reproduction unit 250.

[0042] The amplifier 242 amplifies the color signals in the RAW image using a white balance gain and supplies the amplified color signals to the demosaic processing unit 220. The demosaic processing unit 220 converts the RAW image into a demosaic image and supplies the demosaic image to the color reproduction unit 250. The color reproduction unit 250 corrects the color signals in the demosaic image using CCM and supplies the demosaic image to the gamma correction unit 271.

[0043] Generally, single-chip image sensors have pixel arrays with various patterns when viewed in two-dimensional coordinates, and are equipped with various color filters depending on the application. A common camera signal processing method exists for pixels with these diverse pixel arrays and color filters. This involves demosaicing the spatial phase differences of each color, followed by color reproduction processing at each pixel position. This process converts the signals into the final video signal or the signal required by the digital still camera. To give a specific example, in recent years, in autonomous driving technology using AI (artificial intelligence), three-color RGB or YUV signals with spatially aligned phases at each pixel position are widely used as input vectors in signal processing with recognition algorithms such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), or attention-based transformers. Most of these input signals are generated using technology inherited from traditional digital still cameras and camcorders, which were primarily used for viewing purposes.

[0044] The figure shows a schematic representation of these common and essential signal processing steps. First, pixel data is input sequentially in time series from a pixel array with a Bayer pattern. It passes through a photometry / AWB unit 241 and an amplifier unit 242, and then demosaic processing aligns the spatial phase and number of pixels for each RGB channel. Then, processing such as a color reproduction matrix is ​​performed to adjust the color gamut, and the data is sent to subsequent processing (gamma correction, edge enhancement, YUV conversion, etc.) to produce the final output signal. Note that this configuration example is only a partial example, and various pixel corrections (defect correction, black level adjustment, various shading corrections, etc.) performed before white balance are omitted because they are not directly related to the present invention.

[0045] Examples of such signal processing include those described in documents such as Japanese Patent Application Laid-Open No. 2015-154307 and International Publication No. 2011 / 152174. Among these, demosaic processing is implemented using a wide variety of algorithms, with many examples including the Bayer pattern. Many arrays other than the Bayer pattern also exist, including those described in International Publication No. 2016 / 098641 and International Publication No. 2013 / 145487. The common purpose of these demosaic processes is to generate basic signals for adjusting various spatial frequency components, such as color reproduction, luminance reproduction, and edge processing, in subsequent signal processing, under conditions where the spatial phase of pixel positions is aligned.

[0046] FIG. 4 shows part of the process of FIG. 3, focusing on the CCM used in the amplifier 242 and color reproduction unit 250 before and after demosaic processing used in a general RGB-Bayer type image sensor.

[0047] Furthermore, Fig. 5 shows a specific example of the configuration of Fig. 4 using specific basic function processing (low-pass filter, minimum value extraction processing, average value calculation, NR, multiplier, etc.). In Fig. 5, "LPF" indicates a low-pass filter, "min" indicates minimum value extraction processing, "mean" indicates average value calculation processing, and "x" indicates a multiplier. "+" indicates an adder, and Gm indicates a statistical quantity (average value, etc.) of Gr and Gb.

[0048] Now, with the recent advances in autonomous driving technology, sensitivity in low light conditions has become an issue (for example, to detect areas outside the headlight illumination range), and in addition to the conventional Bayer-type RGB image sensors that mainly use G pixels, many image sensors that use clear pixels to increase the sensitivity of the image sensor are being adopted, as shown in Figure 6.

[0049] However, as shown in Figure 7, the spectral characteristics of image sensors generally vary significantly, resulting in color reproduction issues compared to RGB sensors, even if sensitivity is improved. In Figure 7, "a" indicates the spectral characteristics of each of the R, Gr, Gb, and B pixels. In Figure 7, "b" indicates the spectral characteristics of each of the R, Cr, Cg, and B pixels. The Cr and Cg pixels are clear pixels. In Figure 7, the vertical axis of "a" and "b" indicates relative sensitivity, and the horizontal axis indicates wavelength.

[0050] This color reproduction issue is expressed using a more simplified factor graph in Figure 8. In the factor graph in Figure 8, the circles represent the pixel values ​​of each color, and the squares represent various computing units (arithmetic units, conditional branching units, comparators, etc., which are abstractions of the computing units in Figure 5 using a single symbol). Here, each square is arranged horizontally to represent the same processing (although coefficients, etc., may differ), and vertical squares represent different computing units. In other words, this focuses on the input / output relationships in Figures 4 and 5. Each function is briefly indicated as an annotation on the left side of the figure. Now, let's consider the white point of the light source, which can be said to be the origin of color reproduction.

[0051] Generally, the white point expressed in the CIE 1931 color space and the like is expressed as R = G = B in the RGB space. For a typical D65 light source, the white point, which has coordinates of (0.3127, 0.3290, 0.3583) in the XYZ space, is (0.3290, 0.3290, 0.3290) in the RGB space. While the coordinate position of white varies depending on the light source, widely used standard white points such as the D illuminant are important, and the following discussion will focus on color reproduction under the D illuminant. However, the white points of other light sources (light sources with different color temperatures) are also considered to be fixed points, with only their coordinates being different, so they can be understood to be the same as the D illuminant.

[0052] As can be seen from Figure 8, even if the colors of a white subject have different values ​​in the initial input channel, they are corrected to R=Gr=Gb=B in the upstream photometry / AWB unit 241 and amplifier unit 242. From there, it can be seen that R=G=B is maintained up to the stage preceding the color reproduction matrix (CCM) in the color reproduction unit 250. This is because, while the white balance gain values ​​and color reproduction matrix coefficients differ for each channel, the coefficients of the other calculators are the same for each channel (and this is how it should be). Therefore, since R=G=B is maintained up to the stage preceding the color reproduction unit 250, it becomes necessary for the color reproduction matrix coefficients to maintain R=G=B for its output as well. A commonly used method for achieving this is to impose constraints on this color reproduction matrix.

[0053] Figure 9 shows an example of this constraint. By adjusting the coefficients of each channel so that the sum of them is 1.0, the white point can be treated as a fixed point before and after processing, and white color is preserved. As such, it is clear that in conventional technology, it is important to determine coefficients so that the white point is preserved as a fixed point before and after color reproduction processing and other colors can be reproduced more accurately. However, if this constraint is not imposed, the degree of freedom of the color reproduction matrix may increase, resulting in improvements, so a method is also used in which the white point is allowed to move slightly before and after conversion.

[0054] In such cases, as shown in the example of Figure 10, a white clip or similar is often provided after the color reproduction process to maintain white. The above signal processing is valid when focusing on the RGB sensor. However, based on the above discussion, it is clear that it is impossible to maintain the white point of a D65 light source with the other color filters shown in Figure 6. This raises the following questions in a color space different from the RGB space: 1. Can the white point be maintained? 2. How is the color reproduction matrix defined? 3. What exactly is white balance?

[0055] For example, consider the RCB space. Here, a clear pixel is introduced instead of the G pixel. Then, the question arises as to whether the condition R=C=B is the white point when C is the value of the clear pixel in a D65 light source. In general, the spectral characteristics D(λ) of a D light source that distributes the visible light range with almost no attenuation points, the same subject (same spectral reflectance: O(λ)) and the same optical system characteristics (transmittance, etc.) L(λ) are used to calculate the spectral characteristics S of the G pixel and clear pixel of the RGGB sensor and RCCB sensor shown in FIG. Ｇ (λ) and S Ｃ When we multiply (λ) by each wavelength and integrate, we get the following equation:

[0056] This means that under the same light source, subject (same spectral reflectance) and optical characteristics, when R=G=B, R=B<C. Generally, the following equation holds true:

[0057] In other words, the white point defined in the RGB color space does not coincide with the coordinates where the colors that make up the white point are equal in an arbitrary xyz color space. Figures 11 and 12 explain the geometric meaning of the color reproduction matrix.

[0058] Figure 11 shows an example of a color solid defined in RGB space (e.g., sRGB space) and the white point of a certain light source, as well as representative colors such as magenta (Mg), cyan (Cy), and yellow. In this case, when the input and output colors are in the same RGB space, calculations using the color reproduction matrix involve rotating the solid around the axis connecting the white point and the origin (black) as the rotation axis, and then arbitrarily stretching or shrinking the solid along the rotation axis. These geometric operations are primarily used to determine the conversion coefficients. To actually calculate the coefficients, first, an appropriate color chart (e.g., 24 colors) is photographed under a specific light source (e.g., D65 light source), and the captured raw data is input into a signal processing block (e.g., Figure 5 ). The matrix coefficients are then assumed to be unit matrices. The color reproduction matrix coefficients are obtained by calculating the least-squares distance between the resulting RGB image and the RGB values ​​of each color in the reference RGB image (target image). There are various methods for calculating the color reproduction matrix coefficients, but a commonly known method is to calculate the coefficients by comparing the reference image as shown here. As mentioned above, the transformations of the coefficients obtained in this way are ideally mainly rotation and expansion / contraction in the axial direction in the RGB space (in reality, some arbitrary spatial transformation is also involved, but this is not dominant).

[0059] However, when the input and output coordinate data are in different color spaces, conversion from RCB space to RGB space, for example, means that the color solid in RCB space, enclosed by solid lines as shown in Figure 12, is transformed into the color solid in RGB space enclosed by dotted lines. As shown in the figure, this conversion requires an arbitrary transformation that also involves moving the white point (which indicates R=C=B in RCB space). This indicates that the volume of the color solid before and after conversion can also change significantly. In other words, this is equivalent to an increase in the absolute value of the determinant of the color reproduction matrix. In other words, the larger the transformation itself, the more accurate the coefficient calculation. However, in reality, under conditions where the least-squares distance method described above is considered optimal, it can be said that there is a limit to the accuracy of this color reproduction process.

[0060] 13 shows the white balance gain values, color reproduction matrix coefficients, and determinants obtained under illuminant D for the RGGB sensor and RCCB sensor having the characteristics shown in FIG. 7. In the figure, a shows the white balance gain values, color reproduction matrix coefficients, and determinants obtained under illuminant D for the RGGB sensor. In the figure, b shows the white balance gain values, color reproduction matrix coefficients, and determinants obtained under illuminant D for the RCCB sensor.

[0061] As shown in Figure 5A and Figure 5B, significant differences in gain values ​​and coefficient magnitudes (the degree of deformation of the color solid) are evident. When such differences exist, the greater the deformation, the worse the final color reproduction index ΔE2000 becomes. In fact, the difference between the two is known to be as much as ΔE2000 = 39. Furthermore, there are also white balance gain values ​​that differ by nearly twofold, which, combined with the matrix coefficients, amplifies noise. In other words, when an RCCB sensor is developed using signal processing such as that shown in Figure 5A, the trade-off between color reproducibility and noise can become an issue. Emphasizing color reproducibility tends to increase the white balance gain values ​​and matrix coefficients, amplifying noise at the same time. Conversely, lowering these gains and coefficients to suppress noise can lead to problems with degraded color reproducibility.

[0062] Next, let's consider "What is white balance?" Up until now, we've assumed that the white balance gain of the RCCB sensor has been determined, i.e., R = C = B. This condition of equality between color channels is often required in demosaicing. For example, it is used in RGB space to detect the direction of pixel interpolation to suppress false color in achromatic areas. In Figures 5 and 8, the correlation between pixels is used to find the minimum value based on the magnitude relationship between each color difference (R-G, B-G), and to detect, for example, the direction of an edge, thereby performing correct interpolation. In other words, because it is necessary for the R, G, and B shades in achromatic areas to be purely consistent (white), white balance is often performed before demosaicing. Examples of signal processing based on this principle include Japanese Patent No. 6435560, Japanese Patent Laid-Open No. 2009-147762, and "DLMMSE algorithm from L. Zhang and X. Wu, 'Color demosaicking via directional linear minimum mean square-error estimation,' IEEE Transactions on Image Processing 14 (12), 2167-2178, 2005-12." Of these, the third document, like Japanese Patent No. 6435560 and Japanese Patent Laid-Open No. 2009-147762, is a basic algorithm that uses color differences.

[0063] On the other hand, in the case of RCB space, a condition must exist for R = C = B to be true. Typically, in RGB space, white point coordinates are defined for each light source, as in standards such as CIE 1931. In other words, the purpose of white balance in signal processing is to determine these coordinates while estimating the light source reflected by the subject. However, these coordinates do not exist in color spaces other than RGB, such as RCB space. In other words, some kind of conversion is required. However, no such clear definition exists in any standard. Furthermore, even if R = C = B is true, due to the difference in the white points of the color reproduction described above, the corresponding subject will not appear achromatic (white) even if this gain is applied and the signal values ​​of each channel are made uniform. Therefore, it is not effective in suppressing false colors in achromatic areas in the subsequent stage.

[0064] To summarize the above, it is clear that it is fundamentally impossible for image sensor input signals (e.g., R, C, B) generated in color spaces other than RGB to be compatible with signal processing optimized for conventional RGB space in terms of image quality, such as color reproduction, noise, and false colors. Furthermore, it is not possible to find the white point without some kind of conversion. Recently, as market demands for high sensitivity for automotive image sensors have increased, various RAW images generated in color spaces other than RGB are increasingly being developed using signal processing for conventional RGB sensors.

[0065] The present invention provides an image processing device that can appropriately process RAW signals generated in color spaces other than RGB, eliminate the trade-off between color reproduction and noise, and generate a correct white point, while providing compatibility with RGB sensors in a conventional signal processing pipeline for RGB sensors.

[0066] FIG. 2 shows the signal processing pipeline architecture of the first embodiment of the present invention. The blocks enclosed by dotted lines are different from those of the conventional technology. First, the amplifier 242, which is located before the demosaicing process, is located after the demosaicing process. Furthermore, the proposed method does not involve color-difference-based false color suppression, which imposes constraints on the input signal of the demosaicing process (such as pre-aligning the signal levels of each pixel channel in the achromatic area), as in the conventional method. For example, as described in "Eric Dubois, 'Frequency-domain methods for demosaicking of Bayer-sampled color images,' IEEE Signal Processing Letters 12 (12), 847-850, 2005-12," it has been shown that separating each color channel (e.g., R, G, B) into three frequency-domain components (here, a Fourier transform of the luminance component and two chrominance components) and then reconstructing the image using appropriate intermediate filter processing can achieve processing with good edge preservation, minimal false color, and a consistent spatial phase. This demosaicing process can be applied to any color filter as long as it has a Bayer pattern.

[0067] As such, there is a demosaic process that can be applied to any color filter. Therefore, in the present invention, the above-mentioned problem can be solved by first performing a basis transformation in some way on any color channel (three components) with a consistent spatial phase into RGB space, and then re-applying white balance and color reproduction matrix in RGB space.

[0068] First, consider an arbitrary color space other than RGB that shares a common origin, with C0, C1, and C2 as color filters, as indicated by the dotted arrows in Figure 14. If these three vectors are independent of each other, then a transformation T exists that can convert them into RGB space. In other words, if transformation T is bijective (for example, a 3x3 matrix with an inverse matrix), then a one-to-one correspondence is possible between the C0C1C2 space and the RGB space. If such a coordinate transformation T can be generated, any color filter can be easily converted into RGB space.

[0069] In FIG. 2 , immediately after demosaicing, the color basis transformation unit 230 applies the transformation T shown in FIG. 14 (Color basis transforming in the figure) to convert pixel signals into RGB space. One way to create this transformation T is to use the method shown in FIG. 7 . That is, consider the transformation of the spectral characteristics of different color filters for each wavelength. First, each wavelength is discretized (e.g., sampled in 5-nm increments), and the channel components (C0, C1, C2) and (R, G, B) at each wavelength are considered as coordinate values ​​in the respective color spaces. Assume that (R, C, B) at b in FIG. 7 is transformed to (R, G, B) at a in the same figure. Here, consider the transformation T[...], a three-dimensional to three-dimensional linear transformation. That is, consider the following equation so that the root mean square of the distance from the value (C0, C1, C2) at wavelength λ of each spectral characteristic to (R, G, B) at the same wavelength λ is minimized, for example, in the visible light wavelength range, after passing through transformation T:

[0070] Here, (C0, C1, C2) λ are the coordinates at wavelength λ of the spectral characteristics of an arbitrary color filter, (R, G, B) λ indicates the coordinates at wavelength λ of the spectral characteristics of the RGB sensor. d[...] indicates the distance between two points. The possible range of λ is the visible light region (approximately 380 nm - 700 nm, in 5 nm increments). Now, we define a matrix operation using the transformation T by linearly combining (V1, V2, V3) with arbitrary real numbers a, b, c, d, e, f, g, h, and i, such that T[(V1, V2, V3)] = (aV1 + bV2 + cV3, dV1 + eV2 + fV3, gV1 + hV2 + iV3). Therefore, the transformation T can be rewritten as a 3x3 representation matrix, and the above becomes equivalent to the following:

[0071] When data for each wavelength (C0, C1, C2) and (R, G, B) are collected and defined as 3xN vectors, and a 3x3 matrix T exists between them that satisfies the relationship shown in the following equation, finding this transformation matrix is ​​the same as finding the least squares distance shown in the following equation.

[0072] This can generally be solved using a pseudoinverse matrix. That is, if the 3xN vector on the left side of Equation 4 is P and the 3xN vector on the right side is Q, then by multiplying P = TQ and multiplying both sides by the transpose of Q, the pseudoinverse matrix of QQ^t can be found to determine T. This method is equivalent to the least squares method described above and is expressed by the following equation.

[0073] This method is the same as the method described in Japanese Patent No. 6530751 for separating the IR component from a 4x4 matrix. In this way, we have demonstrated a method for converting the bases of different color spaces. However, while Japanese Patent No. 6530751 primarily utilizes the spectral characteristics of the image sensor as shown in Figure 7, it is known that the spectral distribution of the light source also affects the conversion T. Generally, the modulation of sensor sensitivity due to an arbitrary light source spectrum can be expressed by the following equation:

[0074] That is, it is necessary to determine the conversion T from the spectral characteristics of the image sensor taking into account the influence of the light source spectrum E(λ), that is, taking into account the light source spectrum.

[0075] FIG. 15 shows the conversion T from RCB space to RGB space calculated for each light source using actual spectral characteristics. "Original" in the figure indicates no light source, "D65" indicates D65 light source, and "A" indicates A light source. The spectral characteristics of each image sensor were calculated using FIG. 7. For "Original" (no light source) in FIG. 15, the conversion T was calculated from the characteristics shown in FIG. 7 itself. For the others, the spectrum of each light source was multiplied by the spectral spectrum of the image sensor to determine the conversion coefficient as a new spectral characteristic using the same method as described above.

[0076] Here, |T| in Figure 15 is the value of the determinant of conversion T. As shown, there is not much difference in the spectral shapes between the D65 illuminant and no illuminant, so the conversion can be said to be roughly the same, but it can be seen that the conversion coefficients are different for the A illuminant. It is generally known that when the correlated color temperature differs significantly, the conversion coefficients also differ. So how do we estimate correlated color temperature? Since correlated color temperature was originally defined in XYZ space (or RGB space), this leads to a chicken-and-egg controversy.

[0077] For example, it is conceivable to provide the color basis conversion unit 230 in Figure 2 with multiple matrix calculators to generate a signal using a conversion T that takes into account multiple light source spectra, but this would unnecessarily increase costs and is not considered a good solution. Therefore, a single representative conversion T is placed in the color basis conversion unit 230. This may be a commonly used D light source (such as D65) or an original conversion obtained from the spectral characteristics of an image sensor that does not take the light source into account. In this way, we consider how to estimate the correlated color temperature for any color filter and any light source.

[0078] Now, there is a transformation T from any color filter (C0, C1, C2) to the RGB space of the (R, G, B) sensor. If the subject has a known light source (all white), then AWB (auto white balance) detection estimated in normal RGB space can be applied as is to the signal that has passed through the color basis transformation unit 230 in Figure 2. In other words, under a known light source, even if the transformation T is fixed, the correlated color temperature and white balance gain can be estimated by calibration.

[0079] First, consider the detection values ​​(photometric values) detected for each window used when estimating a normal white balance gain in the photometry and AWB unit 241 in Fig. 2, as shown in Fig. 16. The detection values ​​used here are R, G, and B values ​​that have already been converted by the color basis conversion unit 230, but here they are distinguished from the fact that they have been converted into an R'G'B' space that actually contains errors, taking into account light source dependency.

[0080] The detection value data for each window region obtained in this manner is plotted on a plane such as B' / G'-R' / G', as shown in FIG. 17 . This figure shows an example in which detection values ​​obtained for each window under light source A of an image sensor with an arbitrary color filter are plotted as black circles. Finally, the center of gravity of the plotted values ​​is determined as the position of light source A (white circle in the figure). The photometry / AWB unit 241 repeats this series of operations for each other known light source. Once this calibration has been performed, even if the transformation T is originally light source-dependent, by reconstructing the blackbody locus (obtained by fitting, etc.) based on information obtained from calibration with the actual light source, the correlated color temperature can be estimated as a unique transformation independent of the light source.

[0081] 18 is a flowchart showing an example of a calibration method according to the first embodiment of the present technology. This calibration is started, for example, before the start of imaging. The photometry and AWB unit 241 acquires the detection values ​​of each window (step S901) and converts these detection values ​​to R'G'B' (step S902). The photometry and AWB unit 241 then plots the converted values ​​on the B' / G'-R' / G' plane (step S903). The photometry and AWB unit 241 determines the center of gravity of the plotted values ​​as the coordinates of the set light source (step S904). After step S904, the photometry and AWB unit 241 ends the calibration.

[0082] By performing calibration once for each light source according to the flow chart, it becomes possible to estimate the correlated color temperature of an image sensor equipped with the corresponding color filter using an AWB (auto white balance) algorithm operating in a standard RGB space. Once the B' / G'-R' / G' plane is completed in this manner, the photometry and AWB unit 241 can estimate the correlated color temperature and white balance gain value from each window detection value obtained from an arbitrary subject (e.g., by determining the barycentric coordinates of each detection value as in the calibration described above), as in a standard AWB. This estimation is the same as that obtained in a commonly known RGB space. For example, an estimation method is described in International Publication No. WO 2010 / 095873, although the claimed purpose is different.

[0083] Therefore, the color base conversion unit 230 and photometry / AWB unit 241 in Figure 2 can estimate the correlated color temperature and white balance gain from any color space by using any conversion T. However, the output signal value (main signal) of the color base conversion unit 230, which is purposely distinguished from the R'G'B' signal, will have errors due to light source dependency if left as is. Therefore, this error is absorbed by preparing correction coefficients for each correlated color temperature in the subsequent color reproduction unit 250. In other words, the correlated color temperature estimated by the photometry / AWB unit 241 in Figure 2 can be replaced with a previously prepared CCM as a correction. The CCM correction coefficients can be generated by calibration.

[0084] FIG. 19 is a diagram illustrating an example of a method for calculating correction matrix coefficients according to the first embodiment of the present technology. An external circuit of the image sensor 200 prepares an appropriate color chart and acquires color patch values ​​under known light sources using the image sensor 200 with an arbitrary color filter and a target RGB sensor. The external circuit then applies a transformation T to each patch value and calculates the least-squares distance for each color patch between the target RGB sensor value. This method is similar to the method for calculating the transformation T described above. In this manner, the external circuit can calculate CCM (Color Correction Matrix) coefficients for each correlated color temperature. The external circuit stores the calculated CCM coefficients in a register or the like within the image sensor 200. The CCM coefficients are read out by the color reproduction unit 250.

[0085] As described above, by combining the main signal with the correlated color temperature, white balance gain, and light source dependency corrected, and performing the processing shown in Figure 2, an image sensor with any color filter can achieve the same quality of color reproduction and white point as the target RGB sensor. Note that the CCM in Figure 2 is a simple 3x3 matrix and does not impose the constraint mentioned above (the sum of the row components must be 1.0). Instead, the white clip shown in Figure 10 is introduced in the latter stage. Also, the target RGB values ​​do not have to be the values ​​of the actual sensor. It is common for users to target any RGB value they desire.

[0086] Figure 20 shows an example of the composite coefficients and white balance gain values ​​of the conversion T x CCM for an RCCB sensor with the characteristics shown in Figure 7 under a D65 light source. Clearly, the method of the present invention can achieve conversion with smaller values ​​than the CCM coefficients and white balance gain values ​​obtained by the conventional signal processing method shown in Figure 13. Furthermore, since the value of the determinant indicating the degree of conversion is also small, spatial distortion before and after conversion can be suppressed. Thus, it can be seen that the present invention can achieve color reproduction similar to that of the target RGB sensor with high accuracy, low distortion, and reduced noise by using conversion T and then correcting the error of conversion T through various calibrations. This further improves image quality.

[0087] Furthermore, in Fig. 2, if the basis transformation matrix of the color basis transformation unit 230 is replaced with a unit matrix, it is easy to see that compatibility as a normal RGB sensor is maintained. For example, when the processing of Fig. 2 is performed outside the image sensor 200, the color basis transformation unit 230 can store a unit matrix and a basis transformation matrix and switch the matrix to be applied depending on the type of sensor. The type of sensor is specified by an external setting signal.

[0088] In the dotted line portion of the figure, which is the basis of the present invention, three blocks of processing called the main line system: color base conversion unit 230 → amplification unit 242 → color reproduction unit 250 can be formulated and expressed as follows:

[0089] In the above equation, from the left, there is CCM, white balance gain, the basis transformation matrix in the color basis transformation unit 230, and input signal values ​​(C0, C1, C2). Here, α, 1.0, and β represent the white balance gains of each channel. The other coefficients are the same as those shown above.

[0090] As described above, according to the first embodiment of the present technology, the color base conversion unit 230 converts the color space from the C0C1C2 space to the RGB space, and after the conversion, the amplification unit 242 amplifies the image using a white balance gain, thereby achieving both improved color reproducibility and reduced noise. This makes it possible to improve image quality.

[0091] 2. Second Embodiment In the first embodiment described above, the color base conversion unit 230 converts the color space of the demosaic image, but the configuration is not limited to this as long as it can realize the processing of Equation 7. The image sensor 200 in this second embodiment differs from the first embodiment in that the color base conversion unit detects a RAW image and converts the color space of the detected value.

[0092] 21 is a block diagram showing an example configuration of an image sensor 200 according to a second embodiment of the present technology. The image sensor 200 according to the second embodiment does not include a white clip processing unit 260. Furthermore, instead of the color base conversion unit 230 and the photometry and AWB unit 241, a color base conversion unit 231 and an AWB unit 243 are provided, and instead of the color reproduction unit 250, a color reproduction unit 251 is provided.

[0093] The color basis conversion unit 231 receives a raw image from the pixel array unit 210. The color basis conversion unit 231 detects the raw image and performs color basis conversion processing on the detected value. The color basis conversion unit 231 then supplies the converted detected value to the AWB unit 243.

[0094] The AWB unit 243 estimates a white balance gain and a correlated color temperature based on the detection value from the color basis conversion unit 231, and supplies the results to the color reproduction unit 251. A gain of 1.0 is set in the amplification unit 242, and the RAW image that has essentially bypassed the amplification unit 242 is input to the demosaic processing unit 220.

[0095] The color reproduction unit 251 amplifies the color signals in the demosaic image by a white balance gain and corrects them by CCM.

[0096] As shown in the figure, equivalent processing is possible even with the conventional architecture shown in Figure 3. However, some software algorithm changes are required. As mentioned above, bypassing is achieved by always providing 1.0 to all channels of the amplifier unit 242. Next, in the photometry / AWB 241 block that performs photometry, the following algorithm change is made in software to realize the function of the color basis converter 231. Detection processing such as that shown in Figure 16 is often implemented in hardware. This processing itself remains unchanged, and a transformation T is applied to each detection value (sum(C0), sum(C1), sum(C2)) obtained for an image sensor with any color filter. Plotting these values ​​on the B / G-R / G plane yields results similar to those discussed above.

[0097] In terms of cost, the matrix calculation shown in FIG. 22 is performed for the number of white balance detection frames (number of windows). The coefficients of the conversion T used here are fixed values, as in the previous discussion, and by performing similar calibrations according to FIGS. 17 and 18, the same results as in the previous discussion can be obtained. The only difference from the first embodiment is whether the pre-detection signal values ​​are converted to RGB space in advance using hardware, or whether they are converted after detection by the color base conversion unit 231. Therefore, the diagonal matrices α and β, which are the products of the three matrices formulated above, are generated by AWB along with the correlated color temperature. These are then multiplied by the conversion T, and the correction CCM coefficients corresponding to the correlated color temperature are also calculated, ultimately resulting in a single 3 x 3 matrix.

[0098] As described above, according to the second embodiment of the present technology, the color base conversion unit 231 detects a RAW image and converts the color space of the detected value, so that it is possible to realize the same function as that of the first embodiment in software using conventional hardware such as that shown in FIG. 3 .

[0099] The present invention provides a method for simultaneously generating correlated color temperatures similar to those of RGB sensors and correcting the main line system by devising appropriate calibration, without causing degradation of color reproduction or amplification of noise when using any color filter required for signal processing used in various applications, because the conversion required to achieve this is light source dependent. Furthermore, because the essence of the present invention lies in its principles, it is not limited to the architecture itself, and has the flexibility to be realized by simply executing a software workaround on conventional systems.

[0100] 3. Application Examples to Mobile Bodies The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of mobile body, such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, personal mobility, an airplane, a drone, a ship, or a robot.

[0101] Fig. 23 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile object control system to which the technology according to the present disclosure can be applied. The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in Fig. 23, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside-vehicle information detection unit 12030, an inside-vehicle information detection unit 12040, and an integrated control unit 12050. The functional configuration of the integrated control unit 12050 also includes a microcomputer 12051, an audio / video output unit 12052, and an in-vehicle network I / F (interface) 12053.

[0102] The drivetrain control unit 12010 controls the operation of devices related to the drivetrain of the vehicle in accordance with various programs. For example, the drivetrain control unit 12010 functions as a control device for a drive force generating device for generating a drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force of the vehicle.

[0103] The body system control unit 12020 controls the operation of various devices equipped in the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as headlamps, backup lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves transmitted from a portable device that serves as a key or signals from various switches can be input to the body system control unit 12020. The body system control unit 12020 receives these radio waves or signals and controls the vehicle's door lock device, power window device, lamps, etc.

[0104] The outside-vehicle information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the outside-vehicle information detection unit 12030. The outside-vehicle information detection unit 12030 causes the imaging unit 12031 to capture images outside the vehicle and receives the captured images. The outside-vehicle information detection unit 12030 may perform object detection processing or distance detection processing for people, cars, obstacles, signs, characters on the road surface, etc. based on the received images.

[0105] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. The imaging unit 12031 can output the electrical signal as an image or as distance measurement information. The light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

[0106] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041 that detects the state of the driver is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 may calculate the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing off.

[0107] The microcomputer 12051 can calculate control target values ​​for the driving force generating device, steering mechanism, or braking device based on the information inside and outside the vehicle acquired by the outside-vehicle information detection unit 12030 or the inside-vehicle information detection unit 12040, and output control commands to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the distance between vehicles, maintaining vehicle speed, vehicle collision warning, vehicle lane departure warning, etc.

[0108] In addition, the microcomputer 12051 can perform cooperative control for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on driver operation, by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle obtained by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040.

[0109] Furthermore, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the outside information detection unit 12030. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside information detection unit 12030, and perform cooperative control aimed at preventing glare, such as switching from high beams to low beams.

[0110] The audio / video output unit 12052 transmits at least one of audio and video output signals to an output device capable of visually or audibly notifying information to vehicle occupants or the outside of the vehicle. In the example of Fig. 23, the output devices are exemplified by an audio speaker 12061, a display unit 12062, and an instrument panel 12063. The display unit 12062 may include, for example, at least one of an on-board display and a head-up display.

[0111] FIG. 24 is a diagram showing an example of the installation position of the imaging unit 12031.

[0112] In FIG. 24, the imaging unit 12031 includes imaging units 12101, 12102, 12103, 12104, and 12105.

[0113] The imaging units 12101, 12102, 12103, 12104, and 12105 are provided, for example, at positions such as the front nose, side mirrors, rear bumper, back door, and the top of the windshield inside the vehicle cabin of the vehicle 12100. The imaging unit 12101 provided on the front nose and the imaging unit 12105 provided on the top of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 provided on the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 provided on the rear bumper or back door mainly acquires images of the rear of the vehicle 12100. The imaging unit 12105 provided on the top of the windshield inside the vehicle cabin is mainly used to detect preceding vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc.

[0114] 24 shows an example of the imaging ranges of the imaging units 12101 to 12104. Imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or back door. For example, by overlaying the image data captured by the imaging units 12101 to 12104, an overhead image of the vehicle 12100 viewed from above can be obtained.

[0115] At least one of the image capturing units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the image capturing units 12101 to 12104 may be a stereo camera made up of multiple image capturing elements, or may be an image capturing element having pixels for phase difference detection.

[0116] For example, based on the distance information obtained from the imaging units 12101 to 12104, the microcomputer 12051 can calculate the distance to each three-dimensional object within the imaging ranges 12111 to 12114 and the change in this distance over time (relative speed with respect to the vehicle 12100), thereby extracting as a preceding vehicle, in particular, the three-dimensional object that is the closest three-dimensional object on the path of the vehicle 12100 and traveling in approximately the same direction as the vehicle 12100 at a predetermined speed (e.g., 0 km / h or higher). Furthermore, the microcomputer 12051 can set a vehicle-to-vehicle distance to be maintained in advance in front of the preceding vehicle, and perform automatic braking control (including follow-up stop control), automatic acceleration control (including follow-up start control), etc. In this way, cooperative control can be performed for the purpose of autonomous driving, which runs autonomously without relying on driver operation.

[0117] For example, the microcomputer 12051 classifies and extracts three-dimensional object data regarding three-dimensional objects into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on distance information obtained from the imaging units 12101 to 12104, and can use the data for automatic obstacle avoidance. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. The microcomputer 12051 then determines a collision risk that indicates the risk of collision with each obstacle, and when the collision risk is equal to or greater than a set value and a collision is possible, the microcomputer 12051 can provide driving assistance for collision avoidance by outputting an alarm to the driver via the audio speaker 12061 or the display unit 12062, or by performing forced deceleration or avoidance steering via the drive system control unit 12010.

[0118] At least one of the image capturing units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether a pedestrian is present in the images captured by the image capturing units 12101 to 12104. Such pedestrian recognition is performed, for example, by extracting feature points from the images captured by the image capturing units 12101 to 12104 as infrared cameras and performing pattern matching on a series of feature points that indicate the outline of an object to determine whether the object is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the images captured by the image capturing units 12101 to 12104 and recognizes the pedestrian, the audio / image output unit 12052 controls the display unit 12062 to superimpose a rectangular outline on the recognized pedestrian for emphasis. The audio / image output unit 12052 may also control the display unit 12062 to display an icon or the like indicating the pedestrian at a desired position.

[0119] An example of a vehicle control system to which the technology according to the present disclosure can be applied has been described above. The technology according to the present disclosure can be applied to the imaging unit 12031 of the above-described configuration. Specifically, the image sensor 200 in FIG. 2 can be applied to the imaging unit 12031. By applying the technology according to the present disclosure to the imaging unit 12031, it is possible to obtain a captured image that is easier to see, thereby reducing driver fatigue.

[0120] Note that the above-described embodiment shows an example for realizing the present technology, and the matters in the embodiment and the matters specifying the invention in the claims correspond to each other. Similarly, the matters specifying the invention in the claims and the matters in the embodiment of the present technology having the same name correspond to each other. However, the present technology is not limited to the embodiment, and can be realized by applying various modifications to the embodiment within the scope of the gist thereof.

[0121] The processing procedures described in the above embodiments may be considered as a method having a series of these procedures, or as a program for causing a computer to execute the series of procedures, or as a recording medium for storing the program. Examples of recording media that can be used include CDs (Compact Discs), MDs (MiniDiscs), DVDs (Digital Versatile Discs), memory cards, and Blu-ray (registered trademark) Discs. The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0122] The present technology may also be configured as follows: (1) An image processing device comprising: a color basis conversion unit that performs color basis conversion processing on an image to convert a color space from a first color space to a second color space; and an image quality adjustment processing unit that performs predetermined image quality adjustment processing on the image after the color basis conversion processing. (2) The image processing device according to (1), further comprising: a demosaic processing unit that converts a RAW image into a demosaic image, wherein the image is the RAW image or the demosaic image. (3) The image processing device according to (2), further comprising: a photometry / AWB unit that detects the demosaic image after the color basis conversion processing and estimates a white balance gain; and an amplifier unit that amplifies color signals in the demosaic image after the color basis conversion processing by the white balance gain and supplies the amplified color signals to the image quality adjustment processing unit, wherein the color basis conversion unit performs the color basis conversion processing on the demosaic image, and the image quality adjustment processing unit comprises: a color reproduction unit that corrects the amplified color signals by a color reproduction matrix, and a white clip processing unit that limits the corrected color signals by a predetermined threshold. (4) The image processing device according to (2), further comprising: an AWB unit that estimates a white balance gain, wherein the color basis conversion unit performs the color basis conversion processing on detected values ​​of the RAW image and supplies the detected values ​​to the AWB unit, and the image quality adjustment processing unit comprises: a color reproduction unit that amplifies the color signals in the demosaic image by the white balance gain and corrects them by a color reproduction matrix. (5) The image processing device according to any one of (1) to (4), wherein the first color space is a color space different from an RGB space, and the second color space is an RGB space. (6) The image processing device according to any one of (1) to (5), further comprising a pixel array unit that generates the image. (7) The image processing device according to (6), wherein the pixel array unit has an array of a plurality of pixels including clear pixels, and the clear pixels are provided with transparent filters. (8) The image processing device according to any one of (1) to (7), wherein the image quality adjustment processing unit performs at least one of gamma correction processing, edge enhancement processing, YUV conversion processing, and color conversion processing.(9) An image processing method comprising: a color basis conversion procedure for performing color basis conversion processing on an image to convert a color space from a first color space to a second color space; and an image quality adjustment procedure for performing a predetermined image quality adjustment processing on the image after the color basis conversion processing. (10) A program for causing a computer to execute: a color basis conversion procedure for performing color basis conversion processing on an image to convert a color space from a first color space to a second color space; and an image quality adjustment procedure for performing a predetermined image quality adjustment processing on the image after the color basis conversion processing.

[0123] 100 Imaging device 110 Optical unit 120 DSP circuit 130 Display unit 140 Operation unit 150 Bus 160 Frame memory 170 Storage unit 180 Power supply unit 200 Image sensor 210 Pixel array unit 211 Pixel 220 Demosaic processing unit 230, 231 Color base conversion unit 241 Photometry / AWB unit 242 Amplification unit 243 AWB unit 250, 251 Color reproduction unit 260 White clip processing unit 271 Gamma correction unit 272 Edge enhancement unit 273 YUV conversion unit 274 Color conversion unit 275 DNN monitor

Claims

1. An image processing device comprising: a color base conversion unit that performs color base conversion processing on an image to convert the color space from a first color space to a second color space; and an image quality adjustment processing unit that performs a predetermined image quality adjustment processing on the image after the color base conversion processing.

2. The image processing device according to claim 1, further comprising a demosaic processing unit that converts a RAW image into a demosaic image, and the image is the RAW image or the demosaic image.

3. An image processing device as described in claim 2, further comprising: a photometry / AWB (Auto White Balance) unit that detects the demosaic image after the color base conversion processing and estimates a white balance gain; and an amplifier unit that amplifies color signals in the demosaic image after the color base conversion processing by the white balance gain and supplies the amplified color signals to the image quality adjustment processing unit, wherein the color base conversion unit performs the color base conversion processing on the demosaic image, and the image quality adjustment processing unit comprises: a color reproduction unit that corrects the amplified color signals by a color reproduction matrix, and a white clip processing unit that limits the corrected color signals by a predetermined threshold value.

4. An image processing device as described in claim 2, further comprising an AWB unit that estimates a white balance gain, wherein the color base conversion unit performs the color base conversion processing on the detection value of the RAW image and supplies it to the AWB unit, and the image quality adjustment processing unit includes a color reproduction unit that amplifies the color signals in the demosaic image using the white balance gain and corrects them using a color reproduction matrix.

5. The image processing device according to claim 1, wherein the first color space is a color space different from the RGB space, and the second color space is the RGB space.

6. The image processing device according to claim 1, further comprising a pixel array section for generating the image.

7. The image processing device according to claim 6, wherein the pixel array section has an array of a plurality of pixels including clear pixels, and the clear pixels are provided with transparent filters.

8. The image processing device according to claim 1, wherein the image quality adjustment processing section executes at least one of gamma correction processing, edge enhancement processing, YUV conversion processing, and color conversion processing.

9. An image processing method comprising: a color base conversion procedure for performing a color base conversion process on an image to convert the color space from a first color space to a second color space; and an image quality adjustment process for performing a predetermined image quality adjustment process on the image after the color base conversion process.

10. A program for causing a computer to execute a color base conversion procedure for performing a color base conversion process on an image to convert the color space from a first color space to a second color space, and an image quality adjustment process for performing a predetermined image quality adjustment process on the image after the color base conversion process.

Images