Image processing device, image processing method, and program

Abstract

To improve the image quality in an image processing device that processes an image.SOLUTION: An image processing device includes a color base conversion unit and an image quality adjustment processing unit. In this image processing device, the color base conversion unit of the color base conversion unit and the image quality adjustment processing unit performs color base conversion processing on an image to convert a color space from a first color space to a second color space. Furthermore, the image quality adjustment processing of the color base conversion unit and the image quality adjustment processing unit performs predetermined image quality adjustment processing on the image after the color base conversion processing by the color base conversion.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present technology relates to an image processing device, and more particularly to an image processing device that processes images in various color spaces, an image processing method in the image processing device, and a program for causing a computer to execute the method. [Background technology]

[0002] Conventionally, various types of image processing such as white balance correction have been performed on images to improve image quality. For example, an image processing device has been proposed that performs processing to correct sensitivity differences between pixels before converting a RAW image with a Bayer array into a demosaic image (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-154307 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned conventional technology, the quality of the image of the Bayer array is improved by correcting the sensitivity difference. However, in the above-mentioned image processing device, when performing processing such as white balance correction on an image whose color space is not the RGB space, it becomes difficult to improve the image quality. For example, if the color space is not the RGB space, the gain during the white balance correction must be increased in order to maintain color reproducibility, but if the gain is large, noise increases. On the other hand, if the gain is reduced, noise is reduced, but color reproducibility is reduced. Thus, in the above-mentioned conventional technology, problems such as a trade-off between improved color reproducibility and reduced noise arise, and as a result, it becomes difficult to improve the image quality.

[0005] The present technology has been developed in view of these circumstances, and aims to improve image quality in an image processing device that processes images. [Means for solving the problem]

[0006] The present technology has been made to solve the above-mentioned problems, and a first aspect thereof is an image processing device including a color base conversion unit that performs a color base conversion process on an image to convert a color space from a first color space to a second color space, and an image quality adjustment processing unit that performs a predetermined image quality adjustment process on the image after the color base conversion process, an image processing method in the image processing device, and a program for causing a computer to execute the method, thereby improving image quality.

[0007] In addition, in the first aspect, the image processing device may further include a demosaic processing unit that converts a RAW image into a demosaic image, and the image may be the RAW image or the demosaic image. This brings about an effect that the image quality of the demosaic image is improved.

[0008] In addition, in the first aspect, the present invention may further include a photometry / AWB (Auto White Balance) unit that detects the demosaic image after the color base conversion process and estimates a white balance gain, and an amplifier that amplifies a color signal in the demosaic image after the color base conversion process by the white balance gain and supplies the amplified color signal to the image quality adjustment processor, the color base conversion unit performs the color base conversion process on the demosaic image, and the image quality adjustment processor may include a color reproduction unit that corrects the amplified color signal by a color reproduction matrix, and a white clip processor that limits the corrected color signal by a predetermined threshold. This provides the effect of improving color reproducibility and reducing noise.

[0009] In the first aspect, the image processing apparatus may further include an AWB unit that estimates a white balance gain, the color base conversion unit performs the color base conversion process on the detection value of the RAW image and supplies the result to the AWB unit, and the image quality adjustment processing unit may include a color reproduction unit that amplifies color signals in the demosaic image by the white balance gain and corrects them by a color reproduction matrix. This improves color reproducibility and reduces noise.

[0010] In the first aspect, the first color space may be a color space different from an RGB space, and the second color space may be an RGB space, thereby providing an effect of converting from a space other than the RGB space to the RGB space.

[0011] In addition, in the first aspect, the image sensor may further include a pixel array unit that generates the image, thereby providing an effect that processing is performed within the image sensor.

[0012] In the first aspect, the pixel array unit may include a plurality of pixels including clear pixels, and the clear pixels may be provided with transparent filters, thereby providing an effect of converting a color space of an image including the clear pixels.

[0013] In addition, in the first aspect, the image quality adjustment processing unit may execute at least one of a gamma correction process, an edge enhancement process, a YUV conversion process, and a color conversion process, thereby providing an effect that the image quality is adjusted. [Brief description of the drawings]

[0014] [Figure 1] 1 is a block diagram showing an example of a configuration of an imaging device according to a first embodiment of the present technology. [Diagram 2] 1 is a block diagram showing a configuration example of an image sensor according to a first embodiment of the present technology. [Diagram 3] FIG. 1 is a block diagram showing an example of the configuration of an image sensor according to a conventional technique. [Figure 4] FIG. 1 is a diagram showing an example of processing up to color reproduction in the prior art. [Diagram 5] FIG. 1 is a diagram showing details of processing up to color reproduction in the prior art. [Figure 6] FIG. 1 is a diagram showing an example of a Bayer array in the prior art. [Figure 7] 1 is a graph showing an example of spectral characteristics in the prior art. [Figure 8] 1 is an example of a factor graph in the prior art. [Figure 9] 1 is an example of a constrained color reproduction matrix in the prior art. [Figure 10] 4 is an example of white clip processing according to the first embodiment of the present technology. [Figure 11] FIG. 1 is a diagram showing an example of an RGB space in the prior art. [Figure 12] FIG. 1 is a diagram showing an example of an RCB space in the prior art. [Figure 13] FIG. 1 is a diagram showing an example of a white balance gain and a color reproduction matrix in the prior art. [Figure 14] FIG. 2 is a diagram showing an example of a color space in the first embodiment of the present technology. [Figure 15] FIG. 2 is a diagram illustrating an example of a transformation matrix in the first embodiment of the present technology. [Figure 16] 4 is a diagram showing an example of processing in a photometry / AWB unit in the first embodiment of the present technology; [Figure 17] 3 is a diagram showing an example of a plot of detection values ​​in the first embodiment of the present technology; FIG. [Figure 18] 4 is a flowchart showing an example of a calibration method according to the first embodiment of the present technology. [Figure 19] 4 is a diagram illustrating an example of a method for calculating correction matrix coefficients in the first embodiment of the present technology. FIG. [Figure 20] 3A to 3C are diagrams illustrating an example of white balance gains and a color correction matrix according to the first embodiment of the present technology. [Figure 21] 11 is a block diagram showing a configuration example of an image sensor according to a second embodiment of the present technology. FIG. [Figure 22] FIG. 11 is a diagram illustrating an example of a color base conversion process according to a second embodiment of the present technology. [Diagram 23] 1 is a block diagram showing a schematic configuration example of a vehicle control system; [Figure 24] FIG. 4 is an explanatory diagram showing an example of an installation position of an imaging unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, modes for carrying out the present technology (hereinafter, referred to as embodiments) will be described. The description will be made in the following order. 1. First embodiment (example of performing color basis conversion processing on a demosaic image) 2. Second embodiment (example of performing color basis transformation processing on detection values ​​of RAW images) 3. Application examples for mobile objects

[0016] <1. First embodiment> [Example of imaging device configuration] 1 is a block diagram showing an example of a configuration of an imaging device 100 according to a first embodiment of the present technology. The imaging device 100 is a device for capturing image data, and includes an optical unit 110, an image sensor 200, and a DSP (Digital Signal Processing) circuit 120. The imaging device 100 further includes a display unit 130, an operation unit 140, a bus 150, a frame memory 160, a storage unit 170, and a power supply unit 180. The imaging device 100 may be a digital camera or an electronic device having an imaging function (such as a smartphone or a personal computer). The imaging device 100 is an example of an image processing device described in the claims.

[0017] The optical unit 110 collects light from a subject and guides it to the image sensor 200. The image sensor 200 generates image data by photoelectric conversion. The image sensor 200 processes the generated image data and supplies the processed data to the DSP circuit 120 via a signal line 209.

[0018] The DSP circuit 120 executes a predetermined signal processing on the data from the image sensor 200. The DSP circuit 120 outputs the processed data via a bus 150 to a frame memory 160 or the like.

[0019] The display unit 130 displays various data. For example, a liquid crystal panel or an organic EL (Electro Luminescence) panel is assumed as the display unit 130. The operation unit 140 generates an operation signal in accordance with a user's operation.

[0020] The bus 150 is a common path for the optical unit 110, image sensor 200, DSP circuit 120, display unit 130, operation unit 140, frame memory 160, storage unit 170 and power supply unit 180 to exchange data with one another.

[0021] The frame memory 160 holds image data. The storage unit 170 stores various data such as image data. The power supply unit 180 supplies power to the image sensor 200, the DSP circuit 120, the display unit 130, and the like.

[0022] [Image sensor configuration example] 2 is a block diagram showing an example of a configuration of an image sensor 200 according to the first embodiment of the present technology. The image sensor 200 includes a pixel array unit 210, a demosaic processing unit 220, a color basis conversion unit 230, a photometry / AWB unit 241, an amplifier unit 242, a color reproduction unit 250, and a white clip processing unit 260. The image sensor 200 further includes a gamma correction unit 271, an edge enhancement unit 272, a YUV conversion unit 273, a color conversion unit 274, and a DNN (Deep Neural Network) monitor 275.

[0023] Note that a part or all of the downstream processing (such as demosaic processing) of the pixel array unit 210 can be performed by a circuit external to the image sensor 200 (such as the DSP circuit 120).

[0024] In the pixel array section 210, a plurality of pixels 211 are arranged in a two-dimensional lattice. Each of the pixels 211 generates a pixel signal by photoelectric conversion. The pixel array section 210 generates a RAW image in a color space different from RGB. A RAW image is an image having one piece of color information for each pixel, and is also called a mosaic image. For example, in the pixel array section 210, a plurality of pixels including clear pixels are arranged in an array conforming to a Bayer array.

[0025] Here, the clear pixel is a pixel provided with a transparent filter having a transmittance of a predetermined value (such as 0.5) or more over the entire visible light range. The clear pixel is also called a clarity pixel. The clear pixel is defined as a C1 pixel, and pixels of different colors other than the clear pixel are defined as a C0 pixel and a C2 pixel. For example, an R (Red) pixel and a B (Blue) pixel are used as the C0 pixel and the C2 pixel. The R pixel is a pixel provided with a color filter that selectively transmits light in the red wavelength range, and the B pixel is a pixel provided with a color filter that selectively transmits light in the blue wavelength range.

[0026] In addition, R pixels and G (Green) pixels, or Y (Yellow) pixels and Cy (Cyan) pixels can also be used as C0 pixels and C2 pixels. G pixels are pixels provided with color filters that selectively transmit light in the green wavelength range, and Y pixels are pixels provided with color filters that selectively transmit light in the yellow wavelength range. Cy pixels are pixels provided with color filters that selectively transmit light in the cyan wavelength range. The pixels C0 to C2 are arranged in an array in which the position of the G pixel in the Bayer array is replaced with the C1 pixel. The color space in which these pixels are arranged is hereinafter referred to as a "C0C1C2 space." In the C0C1C2 space, the one in which the C0 pixel and the C2 pixel are R pixel and B pixel is referred to as an "RCB space."

[0027] The pixel array unit 210 supplies a RAW image in the C0C1C2 space to the demosaic processing unit 220 via various circuits such as an ADC (Analog to Digital Converter). In the figure, the circuits between the pixel array unit 210 and the demosaic processing unit 220 are omitted.

[0028] The demosaic processing unit 220 converts a RAW image into a demosaic image. The demosaic image is an image in which missing color information among three colors is complemented for each pixel. The demosaic processing unit 220 supplies the demosaic image to the color basis conversion unit 230.

[0029] The color base conversion unit 230 performs color base conversion processing on the demosaic image, converting the color space from the C0C1C2 space to an RGB space using a basis conversion matrix. Details of the basis conversion matrix will be described later. The color base conversion unit 230 supplies the demosaic image after the color base conversion processing to the photometry / AWB unit 241 and the amplifier 242. The demosaic image after the conversion includes three color signals of R, G, and B for each pixel. Note that although the color base conversion unit 230 converts the C0C1C2 space to the RGB space, the color space to be converted is not limited to the C0C1C2 space.

[0030] Note that the C0C1C2 space is an example of a first color space described in the claims, and the RGB space is an example of a second color space described in the claims.

[0031] According to the processing delay of the photometry / AWB unit 241, the output timing to the amplification unit 242 is delayed with respect to the output timing to the photometry / AWB unit 241. A buffer can also be disposed between the color base conversion unit 230 and the amplification unit 242.

[0032] The photometry / AWB unit 241 measures (in other words, detects) the demosaic image from the color base conversion unit 230 to estimate a white balance gain and a correlated color temperature. The white balance gain includes gains for each of R, G, and B. The photometry / AWB unit 241 supplies the white balance gain to the amplification unit 242 and supplies the correlated color temperature to the color reproduction unit 250.

[0033] The amplifier 242 amplifies the color signals in the demosaic image by a white balance gain. An amplifier that amplifies a corresponding color signal for each color is arranged in the amplifier 242. Since the demosaic image includes three color signals for each pixel, three amplifiers are arranged in the amplifier 242. The amplifier 242 supplies the amplified color signals to the color reproduction unit 250.

[0034] The color reproduction unit 250 corrects the color signal from the amplifier unit 242 using a color correction matrix (CCM). This CCM is also called a linear matrix. The color reproduction unit 250 holds in advance a set of correction matrix coefficients in the CCM for each of a plurality of color temperatures. The color reproduction unit 250 then performs correction using the CCM of the correction matrix coefficients corresponding to the correlated color temperature estimated by the photometry / AWB unit 241, and supplies the corrected demosaic image to the white clip processing unit 260. Details of the processing performed by the color reproduction unit 250 will be described later. Note that the circuits following the color reproduction unit 250 are an example of an image quality adjustment processing unit described in the claims.

[0035] The white clip processing unit 260 limits the corrected color signals by a predetermined threshold value. When the R, G, and B color signals are greater than the upper threshold value, the white clip processing unit 260 outputs the threshold value. When the color signals are less than the threshold value, the white clip processing unit 260 outputs the color signals as they are. The white clip processing unit 260 supplies the processed image to the gamma correction unit 271.

[0036] The gamma correction unit 271 performs gamma correction on the image from the white clip processing unit 260 and supplies the image to the edge enhancement unit 272. The edge enhancement unit 272 performs processing to enhance edges on the image from the gamma correction unit 271 and supplies the image to the YUV conversion unit 273. The YUV conversion unit 273 converts the color space of the image from the edge enhancement unit 272 from RGB to YUV and supplies the image to the color conversion unit 274. The color conversion unit 274 converts the color space of the image from the YUV conversion unit 273 to match the input data format of the DNN monitor 275. For example, YUV is converted to a color space such as Lab, RGB, or CMYK.

[0037] The DNN monitor 275 performs image recognition using a DNN on the image from the color conversion unit 274. The DNN monitor 275 outputs the recognition result and image data to the DSP circuit 120.

[0038] The processes from the gamma correction unit 271 to the DNN monitor 275 are an example of post-stage processes of the white clip processing unit 260, and these processes are executed as necessary. The image sensor 200 can execute only a part of the processes from the gamma correction unit 271 to the DNN monitor 275, or can execute processes different from these.

[0039] Among the above-mentioned components, the parts enclosed by dotted lines, i.e., the color base conversion unit 230, the photometry / AWB unit 241, the amplifier 242, the color reproduction unit 250, and the white clip processing unit 260, are different from the conventional technology. In order to explain the effects of the present invention, a conventional image sensor 200 in which the parts enclosed by dotted lines are replaced with a general configuration will be considered as a comparative example.

[0040] 3 is a block diagram showing an example of a configuration of a conventional image sensor 200. In the conventional technology, the color base converter 230 and the white clip processor 260 are not provided.

[0041] In addition, the pixel array unit 210 generates a RAW image in RGB space and supplies it to a photometry / AWB unit 241 and an amplifier unit 242. The photometry / AWB unit 241 detects the RAW image, estimates a white balance gain and a correlated color temperature, and supplies it to the amplifier unit 242 and a color reproduction unit 250.

[0042] The amplifier 242 amplifies the color signals in the RAW image by a white balance gain and supplies the amplified color signals to the demosaic processor 220. The demosaic processor 220 converts the RAW image into a demosaic image and supplies the demosaic image to the color reproduction unit 250. The color reproduction unit 250 corrects the color signals in the demosaic image by CCM and supplies the demosaic image to the gamma correction unit 271.

[0043] Generally, single-chip image sensors have pixel arrays with various patterns when viewed in two-dimensional coordinates, and are equipped with various color filters depending on the application. There is a common camera signal processing for pixels with these various pixel arrays and color filters. It is a process in which the spatial phase difference of each color is matched by demosaic processing, and then color reproduction processing is performed at each pixel position, and this process converts the signal into the final video signal or the signal required by the digital still camera. To give a specific example, in recent years, in autonomous driving technology for automobiles using AI (Artificial Intelligence), three-color RGB or YUV signals with spatial phases aligned at each pixel position are widely and commonly used as input vectors for signal processing with recognition algorithms such as CNN (Convolutional Neural Network) / RNN (Recurrent Neural Networks) or Attention-based Transformer. Most of these input signals are generated by inheriting the technology of digital still cameras and camcorders that were mainly used for ornamental purposes.

[0044] The figure shows a schematic diagram of these common and essential signal processing. First, pixel data is input in a time-series manner from a pixel array having a Bayer pattern, passes through a photometry / AWB section 241 and an amplifier section 242, and is subjected to demosaic processing to align the spatial phase and number of pixels of the pixel positions of each RGB channel. Then, processing to adjust the color gamut such as a color reproduction matrix is ​​performed, and the signal is sent to subsequent processing (gamma correction, edge emphasis, YUV conversion, etc.) to become the final output signal. Note that this configuration example is only a part of the example, and various pixel corrections (defect correction, black level adjustment, various shading corrections, etc.) before white balance are omitted because they are not directly related to the present invention.

[0045] For example, signal processing described in documents such as JP 2015-154307 A and WO 2011 / 152174 corresponds to this. Among them, demosaic processing is realized by a wide variety of algorithms, and there are many representative Bayer patterns. In addition, there are many arrays other than the Bayer pattern, including the arrays described in WO 2016 / 098641 A and WO 2013 / 145487 A. The common purpose of these demosaic processes is to generate basic signals for adjusting various spatial frequency components such as color reproduction, luminance reproduction, and edge processing in the latter stage under conditions in which the spatial phase of the pixel position is aligned in the latter signal processing.

[0046] FIG. 4 shows a part of the process in FIG. 3, focusing on the CCM used in the amplifier 242 and the color reproduction unit 250 before and after demosaic processing used in a general RGB-Bayer type image sensor.

[0047] Furthermore, Fig. 5 shows a specific example of the configuration of Fig. 4 using specific basic function processing (low-pass filter, minimum value extraction processing, average value calculation, NR, multiplier, etc.). In Fig. 5, "LPF" indicates a low-pass filter, "min" indicates minimum value extraction processing, "mean" indicates average value calculation processing, and "x" indicates a multiplier. "+" indicates an adder, and Gm indicates the statistics of Gr and Gb (average value, etc.).

[0048] Now, with the recent development of autonomous driving technology, sensitivity in low light conditions has become an issue (for example, to detect areas outside the range of headlight illumination), and in addition to the conventional Bayer-type RGB image sensors that mainly use G pixels, many image sensors that use clear pixels to increase the sensitivity, as shown in Figure 6, are now being adopted.

[0049] However, as shown in Figure 7, the spectral characteristics of image sensors generally differ significantly, so even if sensitivity is improved, problems arise in color reproduction compared to RGB sensors. In the figure, a shows the spectral characteristics of the R, Gr, Gb, and B pixels. In the figure, b shows the spectral characteristics of the R, Cr, Cg, and B pixels. The Cr and Cg pixels are clear pixels. In the figure, the vertical axis of a and b shows relative sensitivity, and the horizontal axis shows wavelength.

[0050] This color reproduction issue is expressed using a simpler factor graph in Figure 8. In the factor graph in Figure 8, the circles indicate the pixel values ​​of each color, and the squares represent various computing units (arithmetic units, conditional branches, comparators, etc., which are abstractions of the computing units in Figure 5 using a single symbol). Each square here represents the same processing (however coefficients, etc., can be different) arranged horizontally, and squares arranged vertically represent different computing units. In other words, the focus is on the input / output relationships in Figures 4 and 5. Each function is briefly shown as an annotation on the left side of the figure. Now, let us consider the white point of the light source, which can be said to be the origin of color reproduction.

[0051] Generally, white points expressed in CIE1931 color spaces and the like are expressed as R=G=B in RGB space. For a typical D65 light source, the white point, which has coordinates of (0.3127, 0.3290, 0.3583) in the XYZ space, is (0.3290, 0.3290, 0.3290) in the RGB space. As such, the coordinate position of white differs depending on the light source, but because widely used standard white points such as the D light source are important, the following discussion will focus on color reproduction under the D light source. However, the white points of other light sources (light sources with different color temperatures) are still fixed points, just with different coordinates, so it can be understood that they are the same as the D light source.

[0052] From FIG. 8, even if the colors of a white subject have different values ​​in the first input channel, they are corrected to R=Gr=Gb=B in the upstream photometry / AWB section 241 and amplifier section 242. From then on, it can be seen that R=G=B is maintained up to the upstream stage of the color reproduction matrix (CCM) in the color reproduction section 250. The reason for this is that while the white balance gain value and the coefficients of the color reproduction matrix are different for each channel, the coefficients of the other calculators are the same for each channel (and conversely, this should be the case). From the above, since R=G=B is maintained up to the stage just before the color reproduction section 250, it becomes necessary for the coefficients of the color reproduction matrix to maintain R=G=B even for its output. A method that is often used to achieve this is to impose constraints on this color reproduction matrix.

[0053] Figure 9 shows an example of this constraint. By adjusting the coefficients of each channel so that the sum of them is 1.0, the white point can be treated as a fixed point before and after processing, and the white color is preserved. As such, in the conventional technology, it is important to determine the coefficients so that the white point is preserved as a fixed point before and after color reproduction processing and other colors can be reproduced more accurately. However, if this constraint is not imposed, the degree of freedom of the color reproduction matrix may increase, which may lead to improvements, so a method is also used in which it is acceptable for the white point to move slightly before and after conversion.

[0054] In such cases, as shown in the example of Figure 10, it is common to maintain white by providing a white clip after the color reproduction process. The above signal processing is valid when focusing on the RGB sensor. However, with the other color filters shown in Figure 6, it is clear from the above discussion that it is impossible to maintain the white point of the D65 light source in the first place. For this reason, in a color space different from the RGB space, 1. Can the white point be maintained? 2. How do we define a color reproduction matrix? 3. What is white balance anyway? Issues such as the following arise.

[0055] For example, consider the RCB space. Here, a clear pixel is introduced instead of the G pixel. Then, in the case of a D65 light source, when C is the value of the clear pixel, the condition R = C = B raises the question of whether it is a white point. Generally, for the spectral characteristics D(λ) of a D light source that is distributed almost without attenuation points in the visible light region, the same subject (same spectral reflectance: O(λ)), and the characteristics (such as transmittance) L(λ) of the same optical system, the spectral characteristics S G (λ) and S C (λ) of the G pixel and the clear pixel of the RGGB sensor and the RCCB sensor shown in FIG. 7 are multiplied at each wavelength and integrated, and the following equation holds.

Equation

[0056] This means that when R = G = B under the same light source, subject (same spectral reflectance), and optical characteristics, R = B < C. Generally, the following equation holds.

Equation

[0057] That is, the white point defined in the RGB space does not coincide with the coordinates where each color component is equal on an arbitrary xyz color space. FIGS. 11 and 12 explain the geometric meaning of the color reproduction matrix.

[0058] Figure 11 shows an example of a color solid defined in an RGB space (e.g., sRGB space) and the white point of a certain light source, as well as representative colors such as magenta (Mg), cyan (Cy), and yellow. In this case, when the input and output colors are in the same RGB space, the calculation using the color reproduction matrix involves rotating the axis connecting the white point and the origin (black) as the rotation axis, and then arbitrarily stretching and shrinking the solid along the rotation axis. These geometric operations are mainly determined as conversion coefficients. When calculating the actual coefficients, first, an appropriate color chart (e.g., 24 colors) is photographed under a specific light source (e.g., D65 light source), and the photographed RAW data is input to a signal processing block (e.g., Figure 5) that applies it. In this case, the matrix coefficients are set to a unit matrix. The coefficients of the color reproduction matrix are obtained by calculating the least square distance between the obtained RGB image and the RGB values ​​of each color of the reference RGB image (target image). There are various methods for calculating the coefficients of the color reproduction matrix, but it is generally known that the coefficients are calculated by comparing with the reference image as shown here. As mentioned above, the transformations of the coefficients obtained in this way are ideally mainly rotation and axial stretching in RGB space (in reality, some arbitrary spatial transformation may be involved, but this is not dominant).

[0059] However, in the case of coordinate data in color spaces with different input and output, for example, conversion from RCB space to RGB space means that the color solid in RCB space surrounded by solid lines as shown in FIG. 12 is transformed into a color solid in RGB space surrounded by dotted lines by the conversion. As shown in the figure, this conversion requires arbitrary transformation that also involves the movement of the white point (indicating R=C=B in RCB space). In other words, it shows that the volume of the color solid before and after the conversion can also change significantly. In other words, it is equivalent to the absolute value of the determinant of the color reproduction matrix becoming larger. In other words, the deformation itself becomes larger, so the accuracy of the coefficient calculation is required. However, in reality, under conditions where the method based on the least square distance as described above is said to be the best, it can be said that there is a limit to the accuracy of such color reproduction processing.

[0060] Figure 13 shows the white balance gain values, color reproduction matrix coefficients, and their determinants obtained under light source D for the RGGB sensor and RCCB sensor with the characteristics in Figure 7. In the figure, "a" shows the white balance gain values, color reproduction matrix coefficients, and determinants obtained under light source D for the RGGB sensor. In the figure, "b" shows the white balance gain values, color reproduction matrix coefficients, and determinants obtained under light source D for the RCCB sensor.

[0061] As shown in a and b in the figure, it can be seen that there is a large difference in the gain value and the magnitude of the coefficient (the degree of deformation of the color solid). When there is such a difference, the final color reproduction index ΔE2000 worsens as the deformation becomes larger. In fact, it is known that the difference between the two is about ΔE2000=39. Furthermore, there are also white balance gain values ​​with a difference of nearly twice, which, together with the matrix coefficients, leads to the amplification of noise. In other words, when an RCCB sensor is developed using signal processing such as that shown in Figure 5, the trade-off between color reproducibility and noise can become an issue. If color reproducibility is emphasized, the white balance gain value and matrix coefficients tend to become large, and at the same time, noise is amplified. Conversely, if these gains and coefficients are lowered to suppress noise, the color reproducibility deteriorates.

[0062] Next, let us consider "What is white balance?" Up until now, we have assumed that the white balance gain of the RCCB sensor has been determined, i.e., R=C=B. This condition that each color channel is equal to each other is often necessary in demosaic processing. For example, it is used in directional detection for correct pixel complementation to suppress false color in achromatic areas in RGB space. In Figures 5 and 8, the correlation between pixels is used to find the minimum value using the magnitude relationship of each color difference (RG, BG), and for example, the direction of an edge is detected and correct complementation is performed. In other words, since it is necessary for the R, G, and B shading in achromatic areas to be purely uniform (white), white balance is often performed before demosaic processing. Examples of signal processing based on such a principle include Japanese Patent No. 6435560, Japanese Patent Publication No. 2009-147762, and "DLMMSE algorithm from L. Zhang and X. Wu, "Color demosaicking via directional linear minimum mean square-error estimation," IEEE Transactions on Image Processing 14 (12), 2167-2178, 2005-12." Of these, the third document is a basic algorithm that uses color difference, similar to Japanese Patent No. 6435560 and Japanese Patent Publication No. 2009-147762.

[0063] On the other hand, in the case of RCB space, there must be a condition for R=C=B. Usually, in RGB space, the coordinates of the white point are defined for each light source, as in standards such as CIE1931. In other words, the purpose of the white balance in signal processing is to find these coordinates while estimating the light source reflected on the subject. However, these coordinates do not exist in color spaces other than RGB, such as RCB space. In other words, some conversion is required. However, such a clear definition does not exist in each standard. Furthermore, even if R=C=B is true, due to the difference in the white points of the color reproduction mentioned above, the corresponding subject will not become achromatic (white) even if this gain is applied and the signal values ​​of each channel are made uniform. Therefore, it cannot be effective in suppressing false colors in the achromatic parts in the subsequent stage.

[0064] To summarize the above, it is clear that it is fundamentally impossible for image sensor input signals (e.g. R, C, B) generated in color spaces other than RGB to be compatible with signal processing optimized for the conventional RGB space in terms of image quality such as color reproduction, noise, and false colors. Moreover, it is not possible to find the white point without some kind of conversion. Recently, as market demands for high sensitivity have increased for in-vehicle image sensors, various RAW images generated in color spaces other than RGB are increasingly being developed using signal processing for conventional RGB sensors.

[0065] The present invention provides an image processing device that can properly process RAW signals generated in color spaces other than RGB, eliminate the trade-off between color reproduction and noise, and generate a correct white point, while maintaining compatibility with RGB sensors in a conventional signal processing pipeline for RGB sensors.

[0066] FIG. 2 shows a signal processing pipeline architecture in the first embodiment of the present invention. The blocks enclosed by dotted lines are different from the conventional technology. First, the amplifier 242 in the front stage of the demosaic processing is placed in the rear stage. Then, a method is adopted that does not involve false color suppression processing using color differences that imposes constraints on the input signal of the demosaic processing (such as aligning the signal levels of each pixel channel in the achromatic area in advance) as in the past. For example, as described in "Eric Dubois, "Frequency-domain methods for demosaicking of Bayer-sampled color images", IEEE Signal Processing Letters 12 (12), 847-850, 2005-12.", it is known that processing with good edge preservation and less false color and uniform spatial phase can be realized by separating each color channel (e.g., R, G, B) into three components by frequency domain (here, Fourier transform of the luminance component and two color difference components) and reconstructing them by performing appropriate intermediate filter processing. This demosaic processing can be applied to any color filter as long as it is a Bayer pattern.

[0067] Thus, there exists a demosaic process that can be applied to any color filter. Therefore, in the present invention, the above-mentioned problem can be solved by first performing a basis transformation in some way on any color channel (three components) with a uniform spatial phase into the RGB space, and then reapplying the white balance and color reproduction matrix in the RGB space.

[0068] First, consider an arbitrary color space other than RGB that has a common origin, with C0, C1, and C2 as color filters, as indicated by the dotted arrows in Figure 14. If these three vectors are considered independent of each other, then a transformation T exists that can convert them into RGB space. In other words, if transformation T is bijective (for example, a 3x3 matrix with an inverse matrix), then a one-to-one correspondence is possible between the C0C1C2 space and the RGB space. If such a coordinate transformation T can be generated, any color filter can be easily converted into the RGB space.

[0069] In FIG. 2, immediately after the demosaic process, the color basis conversion unit 230 performs the conversion T in FIG. 14 (Color basis transforming in the figure) to convert the pixel signal into the RGB space. Now, the method of creating this conversion T can be exemplified by the method shown in FIG. 7. That is, consider the conversion for each wavelength of the spectral characteristics of different color filters. First, discretize each wavelength (for example, sample every 5 nm), and consider the channel components (C0, C1, C2) and (R, G, B) at each wavelength as coordinate values ​​in the respective color spaces. Assume that (R, C, B) at b in FIG. 7 is converted to (R, G, B) at a in the same figure. Here, consider the conversion T[...], which is a linear conversion from three dimensions to three dimensions. That is, consider the following equation so that the root mean square of the distance from the value (C0, C1, C2) at the wavelength λ of each spectral characteristic to (R, G, B) at the same wavelength λ is minimized, for example, in the visible light wavelength range, through the conversion T.

number

[0070] Here, (C0, C1, C2) λ are the coordinates at wavelength λ of the spectral characteristics of an arbitrary color filter, (R,G,B) λ indicates the coordinates at wavelength λ of the spectral characteristics of the RGB sensor. d[...] indicates the distance between two points. The possible range of λ is the visible light region (approximately 380nm-700nm, in 5nm increments). Now, we define a matrix operation using transformation T by linearly combining (V1, V2, V3) with arbitrary real numbers a, b, c, d, e, f, g, h, and i, such that T[(V1, V2, V3)]=(aV1+bV2+cV3, dV1+eV2+fV3, gV1+hV2+iV3). Therefore, transformation T can be rewritten as a 3x3 representation matrix, and the above becomes equivalent to the following.

[0071] When data for each wavelength (C0, C1, C2) and (R, G, B) are collected and defined as 3xN vectors, and a 3x3 matrix T exists between them that satisfies the following relationship, finding this transformation matrix is ​​the same as finding the least squares distance of the following equation.

number

[0072] This can generally be solved using a pseudoinverse matrix. That is, if the 3xN vector on the left side of equation 4 is P and the 3xN vector on the right side is Q, then multiply P=TQ and both sides by the transpose of Q to find the pseudoinverse of QQ^t and determine T. This method is equivalent to the least squares method mentioned above and is expressed by the following formula.

number

[0073] This method is the same as the method described in Patent No. 6530751 for separating IR components from a 4x4 matrix. In this way, we have demonstrated one method for converting the bases of different color spaces. However, while Patent No. 6530751 mainly uses the spectral characteristics of the image sensor as shown in Figure 7, it is known that the spectral distribution of the light source also affects the conversion T in reality. In general, the modulation of sensor sensitivity due to an arbitrary light source spectrum can be expressed by the following equation.

number

[0074] In other words, it is necessary to determine the conversion T from the spectral characteristics of the image sensor taking into account the influence of the light source spectrum E(λ), that is, taking into account the light source spectrum.

[0075] Figure 15 shows the conversion T from RCB space to RGB space calculated for each light source using actual spectral characteristics. "Original" in the figure indicates no light source, and "D65" indicates the D65 light source. "A" indicates the A light source. The spectral characteristics of each image sensor were calculated using Figure 7. For Original (no light source) in Figure 15, the conversion T was calculated from the characteristics in Figure 7 itself. For the others, the spectrum of each light source was multiplied by the spectral spectrum of the image sensor, and the conversion coefficients were calculated using a similar method to that described above to create new spectral characteristics.

[0076] Here, |T| in Figure 15 is the value of the determinant of conversion T. As such, there is not a great difference in the spectral shapes of the D65 light source and no light source, so they can be said to be roughly the same conversion, but it can be seen that the conversion coefficients are different for the A light source. It is generally known that when the correlated color temperature differs greatly, the conversion coefficients also differ. So how do we estimate the correlated color temperature? Originally, correlated color temperature was also defined in XYZ space (or RGB space), so if we continue like this, we will fall into a chicken-and-egg controversy.

[0077] For example, it is conceivable to provide the color base conversion unit 230 in Fig. 2 with multiple matrix calculators to generate a signal using a conversion T that takes into account multiple light source spectra, but this is not a good idea as it would unnecessarily increase costs. Therefore, one representative conversion T is placed in the color base conversion unit 230. This may be the commonly used D light source (D65, etc.), or an (original) conversion obtained from the spectral characteristics of an image sensor that does not take the light source into account. In this way, we consider how to estimate the correlated color temperature for an arbitrary color filter and an arbitrary light source.

[0078] Now, there is a conversion T from any color filter (C0, C1, C2) to the RGB space of the (R, G, B) sensor. If the subject has a known light source (all white), the detection of AWB (auto white balance) estimated in the normal RGB space can be applied as is to the signal that has passed through the color basis conversion unit 230 in Fig. 2. In other words, under a known light source, even if the conversion T is fixed, the correlated color temperature and white balance gain can be estimated by calibration.

[0079] First, consider the detection values ​​(photometric values) detected for each window used when estimating normal white balance gain in the photometry / AWB unit 241 in Fig. 2 as shown in Fig. 16. The detection values ​​used here are R, G, B values ​​that have already been converted by the color base conversion unit 230, but here we distinguish that they have been converted into an R'G'B' space that actually contains errors, taking into account light source dependency.

[0080] The detection value data of each window region thus obtained is plotted on a plane such as B' / G'-R' / G', as exemplified in FIG. 17. In the figure, an example is shown in which detection values ​​obtained for each arbitrary window under light source A of an image sensor having an arbitrary color filter are plotted as black circles. Finally, the center of gravity of the plotted values ​​is determined as the position of light source A (white circle in the figure). The photometry / AWB unit 241 repeats the series of operations for each other known light source. Once such a calibration is performed, even if the conversion T originally depends on the light source, the correlated color temperature can be estimated as a unique conversion independent of the light source by reconstructing (obtaining by fitting, etc.) the blackbody locus based on the information obtained by calibration with the actual light source.

[0081] 18 is a flowchart showing an example of a calibration method in the first embodiment of the present technology. This calibration is started, for example, before the start of imaging. The photometry / AWB unit 241 acquires detection values ​​of each window (step S901) and converts the detection values ​​to R'G'B' (step S902). Then, the photometry / AWB unit 241 plots the converted values ​​on the B' / G'-R' / G' plane (step S903). The photometry / AWB unit 241 determines the center of gravity of the plotted values ​​as the coordinates of the set light source (step S904). After step S904, the photometry / AWB unit 241 ends the calibration.

[0082] By performing calibration once for each light source according to the flow of the figure, it becomes possible to estimate the correlated color temperature of an image sensor having the color filter by an AWB (auto white balance) algorithm operating in a normal RGB space. When the B' / G'-R' / G' plane is completed in this way, the photometry / AWB unit 241 can estimate the correlated color temperature and white balance gain value from each window detection value obtained from an arbitrary subject (for example, by obtaining the barycentric coordinate of each detection value as in the above calibration) as in a normal AWB. Such estimation is the same as that obtained in a generally widely known RGB space. For example, there is an estimation method described in International Publication No. 2010 / 095873, although the claimed purpose is different.

[0083] Therefore, the color base conversion unit 230 and photometry / AWB unit 241 in Fig. 2 can estimate the correlated color temperature and white balance gain from any color space by using any conversion T. However, the output signal value (main line signal) of the color base conversion unit 230, which is purposely distinguished from the R'G'B', will have errors due to light source dependency if left as is. Therefore, this error is absorbed by preparing correction coefficients for each correlated color temperature in the next stage, the color reproduction unit 250. In other words, an operation is performed to replace the correlated color temperature estimated by the photometry / AWB unit 241 in the same figure with a previously prepared CCM as a correction. Here, the CCM correction coefficients can be generated by calibration.

[0084] FIG. 19 is a diagram showing an example of a method for calculating correction matrix coefficients in the first embodiment of the present technology. The external circuit of the image sensor 200 prepares an appropriate color chart, and acquires color patch values ​​under known light sources by the image sensor 200 having an arbitrary color filter and the target RGB sensor. After that, the external circuit obtains the least square distance for each color patch between the value obtained by applying the conversion T to the value of each patch and the value of the target RGB sensor. This method is the same as the method for obtaining the conversion T. In this way, the external circuit can obtain CCM (Color Correction Matrix) coefficients for each correlated color temperature. The external circuit stores the obtained CCM coefficients in a register or the like in the image sensor 200. The CCM coefficients are read out by the color reproduction unit 250.

[0085] From the above, by combining the main signal that has been corrected for correlated color temperature, white balance gain, and light source dependency, and performing the processing shown in Figure 2, an image sensor with any color filter can achieve the same quality of color reproduction and white point as the target RGB sensor. Note that the CCM in the figure is a simple 3x3 matrix and does not impose the above-mentioned constraint (the sum of row components is 1.0). Instead, the white clip in Figure 10 is introduced in the latter stage. Also, the target RGB values ​​do not have to be the values ​​of the actual sensor. It is common for users to target any RGB value they want.

[0086] FIG. 20 is an example showing the synthesis coefficients of conversion T×CCM and the white balance gain values ​​under the D65 light source of the RCCB sensor having the characteristics of FIG. 7. It is clear that the method of the present invention can convert with smaller values ​​than the CCM coefficients and white balance gain values ​​obtained by the conventional signal processing shown in FIG. 13. Furthermore, since the value of the determinant indicating the degree of conversion is also small, the distortion of the space before and after conversion can be suppressed. In this way, it can be seen that the present invention can achieve color reproduction similar to that of the target RGB sensor with high accuracy, low distortion, and suppressed noise by correcting the error of conversion T using conversion T and various calibrations. This can further improve image quality.

[0087] Also, in Fig. 2, it is easy to see that compatibility as a normal RGB sensor is maintained when the basis conversion matrix of the color base conversion unit 230 is replaced with a unit matrix. For example, when the process of Fig. 2 is executed outside the image sensor 200, the color base conversion unit 230 can hold a unit matrix and a basis conversion matrix and switch the matrix to be applied depending on the type of sensor. The type of sensor is specified by a setting signal from the outside.

[0088] In the dotted line portion of the figure, which is the basis of the present invention, the three blocks of processing called the main line system: color base conversion unit 230 →amplification unit 242 →color reproduction unit 250 can be formulated and expressed as follows.

number

[0089] From the left in the above formula, the order is CCM, white balance gain, the basis conversion matrix in the color basis conversion unit 230, and the input signal value (C0, C1, C2). Here, α, 1.0, and β indicate the white balance gain of each channel. The other coefficients are the same as those shown above.

[0090] In this way, according to the first embodiment of the present technology, the color base conversion unit 230 converts the color space from the C0C1C2 space to the RGB space, and after the conversion, the amplifier 242 amplifies the image by the white balance gain, so that it is possible to improve color reproducibility and reduce noise at the same time. This makes it possible to improve image quality.

[0091] <2. Second embodiment> In the above-described first embodiment, the color base converter 230 converts the color space of the demosaic image, but the present invention is not limited to this configuration as long as the processing of Equation 7 can be realized. The image sensor 200 in the second embodiment differs from the first embodiment in that the color base converter detects a RAW image and converts the color space of the detected value.

[0092] 21 is a block diagram showing a configuration example of an image sensor 200 according to a second embodiment of the present technology. The image sensor 200 according to the second embodiment does not include a white clip processing unit 260. Also, instead of the color base conversion unit 230 and the photometry / AWB unit 241, a color base conversion unit 231 and an AWB unit 243 are arranged, and instead of the color reproduction unit 250, a color reproduction unit 251 is arranged.

[0093] The color base converter 231 receives a RAW image from the pixel array unit 210. The color base converter 231 detects the RAW image and performs color base conversion processing on the detected value. The color base converter 231 then supplies the converted detected value to the AWB unit 243.

[0094] The AWB unit 243 estimates a white balance gain and a correlated color temperature based on the detection value from the color base conversion unit 231, and supplies them to the color reproduction unit 251. In addition, a gain of 1.0 is set in the amplification unit 242, and the RAW image that has essentially bypassed the amplification unit 242 is input to the demosaic processing unit 220.

[0095] The color reproduction unit 251 amplifies the color signals in the demosaic image by a white balance gain and corrects them by CCM.

[0096] As shown in the figure, equivalent processing is possible even with the conventional architecture shown in Figure 3. However, some software algorithm changes are required. As mentioned above, by always giving 1.0 to all channels of the amplifier 242, they are bypassed. Next, in the block that performs photometry in the photometry / AWB 241, the following algorithm change is made by software to realize the function of the color basis conversion unit 231. In most cases, detection processing as shown in Figure 16 is implemented in hardware. This process itself is not changed, and conversion T is applied to each detection value (sum(C0), sum(C1), sum(C2)) obtained for an image sensor with any color filter. By plotting them on the B / GR / G plane, the same results as those discussed so far can be obtained.

[0097] In terms of cost, the matrix calculation in FIG. 22 is performed for several white balance detection frames (number of windows). The coefficients of the conversion T handled here are fixed values ​​as in the above discussion, and the same results as in the above discussion can be obtained by performing similar calibration according to FIGS. 17 and 18. The only difference from the first embodiment is whether the signal values ​​before detection are converted to RGB space in advance by hardware, or converted after detection by the color base conversion unit 231. Therefore, α and β, which correspond to the diagonal matrix among the products of the three matrices formulated above, are generated by AWB together with the correlated color temperature, and these are multiplied by the conversion T, and the correction CCM coefficient according to the correlated color temperature is also calculated, and finally, they can be summarized into one 3x3 matrix.

[0098] As described above, according to the second embodiment of the present technology, the color base conversion unit 231 detects a RAW image and converts the color space of the detected value, so that a function similar to that of the first embodiment can be realized in software using conventional hardware such as that shown in FIG. 3.

[0099] The present invention provides a method for generating correlated color temperatures similar to those of RGB sensors and correcting the main line system simultaneously by devising appropriate calibration, without causing deterioration of color reproduction or amplification of noise when using any color filter required for signal processing used in various applications, and because the conversion required to achieve this is light source dependent.In addition, since the essence of the present invention lies in its principles, it is not limited to the architecture itself, and has the flexibility to be realized by simply executing software workarounds even for conventional systems.

[0100] <3. Application examples for mobile objects> The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of moving body such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility device, an airplane, a drone, a ship, a robot, etc.

[0101] FIG. 23 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a moving object control system to which the technology according to the present disclosure can be applied. The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in Fig. 23, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside-vehicle information detection unit 12030, an inside-vehicle information detection unit 12040, and an integrated control unit 12050. Also, as functional configurations of the integrated control unit 12050, a microcomputer 12051, an audio / video output unit 12052, and an in-vehicle network I / F (interface) 12053 are illustrated.

[0102] The drivetrain control unit 12010 controls the operation of devices related to the drivetrain of the vehicle in accordance with various programs. For example, the drivetrain control unit 12010 functions as a control device for a drive force generating device for generating a drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force of the vehicle.

[0103] The body system control unit 12020 controls the operation of various devices mounted on the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as head lamps, back lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves or signals of various switches transmitted from a portable device that replaces a key may be input to the body system control unit 12020. The body system control unit 12020 receives the input of these radio waves or signals and controls the door lock device, power window device, lamps, and the like of the vehicle.

[0104] The outside-vehicle information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the outside-vehicle information detection unit 12030. The outside-vehicle information detection unit 12030 causes the imaging unit 12031 to capture an image outside the vehicle and receives the captured image. The outside-vehicle information detection unit 12030 may perform object detection processing or distance detection processing for a person, a car, an obstacle, a sign, or characters on a road surface, based on the received image.

[0105] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal according to the amount of light received. The imaging unit 12031 can output the electrical signal as an image, or as distance measurement information. The light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

[0106] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041 that detects the state of the driver is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 may calculate the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing.

[0107] The microcomputer 12051 can calculate control target values ​​for the driving force generating device, the steering mechanism, or the braking device based on the information inside and outside the vehicle acquired by the outside-of-vehicle information detection unit 12030 or the inside-vehicle information detection unit 12040, and output a control command to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the inter-vehicle distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane departure warning, etc.

[0108] In addition, the microcomputer 12051 can perform cooperative control for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on the driver's operation, by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle acquired by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040.

[0109] Furthermore, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the outside-vehicle information detection unit 12030. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside-vehicle information detection unit 12030, and perform cooperative control for the purpose of preventing glare, such as switching from high beams to low beams.

[0110] The audio / video output unit 12052 transmits at least one output signal of audio and video to an output device capable of visually or audibly notifying information to passengers in the vehicle or the outside of the vehicle. In the example of Fig. 23, an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are exemplified as the output device. The display unit 12062 may include at least one of an on-board display and a head-up display, for example.

[0111] FIG. 24 is a diagram showing an example of the installation position of the imaging unit 12031.

[0112] In FIG. 24, the imaging unit 12031 includes imaging units 12101, 12102, 12103, 12104, and 12105.

[0113] The imaging units 12101, 12102, 12103, 12104, and 12105 are provided at positions such as the front nose, side mirrors, rear bumper, back door, and the upper part of the windshield inside the vehicle cabin of the vehicle 12100. The imaging unit 12101 provided at the front nose and the imaging unit 12105 provided at the upper part of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 provided at the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 provided at the rear bumper or back door mainly acquires images of the rear of the vehicle 12100. The imaging unit 12105 provided at the upper part of the windshield inside the vehicle cabin is mainly used to detect a preceding vehicle, a pedestrian, an obstacle, a traffic light, a traffic sign, a lane, or the like.

[0114] 24 shows an example of the imaging ranges of the imaging units 12101 to 12104. Imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or back door. For example, an overhead image of the vehicle 12100 viewed from above is obtained by superimposing the image data captured by the imaging units 12101 to 12104.

[0115] At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera made up of a plurality of imaging elements, or may be an imaging element having pixels for detecting a phase difference.

[0116] For example, the microcomputer 12051 can extract, as a preceding vehicle, a three-dimensional object that is the closest three-dimensional object on the travel path of the vehicle 12100 and travels at a predetermined speed (for example, 0 km / h or more) in approximately the same direction as the vehicle 12100, by calculating the distance to each three-dimensional object in the imaging ranges 12111 to 12114 and the change over time of this distance (relative speed with respect to the vehicle 12100) based on the distance information obtained from the imaging units 12101 to 12104. Furthermore, the microcomputer 12051 can set a vehicle distance to be secured in advance in front of the preceding vehicle, and perform automatic brake control (including follow-up stop control) and automatic acceleration control (including follow-up start control). In this way, cooperative control can be performed for the purpose of automatic driving that travels autonomously without relying on the driver's operation.

[0117] For example, the microcomputer 12051 classifies and extracts three-dimensional object data on three-dimensional objects, such as two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects, based on the distance information obtained from the imaging units 12101 to 12104, and can use the data for automatic obstacle avoidance. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. Then, the microcomputer 12051 determines a collision risk indicating the degree of danger of collision with each obstacle, and when the collision risk is equal to or exceeds a set value and there is a possibility of collision, the microcomputer 12051 can provide driving assistance for collision avoidance by outputting an alarm to the driver via the audio speaker 12061 or the display unit 12062, or by performing forced deceleration or avoidance steering via the drive system control unit 12010.

[0118] At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether or not a pedestrian is present in the captured images of the imaging units 12101 to 12104. The recognition of such a pedestrian is performed, for example, by a procedure of extracting feature points in the captured images of the imaging units 12101 to 12104 as infrared cameras, and a procedure of performing pattern matching processing on a series of feature points that indicate the contour of an object to determine whether or not the object is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the captured images of the imaging units 12101 to 12104 and recognizes the pedestrian, the audio / image output unit 12052 controls the display unit 12062 to superimpose a rectangular contour line for emphasis on the recognized pedestrian. The audio / image output unit 12052 may also control the display unit 12062 to display an icon or the like indicating a pedestrian at a desired position.

[0119] An example of a vehicle control system to which the technology according to the present disclosure can be applied has been described above. The technology according to the present disclosure can be applied to the imaging unit 12031 among the configurations described above. Specifically, the image sensor 200 in FIG. 2 can be applied to the imaging unit 12031. By applying the technology according to the present disclosure to the imaging unit 12031, a captured image that is easier to see can be obtained, thereby reducing driver fatigue.

[0120] The above-mentioned embodiment shows an example for realizing the present technology, and the matters in the embodiment and the matters specified in the claims correspond to each other. Similarly, the matters specified in the claims and the matters in the embodiment of the present technology having the same name correspond to each other. However, the present technology is not limited to the embodiment, and can be realized by applying various modifications to the embodiment within the scope of the gist of the technology.

[0121] The processing procedures described in the above embodiments may be regarded as a method having a series of these procedures, or as a program for causing a computer to execute the series of procedures or a recording medium for storing the program. For example, a CD (Compact Disc), an MD (MiniDisc), a DVD (Digital Versatile Disc), a memory card, a Blu-ray (registered trademark) Disc, etc. may be used as this recording medium. It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0122] The present technology can also be configured as follows. (1) a color base conversion unit that performs a color base conversion process on an image to convert a color space from a first color space to a second color space; an image quality adjustment processing unit that performs a predetermined image quality adjustment process on the image after the color basis conversion process; An image processing device comprising: (2) further comprising a demosaic processing unit for converting a RAW image into a demosaic image; The image is the RAW image or the demosaic image. The image processing device according to (1) above. (3) a photometry / AWB unit that detects the demosaic image after the color basis conversion process and estimates a white balance gain; an amplifier that amplifies a color signal in the demosaic image after the color basis conversion process by the white balance gain and supplies the amplified color signal to the image quality adjustment processor; Further comprising: The color basis conversion unit performs the color basis conversion process on the demosaic image, The image quality adjustment processing unit a color reproduction unit that corrects the amplified color signal using a color reproduction matrix; a white clip processing unit that limits the corrected color signal by a predetermined threshold value; Equipped The image processing device according to (2) above. (4) further comprising an AWB unit for estimating a white balance gain; the color base conversion unit performs the color base conversion process on the detection value of the RAW image and supplies the result to the AWB unit; The image quality adjustment processing unit includes a color reproduction unit that amplifies color signals in the demosaic image by the white balance gain and corrects the color signals by a color reproduction matrix. The image processing device according to (2) above. (5) the first color space is a color space different from the RGB space; The second color space is an RGB space. The image processing device according to any one of (1) to (4). (6) Further comprising a pixel array section for generating the image. The image processing device according to any one of (1) to (5). (7) A plurality of pixels including clear pixels are arranged in the pixel array section, The clear pixels are provided with transparent filters. The image processing device according to (6) above. (8) The image quality adjustment processing unit executes at least one of gamma correction processing, edge enhancement processing, YUV conversion processing, and color conversion processing. The image processing device according to any one of (1) to (7). (9) a color basis conversion step of performing a color basis conversion process on the image to convert a color space from a first color space to a second color space; an image quality adjustment processing step of performing a predetermined image quality adjustment processing on the image after the color basis conversion processing; An image processing method comprising: (10) a color basis conversion step of performing a color basis conversion process on the image to convert a color space from a first color space to a second color space; an image quality adjustment process for performing a predetermined image quality adjustment process on the image after the color basis conversion process; A program for causing a computer to execute the following. [Explanation of symbols]

[0123] 100 Imaging device 110 Optics Department 120 DSP circuits 130 Display section 140 Operation section 150 Bus 160 Frame Memory 170 Storage section 180 Power supply section 200 Image Sensor 210 Pixel array section 211 pixels 220 Demosaic Processing Unit 230, 231 Color base conversion unit 241 Photometry / AWB section 242 Amplification section 243 AWB Department 250, 251 Color reproduction section 260 White Clip Processing Unit 271 Gamma correction section 272 Edge Enhancement 273 YUV conversion section 274 Color conversion unit 275 DNN Monitor

Claims

1. a color base conversion unit that performs color base conversion processing on the image to convert a color space from a first color space to a second color space; an image quality adjustment processing unit that performs a predetermined image quality adjustment process on the image after the color basis conversion process; An image processing device comprising:

2. The image processing apparatus further includes a demosaic processing unit that converts the raw image into a demosaic image, The image is the RAW image or the demosaic image.

2. The image processing device according to claim 1.

3. a photometry and AWB (Auto White Balance) unit that detects the demosaic image after the color basis conversion process and estimates a white balance gain; an amplifier that amplifies a color signal in the demosaic image after the color basis conversion process by the white balance gain and supplies the amplified color signal to the image quality adjustment processor; Further comprising: The color basis conversion unit performs the color basis conversion process on the demosaic image, The image quality adjustment processing unit a color reproduction unit that corrects the amplified color signal using a color reproduction matrix; a white clip processing unit that limits the corrected color signal by a predetermined threshold value; Equipped 3. The image processing device according to claim 2.

4. Further comprising an AWB unit for estimating a white balance gain, the color base conversion unit performs the color base conversion process on the detection value of the RAW image and supplies the result to the AWB unit; The image quality adjustment processing unit includes a color reproduction unit that amplifies color signals in the demosaic image by the white balance gain and corrects the color signals by a color reproduction matrix.

3. The image processing device according to claim 2.

5. the first color space is a color space different from an RGB space; The second color space is an RGB space.

2. The image processing device according to claim 1.

6. The image generating device further includes a pixel array portion for generating the image.

2. The image processing device according to claim 1.

7. A plurality of pixels including clear pixels are arranged in the pixel array section, The clear pixels are provided with transparent filters.

7. The image processing device according to claim 6.

8. The image quality adjustment processing unit executes at least one of a gamma correction process, an edge enhancement process, a YUV conversion process, and a color conversion process.

2. The image processing device according to claim 1.

9. a color base conversion procedure for performing a color base conversion process on the image to convert a color space from a first color space to a second color space; an image quality adjustment process for performing a predetermined image quality adjustment process on the image after the color basis conversion process; An image processing method comprising:

10. a color base conversion procedure for performing a color base conversion process on the image to convert a color space from a first color space to a second color space; an image quality adjustment process for performing a predetermined image quality adjustment process on the image after the color basis conversion process; A program for causing a computer to execute the following.

Images