Image processing apparatus and image processing method

Abstract

This image processing device converts original image data, obtained by imaging performed by an imaging element through a color filter including segments of a red-based color and a complementary color, into primary color-based image data, and acquires a statistic value of a plurality of pixel data items in the primary color-based image data. The image processing device calculates a correction parameter using the acquired statistic value, corrects the original image data on the basis of the correction parameter, and generates corrected image data. In this way, it is possible to perform an appropriate correction process on the image data obtained by imaging performed through the color filter including the segments of a red-based color and a complementary color.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus and an image processing method.

[0002] This application claims priority based on U.S. Provisional Application No. 62 / 954,056 filed on December 27, 2019, and incorporates by reference all of the descriptions set forth in the above application.

Background Art

[0003] Image data obtained by imaging with a digital camera or the like is subjected to various correction processes by an image processing apparatus or the like in order to obtain an optimal image. For example, a process of improving the brightness in accordance with the skin color of a person included in the image data is performed. Alternatively, a shading correction coefficient is calculated based on a statistical value of pixel values calculated from blocks of image data divided into a plurality, or a white balance correction coefficient is calculated. This type of correction process is often performed using image data in the RGB (red, green, blue) color space.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Patent Document 5

Patent Document 6

Summary of the Invention

Problems to be Solved by the Invention

[0005] Recently, a technology has been known in which an imaging device such as a camera is mounted on a moving body such as an automobile, and objects (other moving bodies, signals, signs, white lines on the road, people, etc.) around the moving body are recognized from images captured by the imaging device. For this type of imaging device, for example, improvement in recognition of red-based colors such as stop lamps, signals or no-entry marks, improvement in recognition of yellow-based colors such as sodium lamps, or improvement in light reception sensitivity in dark places such as at night is required. In order to meet these requirements, a RYeCy (red, yellow, cyan) filter may be used instead of an RGB filter in the imaging device.

[0006] However, noise removal processing, white balance correction, color correction processing, etc. are carried out using image data in the RGB color space, and no method for carrying out correction processing of image data in the RYeCy color space has been proposed.

[0007] The present invention has been made in view of the above points, and an object thereof is to generate a corrected image with excellent image quality in the correction processing of image data captured through a color filter including segments of a red-based color and at least one complementary color.

Means for Solving the Problems

[0008] In one aspect of the present invention, an image processing apparatus is an image processing apparatus that performs correction processing of original image data generated by an image sensor that receives light with a plurality of pixels through a color filter including segments of a red-based color and at least one complementary color, the image processing apparatus including: a conversion process for converting the original image data into original color system image data represented in an original color system color space; a statistical acquisition process for acquiring statistical values of a plurality of pixel data corresponding to the plurality of pixels in the original color system image data; a parameter calculation process for calculating correction parameters using the statistical values; and a correction process for correcting the original image data based on the correction parameters to generate corrected image data.

Effects of the Invention

[0009] According to the disclosed technology, correction processing of image data captured through a color filter including red-based colors and segments of at least one complementary color can be appropriately performed.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments will be described with reference to the drawings. In the following, the same reference numerals as the signal names are used for the signal lines through which information such as signals is transmitted. One signal line shown in the drawings may be composed of a plurality of bits. Also, in the following description, image data may sometimes be simply referred to as an image.

[0012] (First Embodiment) FIG. 1 shows an example of an image processing system including an image processing apparatus according to the first embodiment. The image processing system 10 shown in FIG. 1 is mounted on a moving body 20 such as an automobile, for example. Imaging devices 22A, 22B, 22C, and 22D such as cameras are installed in front of, behind, to the left, and to the right of the moving direction D of the moving body 20. Hereinafter, when the imaging devices 22A, 22B, 22C, and 22D are described without distinction, they are also referred to as the imaging device 22.

[0013] Note that the number and installation positions of the imaging devices 22 installed on the moving body 20 are not limited to those shown in FIG. 1. For example, the imaging device 22 may be installed only in front of the moving body 20, or only in front and behind. Alternatively, the imaging device 22 may be grounded to the ceiling of the moving body 20. Also, the moving body 20 on which the image processing system 10 is mounted is not limited to an automobile, and may be, for example, a transport robot or a drone operating in a factory.

[0014] The image processing system 10 includes an image processing apparatus 12, a display device 14, and an information processing apparatus 16 connected to the image processing apparatus 12. In FIG. 1, for the sake of easy understanding, the image processing system 10 is superimposed on an image diagram of the moving body 20 viewed from above. However, in reality, the image processing apparatus 12 and the information processing apparatus 16 are mounted on a control board or the like mounted on the moving body 20, and the display device 14 is installed at a position visible to a person inside the moving body 20. Note that the image processing apparatus 12 may be mounted on a control board or the like as a part of the information processing apparatus 16.

[0015] The image processing apparatus 12 is connected to each imaging apparatus 22 via a signal line or wirelessly, and acquires image data indicating an image around the moving body 20 captured by each imaging apparatus 22. The image processing apparatus 12 performs image processing (correction processing) on the image data acquired from each imaging apparatus 22, and outputs the result of the image processing to at least one of the display device 14 and the information processing apparatus 16.

[0016] The display device 14 is, for example, a side mirror monitor, a rear mirror monitor, or a car navigation device installed in the moving body 20. Note that the display device 14 may be a display provided on a dashboard or the like, or a head-up display that projects an image onto a projection plate or a windshield or the like. Further, the image processing system 10 may not include the display device 14.

[0017] The information processing apparatus 16 includes a computer such as a processor that performs recognition processing or the like based on the image data received via the image processing apparatus 12. For example, the information processing apparatus 16 mounted on the moving body 20 detects other moving bodies, signals, signs, white lines on the road, and people by performing recognition processing on the image data, and determines the situation around the moving body 20 based on the detection result. Note that the information processing apparatus 16 may include an automatic driving control device that controls the movement, stop, right turn, and left turn of the moving body 20.

[0018] Figure 2 shows an overview of the semiconductor device on which the image processing apparatus 12 of FIG. 1 is mounted. For example, the image processing apparatus 12 is included in the semiconductor chip 32 of the semiconductor device 30 shown in FIG. 2. Although transistors, wirings, etc. for realizing the functions of the image processing apparatus 12 are provided in the semiconductor chip 32, illustration thereof is omitted.

[0019] The semiconductor chip 32 is respectively connected to a plurality of wirings 36 provided on the wiring board 34 via a plurality of bumps BP1. The plurality of wirings 36 are connected to bumps BP2 which are external connection terminals provided on the back surface of the wiring board 34. Then, by connecting the bumps BP2 to a control board or the like, the semiconductor device 30 is mounted on the control board or the like.

[0020] Note that the information processing apparatus 16 of FIG. 1 may have the same configuration as the semiconductor device 30, or may have a semiconductor device similar to the semiconductor device 30. Also, the image processing apparatus 12 and the information processing apparatus 16 may be included in one semiconductor device.

[0021] Figure 3 shows an example of the arrangement of pixels PX of the image sensor IMGS mounted on each imaging device 22 of FIG. 1. Although not particularly limited, for example, the image sensor IMGS is a CMOS (Complementary Metal Oxide Semiconductor) image sensor, which is an example of an imaging element.

[0022] The pixels PX are arranged in a matrix, and each pixel PX has a light receiving element and a color filter provided on the light receiving element. R, Ye, and Cy attached to each pixel PX indicate the colors of the respective segments of the color filter, where R means red, Ye means yellow, and Cy means cyan. That is, each imaging device 22 receives light with a plurality of pixels through the RYeCy color filter and generates image data. R is a primary color system, and Ye and Cy are complementary color systems.

[0023] In the example shown in FIG. 3, pixel groups each consisting of a total of four pixels arranged in two rows and two columns are repeatedly arranged in the vertical and horizontal directions. For example, a pixel group has a pixel R arranged in the upper left, a pixel Ye arranged in the upper right, a pixel Ye arranged in the lower left, and a pixel Cy arranged in the lower right. Each pixel R, Ye, Cy outputs a signal indicating the intensity of light of the corresponding color according to the intensity of the received light.

[0024] Then, each imaging device 22 shown in FIG. 1 generates image data indicating the scenery around the moving body 20 based on the signals indicating the intensity of light output for each pixel R, Ye, Cy, and after performing image processing on the generated image data, outputs it to the image processing device 12. The imaging device 22 including the image sensor IMGS shown in FIG. 3 outputs image data having color information of R, Ye, Cy.

[0025] Note that the color filter of each pixel group of the image sensor IMGS is not limited to RYeYeCy. For example, instead of pixel R, a magenta pixel M may be arranged, or other red-based color pixels may be arranged. In other words, instead of the red segment of the color filter, a magenta or other red-based color segment may be arranged. Also, the arrangement of the pixels within the pixel group and the arrangement pattern of the pixel groups are not limited to those in FIG. 3 and may be changed as appropriate.

[0026] FIG. 4 shows an example of the configuration of the image processing device 12 in FIG. 1. The image processing device 12 has, for example, an interface unit 121, a CPU 122, a main storage device 123, an auxiliary storage device 124, and an image processing unit 125, etc., which are connected to each other via a bus BUS.

[0027] The interface unit 121 receives the image data output from each imaging device 22 or at least one of the imaging devices 22. Also, the interface unit 121 receives the corrected image data generated by the image processing unit 125 and outputs the received image data to at least one of the display device 14 and the information processing device 16.

[0028] The CPU 122 controls the overall operation of the image processing apparatus 12 by executing, for example, an image processing program. For example, the main memory device 123 is a semiconductor memory such as a DRAM (Dynamic Random Access Memory). For example, the main memory device 123 stores an image processing program transferred from the auxiliary storage device 124 and work data used by the image processing unit 125 during image processing.

[0029] For example, the auxiliary storage device 124 is an HDD (Hard Disk Drive) or an SSD (Solid State Drive), etc. As an example, the auxiliary storage device 124 stores image data received from the imaging device 22, parameters used by the image processing unit 125 for image processing, and an image processing program.

[0030] The image processing unit 125 acquires the image data output from each imaging device 22 via the interface unit 121. The image processing unit 125 performs image processing such as conversion processing or correction processing on the acquired image data, and generates corrected image data.

[0031] For example, the image processing unit 125 converts the image data represented in the RYeCy color space received from the imaging device 22 having the image sensor IMGS shown in FIG. 3 into the RGB (Red, Green, Blue) color space. Then, the image processing unit 125 calculates correction parameters for correction processing using the image data converted into the RGB color space, and performs correction processing on the image data represented in the RYeCy color space using the calculated correction parameters to generate corrected image data. Further, the image processing unit 125 converts the corrected image data represented in the RYeCy color space into the RGB color space to generate RGB corrected image data. The RGB color space is an example of a primary color system color space, and the RGB image data is an example of primary color system image data.

[0032] The image processing apparatus 12 outputs the corrected image data converted into the RGB color space to the display device 14 via the interface unit 121. Further, the image processing apparatus 12 outputs the corrected image data represented in the RYeCy color space to the information processing apparatus 16 via the interface unit 121. Thereby, the information processing apparatus 16 can perform processes such as object recognition using the corrected image data in the RYeCy color space instead of the corrected image data converted from a color space other than RYeCy.

[0033] In this case, for example, the intensity of red light such as the red color of a traffic signal received by pixel R and the brake lamp of an automobile can be made higher than when calculated from other pixels. Also, the intensity of the yellow reflected light of an object illuminated by yellow light such as a sodium lamp received by pixel Ye can be made higher than when calculated from other pixels. As a result, the recognition accuracy of an object illuminated by a red traffic signal lamp, a sodium lamp, or the like can be improved.

[0034] FIG. 5 shows an example of the correction process performed by the image processing unit 125 in FIG. 4. That is, FIG. 5 shows an example of the image processing method by the image processing apparatus 12. For example, the process shown in FIG. 5 is performed by the hardware of the image processing unit 125. The image processing unit 125 may be designed by an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit), etc.

[0035] In this embodiment, the image processing unit 125 receives the image data represented in the RYeCy color space output from the imaging device 22 and performs at least one type of correction process in step S100. At this time, the image processing unit 125 converts the image data represented in the RYeCy color space into the RGB color space, and calculates correction parameters used for correcting the image data represented in the RYeCy color space using the image data represented in the RGB color space.

[0036] The image data obtained through the RYeCy color filter has higher luminance but lower color discrimination accuracy compared to the image data obtained through the RGB color filter. This is because the overlapping portion between the wavelength regions detected by the Ye pixels and the wavelength regions detected by the R and Cy pixels is larger than the overlapping portion of the wavelength regions detected by each of the RGB pixels.

[0037] In this embodiment, the correction parameters used for the correction process of the RYeCy image are calculated using an RGB image with higher color discrimination accuracy compared to the RYeCy image. By performing the correction process of the RYeCy image with higher luminance compared to the RGB image using the RGB color space, the image quality of the RYeCy corrected image can be improved compared to directly correcting the RYeCy image.

[0038] An example of the correction process performed in step S100 will be described with reference to FIGS. 6 to 13. Hereinafter, the image data represented in the RYeCy color space is also referred to as the RYeCy image, and the image data represented in the RGB color space is also referred to as the RGB image.

[0039] First, in step S101, the image processing unit 125 acquires, for example, the RYeCy image output from the imaging device 22. Next, in step S102, the image processing unit 125 converts the RYeCy image into an RGB image represented in the RGB color space.

[0040] Next, in step S103, the image processing unit 125 calculates statistical values (statistics), such as signal values representing the intensities of each of the R, G, and B pixels, for the entire range or a predetermined partial range of the RGB image. The process of step S103 is an example of the statistical acquisition process. Next, in step S104, the image processing unit 125 calculates the correction parameters to be used for correcting the RYeCy image acquired in step S101 based on the calculated statistical values. The process of step S104 is an example of the parameter calculation process. The correction parameters calculated by the image processing unit 125 differ depending on the content of the correction process performed in step S100.

[0041] Next, in step S105, the image processing unit 125 performs correction processing on the RYeCy image acquired in step S101 using the correction parameters calculated in step S104, and generates corrected image data (RYeCy corrected image) represented in the color space of RYeCy. The process of step S105 is an example of correction processing for generating corrected image data.

[0042] Note that the image processing unit 125 can convert the correction parameters calculated using the RGB image into correction parameters for the RYeCy image by performing a process reverse to the process when converting from the RYeCy image to the RGB image on the correction parameters. The conversion of the correction parameters may be performed in step S104.

[0043] Next, in step S110, the image processing unit 125 outputs the corrected image data (RYeCy corrected image) corrected in step S105 as image processing data to, for example, the information processing apparatus 16 in FIG. 1. In parallel with the process of step S110, in step S120, the image processing unit 125 converts the RYeCy corrected image into an RGB corrected image represented in the RGB color space. Next, in step S130, the image processing unit 125 outputs the RGB corrected image as display data to, for example, the display device 14 in FIG. 1.

[0044] In this way, the RGB image output from the image processing unit 125 to the outside (for example, the display device 14) is not the RGB image converted in step S102, but the RGB corrected image converted from the RYeCy corrected image. Therefore, for example, it is possible to prevent an RGB image with missing color information during use in step S100 from being output from the image processing unit 125. Also, even when the conversion process to the RGB image in step S102 is performed using a simple matrix arithmetic formula, the image processing unit 125 can output the RGB corrected image as a normal RGB image.

[0045] After steps S110 and S130, in step S140, if the image processing unit 125 determines to continue the correction process, the process returns to step S101 to perform at least one type of correction process. On the other hand, in step S70, if the image processing unit 125 determines the end of the correction process, the correction process shown in FIG. 5 ends. For example, when the image processing unit 125 receives the next RYeCy image data from the imaging device 22, the process returns to step S101 to continue the correction process, and when the image processing unit 125 does not receive the RYeCy image data from the imaging device 22, the correction process ends.

[0046] FIG. 6 shows an example of a noise removal process, which is one of the correction processes performed in step S100 of FIG. 5. That is, FIG. 6 shows an example of an image processing method by the image processing apparatus 12. Step S30 in FIG. 6 corresponds to step S100 in FIG. 5. Also, steps S31, S32, and S33 in FIG. 6 correspond to steps S101, S102, and S103 in FIG. 5, respectively. Steps S341 and S342 in FIG. 6 correspond to step S104 in FIG. 5. Steps S351 and S352 in FIG. 6 correspond to step S105 in FIG. 5.

[0047] In step S31, the image processing unit 125 acquires, for example, the RYeCy image output from the imaging device 22. Next, in step S32, the image processing unit 125 converts the RYeCy image into an RGB image represented in the RGB color space (YCbCr coordinate system). In the YCbCr coordinate system, the image data of each pixel is represented by the luminance information Y and the color difference information Cb and Cr. Note that the image processing unit 125 may convert the RYeCy image into the Lab color space.

[0048] Also, the conversion to the RGB image is performed by matrix operations according to the standard, but it may be performed by a simple matrix operation formula. The pseudo RGB image converted by the simple matrix operation formula has low color reproducibility while reducing the noise of the RGB components. Step S32 The generated RGB image is used to remove noise and is not used for display on the display device 14, so there is no problem even if the color reproducibility is low.

[0049] Furthermore, by using a simple matrix arithmetic expression, the load on the image processing unit 125 can be reduced. As a result, the processing time required for the correction processing can be shortened, and the power consumption of the image processing unit 125 can be reduced. The simple matrix arithmetic expression will be described with reference to FIG. 7.

[0050] Next, in step S33, the image processing unit 125 performs noise removal processing (NR) on the color difference information Cb and Cr of the image in the YCbCr coordinate system converted in step S32. In other words, the image processing unit 125 acquires the color difference information Cb and Cr with noise removed as statistical values (statistics). By removing the noise of the color difference information Cb and Cr that does not include the luminance information Y and is closer to the visual characteristics, color noise can be effectively removed.

[0051] Next, in step S341, the image processing unit 125 performs an inverse conversion to the RYeCy space on the image data in the RGB space in which the noise of the color difference information Cb and Cr has been removed, thereby converting it to the actual color space. As a result, an RYeCy image in which the color components are accurately reproduced can be generated. In addition, it is possible to suppress the occurrence of problems such as excessive erasure of a specific color during noise removal.

[0052] Next, in step S342, the image processing unit 125 obtains the difference between the RYeCy image with color noise removed obtained by the inverse conversion in step S341 and the RYeCy image obtained in step S31. As a result, the image processing unit 125 can obtain the color noise component as the difference. The color noise component, which is the difference, is an example of a correction parameter.

[0053] The image processing unit 125 performs step S351 in parallel with the processes of steps S32, S33, S341, and S342. In step S351, the image processing unit 125 uses the RYeCy image to remove noise from the luminance component. Since step S351 is performed on the RYeCy image, the noise removal performance can be improved compared to the case of performing noise removal processing on the RGB image. Note that step S351 may be performed after step S352. Also, if there is little noise in the luminance component, step S351 may be omitted.

[0054] Next, in step S352, the image processing unit 125 subtracts the color noise component acquired in step S342 from the RYeCy image in which the noise of the luminance component has been removed in step S351. Then, the image processing unit 125 generates a RYeCy corrected image, which is a RYeCy image with noise removed, by the subtraction, and ends the process shown in FIG. 6. After this, the image processing unit 125 performs steps S110 and S120 shown in FIG. 5.

[0055] FIG. 7 shows an example of arithmetic expressions for mutually color-converting the RYeCy image and the RGB image. In FIG. 7, two arithmetic expressions (1), (3) for converting the RYeCy image to the RGB image and two arithmetic expressions (2), (4) for converting the RGB image to the RYeCy image are shown.

[0056] For example, the arithmetic expressions (1), (3) are used in step S102, S120 shown in FIG. 5, step S32 shown in FIG. 6, or the process of converting the RYeCy image to the RGB image described later. Also, the arithmetic expressions (2), (4) are used in step S34 shown in FIG. 6, or the process of converting the RGB image to the RYeCy image described later.

[0057] Arithmetic formula (1) shows the general formula for converting a RYeCy image into an RGB image. For example, in the conversion from a RYeCy image to an RGB image, white balance WB (White Balance) and color correction CC (Color Correction) are applied. The values used for white balance WB and color correction CC are dynamically calculated and obtained, for example, so that the color of the image output by AWB (Auto White Balance) processing becomes appropriate.

[0058] Arithmetic formula (2) shows the formula for converting an RGB image into a RYeCy image. Arithmetic formula (2) is the formula defined by the ITU-R (International Telecommunication Union Radiocommunication Sector) standard BT.601. Note that the values included in the formula used for the conversion from an RGB image to a RYeCy image may be other than the values shown in arithmetic formula (2) of FIG. 7.

[0059] In the color conversion method using arithmetic formulas (1) and (2), accurate calculations can be performed compared to the case of using arithmetic formulas (3) and (4), so color conversion can be performed with high accuracy.

[0060] In the simplified formula, since the processing amount decreases compared to the case of using arithmetic formulas (1) and (2), the calculation cost decreases. When arithmetic formula (4) is expanded, it becomes "Y = 0.25R - 0.25R + 0.75Ye - 0.25Ye + 0.5Cy = 0.5Ye + 0.5Cy". Therefore, Y becomes the simple average value of Ye and Cy, and a Y component with less noise is output.

[0061] FIG. 8 shows another example of the processing performed by the image processing unit 125 in FIG. 4. That is, FIG. 8 shows an example of an image processing method by the image processing apparatus 12. For the processing similar to that in FIG. 5, the same reference numerals are given and detailed description thereof is omitted. Steps S110, S120, S130, and S140 in FIG. 8 are the same processing as steps S110, S120, S130, and S140 in FIG. 5, respectively. Steps S10, S20, S30, S40, S50, and S60 in FIG. 8 are the processing corresponding to step S100 in FIG. 5. That is, in this embodiment, a plurality of types of correction processing are sequentially performed.

[0062] In step S10, white balance correction processing is performed, and in step S20, demosaicing processing is performed. In step S30, noise removal processing is performed, and in step S40, color correction processing is performed. In step S50, tone mapping processing is performed, and in step S60, edge enhancement processing is performed.

[0063] The noise removal processing in step S30 is the same as the noise removal processing described in FIG. 6. Note that the image processing unit 125 may perform at least one of the processes in steps S10, S20, S30, S40, S50, and S60. Also, the execution order of steps S10, S20, S30, S40, S50, and S60 may be changed. Further, for the correction processing performed second and later, the RYeCy corrected image generated by the previous image processing is input as the original image data.

[0064] In step S100 shown in FIG. 8, when performing the white balance correction processing shown in step S10, the white balance correction processing included in arithmetic expressions (1) and (3) in FIG. 7 may not be performed in the conversion processing to individual RGB images. For example, the conversion processing to individual RGB images is step S102 in FIG. 5, step S32 in FIG. 6, and the like.

[0065] FIG. 9 shows an example of white balance correction processing, which is one of the correction processes performed in step S10 of FIG. 8. Also, steps S11, S12, S23, S14, and S15 in FIG. 9 respectively correspond to steps S101, S102, S103, S104, and S105 in FIG. 5.

[0066] In the white balance correction process, as shown in the arithmetic expressions (1) and (3) of FIG. 7, color correction is performed by applying gains to R, Ye, and Cy. Assuming the correction parameters are Wr, Wy, and Wc, the image data R', Ye', and Cy' after the white balance correction process can be calculated by "R' = WrR", "Ye' = WyYe", and "Cy' = WcCy" respectively.

[0067] For example, the correction parameters Wr, Wy, and Wc are calculated based on the statistical values of the image by AWB (Auto White Balance) processing. For example, the image processing unit 125 calculates the average value of the pixel values in the area that seems to be achromatic in the image, and obtains the correction parameters Wr, Wy, and Wc so that the average value becomes correctly achromatic.

[0068] In step S11, the image processing unit 125 acquires the RYeCy image output from the imaging device 22 or the RYeCy corrected image corrected in the previous stage. Hereinafter, the RYeCy corrected image before performing the correction process acquired from the previous stage is referred to as the RYeCy image.

[0069] Next, in step S12, the image processing unit 125 converts the RYeCy image into the RGB color space. At this time, the image processing unit 125 may convert it into the RGB color space according to the standard, or may convert it into the RGB color space by a simplified arithmetic expression.

[0070] Next, in step S13, the image processing unit 125 calculates the statistical value (average value) of colors using the RGB color space converted in step S12. Note that since it is difficult to convert to an accurate RGB space for an image before white balance correction processing, the RYeCy image may be converted in advance to the XYZ color space obtained by calibration.

[0071] Next, in step S14, the image processing unit 125 obtains color correction parameters for color correction using the statistical values obtained in step S13. The color correction parameters obtained in step S14 are an example of the first color correction parameters. Next, in step S15, the image processing unit 125 performs white balance correction processing using the color correction parameters obtained in step S14, generates a RYeCy corrected image, and ends the processing shown in FIG. 9.

[0072] By obtaining color correction parameters for white balance correction processing using the image data in the RGB space, it is possible to obtain color correction parameters such that the image displayed on the display device 14 or the like appears to have an average correct color when generating a RYeCy corrected image.

[0073] FIG. 10 shows an example of a demosaicing process, which is one of the correction processes performed in step S20 of FIG. 8. The demosaicing process can be performed without converting the RYeCy image to an RGB image. Therefore, in step S20, only step S21 corresponding to step S101 in FIG. 5 and step S25 corresponding to step S105 in FIG. 5 are performed.

[0074] In step S21, the image processing unit 125 obtains the RYeCy image output from the imaging device 22 or the RYeCy corrected image corrected in the previous stage. Next, in step S25, the image processing unit 125 performs demosaicing processing on the RYeCy image obtained in step S21 to generate a RYeCy corrected image, and ends the processing shown in FIG. 10.

[0075] In the demosaicing process, for example, as shown in FIG. 3, when the R, Ye, and Cy pixels are arranged in a Bayer pattern, the image processing unit 125 performs a process of interpolating the pixel data of the surrounding pixels of the same color and obtaining the pixel data of R, Ye, and Cy for each pixel. The demosaicing process of the image sensor IMGS including the pixels of the RYeYeCy array can be performed using the demosaicing process of the Bayer array image sensor IMGS including the pixels of the RGGB array.

[0076] FIG. 11 shows an example of a color correction process, which is one of the correction processes performed in step S40 of FIG. 8. Steps S41, S42, S43, S44, and S45 in FIG. 11 respectively correspond to steps S101, S102, S103, S104, and S105 in FIG. 5.

[0077] First, in step S41, the image processing unit 125 acquires the RYeCy image output from the imaging device 22 or the RYeCy corrected image corrected in the previous stage. Next, in step S42, the image processing unit 125 converts the RYeCy image into the RGB color space.

[0078] Next, in step S43, the image processing unit 125 determines the position of each pixel value in the RGB space using the RGB color space converted in step S42. Next, in step S44, the image processing unit 125 acquires the color correction parameters to be applied based on the position of each pixel obtained in step S43. The color correction parameters acquired in step S44 are an example of the second color correction parameters.

[0079] Next, in step S45, the image processing unit 125 performs color correction processing on the RYeCy image using the color correction parameters acquired in step S44, generates an RYeCy corrected image, and ends the process shown in FIG. 11. By checking the position of each pixel in the RGB space using the RGB image, color correction parameters that match the visual characteristics can be acquired, and appropriate color correction processing can be performed using the acquired color correction parameters.

[0080] In addition, when setting color correction parameters for the R data, G data, and B data of all pixels, since the amount of data becomes enormous, the data may be decimated in a grid pattern, and the color correction parameters may be obtained only for the grid points. Then, for pixels other than the grid points, for example, the color correction parameters may be obtained by interpolation through linear interpolation processing. Thereby, the amount of data to be processed in the color correction process can be reduced, and the load on the image processing unit 125 can be reduced.

[0081] Also, if the grid-like decimation is performed as R, Ye, Cy, since the processing is performed in a curved space with respect to the RGB space, the intended interpolation processing may not be possible. However, as shown in FIG. 11, after converting the RYeCy space to the RGB space, by decimating the data in a grid pattern, the intended interpolation processing can be performed, and correct color correction parameters can be obtained. Note that, for example, a 3×3 matrix applied to RYeCy is used as the color correction parameter.

[0082] FIG. 12 shows an example of tone mapping processing, which is one of the correction processes performed in step S50 of FIG. 8. Steps S51, S52, S53, and S54 in FIG. 12 respectively correspond to steps S101, S102, S103, and S104 in FIG. 5. Steps S551 and S552 in FIG. 12 correspond to step S105 in FIG. 5.

[0083] First, in step S51, the image processing unit 125 acquires the RYeCy image output from the imaging device 22 or the RYeCy corrected image corrected in the previous stage. Next, in step S52, the image processing unit 125 converts the RYeCy image into an RGB image represented in the RGB color space (YCbCr coordinate system).

[0084] Next, in step S53, the image processing unit 125 detects the color density of each pixel in the RGB image. Next, in step S54, the image processing unit 125 calculates the tone control intensity α based on the detected color density of each pixel. For example, when the tone control process is uniformly applied to the entire image, there may be drawbacks such as the bright areas with dark colors being overcorrected and looking fluorescent. To suppress the occurrence of such drawbacks, based on the color density of each pixel in the RGB image detected in step S53, the tone control intensity α is calculated such that the darker the location, the smaller the value.

[0085] The image processing unit 125 performs step S551 in parallel with the processes of steps S52, S53, and S54. In step S551, the image processing unit 125 corrects the brightness of the RYeCy image using a technique such as tone control. For example, in tone control, a Look Up Table (LUT) is used to correct the brightness of the input pixel value to the brightness of the output pixel value. The graph shown at the bottom of FIG. 12 shows an example of the relationship between the input pixel value and the output pixel value included in the LUT.

[0086] Then, in step S552, the image processing unit 125 performs a blending process of blending the RYeCy image before and after the tone control in step S551 based on the tone control intensity α calculated in step S54, and ends the process shown in FIG. 12. For example, as the blending process, the image processing unit 125 performs an operation of "α * image after tone control+(1 - α) * image before tone control". The symbol "*" indicates multiplication. Thereby, the correction of the darker areas can be weakened, and drawbacks such as the bright areas with dark colors being overcorrected and looking fluorescent can be suppressed.

[0087] By using an RGB image (YCbCr coordinate system) to detect the color density from the chrominance difference information Cb and Cr that is closer to the visual characteristics, the detection accuracy can be improved compared to the case of detecting the color density using a RYeCy image. As a result, the calculation accuracy of the tone control intensity α can be improved, and appropriate blending processing can also be performed even when using a RYeCy image.

[0088] FIG. 13 shows an example of an edge enhancement process which is one of the correction processes performed in step S60 of FIG. 8. Steps S61, S62, S63, S64, and S65 in FIG. 13 respectively correspond to steps S101, S102, S103, S104, and S105 in FIG. 5.

[0089] First, in step S61, the image processing unit 125 acquires a RYeCy image output from the imaging device 22 or a RYeCy corrected image corrected in the previous stage. Next, in step S62, the image processing unit 125 converts the RYeCy image into an RGB image represented in the RGB color space (YCbCr coordinate system).

[0090] Next, in step S63, the image processing unit 125 extracts an edge component from, for example, the luminance values (pixel values) of a predetermined number of adjacent pixels in the RGB image converted in step S62. By using the RGB image (YCbCr coordinate system) to extract the edge component from the luminance information Y, the extraction accuracy of the edge component can be improved compared to the case of extracting the edge component using a RYeCy image. Also, by using the RGB image to extract the edge component from the luminance value, edge enhancement closer to the visual characteristics can be performed.

[0091] Next, in step S64, the image processing unit 125 inverse-transforms the edge components in the RGB space extracted in step S63 into the RYeCy space to obtain the edge components of R, Ye, and Cy. Next, in step S65, the image processing unit 125 adds the edge components of R, Ye, and Cy to the RYeCy image to generate an RYeCy corrected image with enhanced edges, and ends the process shown in FIG. 13. Note that, in order to suppress the enhancement of noise due to edge enhancement, instead of converting to an RGB image with accurate color reproducibility, pseudo Y, Cb, and Cr may be used for edge enhancement.

[0092] As described above, in this embodiment, the RYeCy image is converted into an RGB image with higher color discrimination accuracy than the RYeCy image, and the correction parameters used for correcting the RYeCy image are calculated using the RGB image. By performing the correction process of the RYeCy image with higher luminance compared to the RGB image using the RGB color space, the image quality of the RYeCy corrected image can be improved compared to the case of directly correcting the RYeCy image. That is, it is possible to generate an RYeCy corrected image with excellent image quality in the correction process of the RYeCy image captured through the RYeCy filter.

[0093] Also, for example, by performing inverse transformation of the image data in the RGB space from which the noise of the color difference information Cb and Cr has been removed into the RYeCy space and using it as the correction parameter, an RYeCy image in which the color components are accurately reproduced can be generated.

[0094] In the noise removal process, by using an RGB image (YCbCr coordinate system) and removing the noise of the color difference information Cb and Cr that does not include the luminance information Y and is closer to the visual characteristics, color noise can be effectively removed. As a result, an RYeCy image in which the color components are accurately reproduced can be generated. In addition, it is possible to suppress the occurrence of problems such as excessive erasure of a specific color during noise removal.

[0095] In the white balance correction process, by obtaining color correction parameters using the image data in the RGB space, when generating a RYeCy corrected image, it is possible to obtain color correction parameters such that the image displayed on the display device 14 or the like appears to have an average correct color.

[0096] In the color correction process, by using an RGB image and checking the position of each pixel in the RGB space, it is possible to obtain color correction parameters that match the visual characteristics, and appropriate color correction processing can be performed using the obtained color correction parameters. Also, after converting the RYeCy space to the RGB space, by thinning out the data in a grid pattern, the intended interpolation processing can be performed, and correct color correction parameters can be obtained.

[0097] In the tone mapping process, by using an RGB image (YCbCr coordinate system) and detecting the color density from the color difference information Cb, Cr that is closer to the visual characteristics, the detection accuracy can be improved compared to the case of detecting the color density using a RYeCy image. As a result, the calculation accuracy of the tone control intensity α can be improved, and appropriate blending processing can also be performed even when using a RYeCy image.

[0098] In the edge enhancement process, by using an RGB image (YCbCr coordinate system) and extracting the edge component from the luminance information Y, the extraction accuracy of the edge component can be improved compared to the case of extracting the edge component using a RYeCy image. Also, by using an RGB image and extracting the edge component from the luminance value, edge enhancement closer to the visual characteristics can be performed.

[0099] (Second Embodiment) FIG. 14 shows an example of the processing performed by the image processing unit mounted on the image processing apparatus in the second embodiment. The image processing apparatus and the image processing unit of this embodiment are the same as, for example, the image processing apparatus 12 and the image processing unit 125 shown in FIG. 4. Further, the image processing system including the image processing apparatus 12 of this embodiment has, for example, the same configuration as the image processing system 10 shown in FIG. 1, and includes the image processing apparatus 12, the display device 14, and the information processing apparatus 16, and is mounted on a moving body 20 such as an automobile.

[0100] The processing shown in FIG. 14 is performed by the image processing unit 125 in FIG. 4. Detailed description of the same processing as in FIG. 5 is omitted. The processing in steps S201, S210, S220, S230, and S240 is the same as the processing in steps S101, S110, S120, S130, and S140 in FIG. 5, respectively.

[0101] Also in this embodiment, as in FIG. 5, the image processing unit 125 receives the image data represented in the RYeCy color space output from the imaging device 22 and performs at least one type of correction processing in step S200. However, after step S201, the image processing unit 125 obtains a statistical value using the image data represented in the RYeCy color space in step S203. Further, after step S203, the image processing unit 125 calculates a statistical value (statistic), such as a signal value representing the intensity of each pixel of R, Ye, and Cy, for the entire range or a predetermined partial range of the RYeCy image in step S204. That is, the image processing unit 125 calculates the correction parameter without generating the image data represented in the RGB color space.

[0102] Then, after step S204, the image processing unit 125 performs the correction processing of the RYeCy image obtained in step S201 using the correction parameter calculated in step S204, and generates correction image data (RYeCy corrected image) represented in the RYeCy color space. After that, as in FIG. 5, the image processing unit 125 performs the processing in steps S210, S220, S230, and S240.

[0103] As described above, in this embodiment, since the image processing unit 125 does not perform the conversion to the RGB image for calculating the correction parameters, the processing load on the image processing unit 125 can be reduced as compared with FIG. 5.

[0104] (Third Embodiment) FIGS. 15 and 16 show an example of the processing performed by the image processing unit mounted on the image processing apparatus in the third embodiment. The image processing apparatus and the image processing unit of this embodiment are the same as, for example, the image processing apparatus 12 and the image processing unit 125 shown in FIG. 4. Further, the image processing system including the image processing apparatus 12 of this embodiment has, for example, the same configuration as the image processing system 10 shown in FIG. 1, and includes the image processing apparatus 12, the display device 14, and the information processing apparatus 16, and is mounted on a moving body 20 such as an automobile.

[0105] The processing shown in FIGS. 15 and 16 is performed by the image processing unit 125 in FIG. 4. Detailed description of the same processing as in FIG. 8 is omitted. The processing in steps S10, S20, S30, S40, S50, and S60 is the same as the processing in steps S10, S20, S30, S40, S50, and S60 in FIG. 8, respectively. Further, the processing in steps S110, S120, S130, and S140 is the same as the processing in steps S110, S120, S130, and S140 in FIG. 8, respectively.

[0106] In this embodiment, before each of the processes in steps S10, S20, S30, S40, S50, and S60, the image processing unit 125 determines whether or not to perform each process, and performs only the processes determined to be performed. For example, the image processing unit 125 determines whether or not to perform each process based on the value of a register or the value of a flag provided in the image processing apparatus 12 corresponding to each process.

[0107] In step S8 of FIG. 15, the image processing unit 125 performs the white balance processing in step S10 only when it determines to perform the white balance processing. The white balance processing in step S10 is the same as the white balance processing in FIG. 9. In step S18, the image processing unit 125 performs the demosaicing processing in step S20 only when it determines to perform the demosaicing processing. The demosaicing processing in step S20 is the same as the demosaicing processing in FIG. 10.

[0108] In step S28, the image processing unit 125 performs the noise removal processing in step S30 only when it determines to perform the noise removal processing. The noise removal processing in step S30 is the same as the noise removal processing in FIG. 6. In step S38, the image processing unit 125 performs the color correction processing in step S40 only when it determines to perform the color correction processing. The color correction processing in step S40 is the same as the color correction processing in FIG. 11.

[0109] In step S48 of FIG. 16, the image processing unit 125 performs the tone mapping processing in step S50 only when it determines to perform the tone mapping processing. The tone mapping processing in step S50 is the same as the tone mapping processing in FIG. 12. In step S58, the image processing unit 125 performs the edge enhancement processing in step S60 only when it determines to perform the edge enhancement processing. The edge enhancement processing in step S60 is the same as the edge enhancement processing in FIG. 13.

[0110] As described above, even in this embodiment, the same effects as those of the above-described embodiments can be obtained. Further, in this embodiment, arbitrary combinations of processes can be freely performed based on register values, flag values, or the like. Therefore, regardless of the type of process to be performed, the hardware of the image processing unit 125 can be made common. For example, even when the image processing unit 125 is designed as an ASIC (Application Specific Integrated Circuit), arbitrary combinations of processes can be freely performed by one ASIC. Since the image processing unit 125 can be made common, the cost of the image processing unit 125 in which the image processing unit 125 is mounted can be reduced, and the cost of the image processing system 10 can be reduced.

[0111] As described above, the present invention has been described based on each embodiment. However, the present invention is not limited to the requirements shown in the above embodiments. Regarding these points, changes can be made without departing from the gist of the present invention, and can be appropriately determined according to the application form.

Explanation of Reference Numerals

[0112] 10 Image processing system 12 Image processing apparatus 14 Display device 16 Information processing apparatus 20 Moving body 22, 22A, 22B, 22C, 22D Imaging device 30 Semiconductor device 32 Semiconductor chip 34 Wiring board 36 Wiring 121 Interface unit 122 CPU 123 Main storage device 124 Auxiliary storage device 125 Image processing unit BP1, BP2 Bump BUS Bus IMGS Image sensor PX Pixel

Claims

1. An image processing apparatus that performs correction processing on original image data generated by an image sensor that receives light with a plurality of pixels through a color filter including segments of a red-based color and at least one complementary color, a conversion process of converting the original image data into original color system image data represented in an original color system color space, a statistical value acquisition process of acquiring statistical values of a plurality of pixel data corresponding to the plurality of pixels in the original color system image data, a parameter calculation process of calculating correction parameters using the statistical values, a correction process of correcting the original image data based on the correction parameters to generate corrected image data; An image processing apparatus that performs the above.

2. In the correction process, after converting the correction parameters into correction parameters represented in the color space of the original image data, the original image data is corrected using the converted correction parameters The image processing apparatus according to claim 1.

3. Performing a plurality of types of correction processes sequentially, and for the correction processes performed second and later, the corrected image data generated by the previous image process is input as the original image data The image processing apparatus according to claim 1 or claim 2.

4. Converting the corrected image data generated in the last correction process into original color system image data represented in an original color system color space, Outputting the converted original color system image data as display data, Outputting the corrected image data generated in the last correction process as image processing data The image processing apparatus according to claim 3.

5. One of the plurality of types of correction processes is a noise removal process, In the conversion process, the original image data is converted into original color system image data represented by luminance information and color difference information, In the statistical value acquisition process, by performing noise removal on the color difference information of the original color system image data, original color system image data with color noise removed is generated, In the parameter calculation process, the original color system image data with color noise removed is converted into the color space of the original image data, and the difference from the original image data is obtained to acquire the color noise in the color space of the original image data, In the correction process, subtracting the color noise in the color space of the original image data from the original image data The image processing apparatus according to claim 3 or claim 4.

6. One of the plurality of types of correction processes is a white balance correction process, In the statistical value acquisition process, calculating the color statistical values of the original color system image data obtained in the conversion process, In the parameter calculation process, a first color correction parameter is obtained using the statistical value, In the correction process, the white balance of the original image data is corrected using the first color correction parameter The image processing apparatus according to any one of claims 3 to 5.

7. One of the plurality of types of correction processes is a color correction process, In the statistical value acquisition process, using the original color system image data obtained by the conversion process, the positions of the pixel values of the plurality of pixels in the original color system color space are obtained, In the parameter calculation process, a second color correction parameter is obtained based on the positions of the pixel values of the plurality of pixels in the original color system color space, In the correction process, the color of the original image data is corrected using the second color correction parameter The image processing apparatus according to any one of claims 3 to 6.

8. One of the plurality of types of correction processes is a tone mapping process, In the conversion process, the original image data is converted into original color system image data represented by luminance information and color difference information, In the statistical value acquisition process, the color density of the plurality of pixels in the original color system image data is detected, In the parameter calculation process, based on the detected color density of each pixel, a tone control intensity is calculated, In the correction process, tone control of the original image data is performed, and a blending process of blending the original image data and the original image data after the tone control is performed based on the tone control intensity The image processing apparatus according to any one of claims 3 to 7.

9. One of the plurality of types of correction processes is an edge enhancement process, In the conversion process, the original image data is converted into original color system image data represented by luminance information and color difference information, In the statistical value acquisition process, edge components are detected from the pixel values of the plurality of pixels in the original color system image data, In the parameter calculation process, the edge component of the original color system image data is converted into the color space of the original image data to obtain the edge component of the original image data, In the correction process, the edge component in the color space of the original image data is added to the original image data The image processing apparatus according to any one of claims 3 to 8.

10. Perform correction processing on the original image data generated by the image sensor that receives light through the color filter including red, yellow, and cyan segments The image processing apparatus according to any one of claims 1 to 9

11. Perform correction processing on the original image data generated by the image sensor that receives light through the color filter including magenta, yellow, and cyan segments The image processing apparatus according to any one of claims 1 to 9

12. An image processing method for an image processing apparatus that performs correction processing on original image data generated by an image sensor that receives light with a plurality of pixels through a color filter including a red-based color and segments of at least one complementary color, comprising: Performing a conversion process of converting the original image data into original color system image data represented in an original color system color space; Performing a statistical acquisition process of acquiring statistical values of a plurality of pixel data corresponding to the plurality of pixels in the original color system image data; Performing a parameter calculation process of calculating correction parameters using the statistical values; Performing a correction process of correcting the original image data based on the correction parameters to generate corrected image data Image processing method

Images