IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, AND IMAGE PROCESSING PROGRAM

Abstract

To provide an image processing apparatus which can perform optimized image processing in accordance with color components, an image processing method, and an image processing program.SOLUTION: An image processing apparatus 3 determines whether to execute first interpolation processing by means of a Gr / Gb component interpolation processing unit 240 which interpolates first and second color components as the same color component in one map, or to execute second interpolation processing by means of a Gr component interpolation processing unit 220 and a Gb component interpolation processing unit 230 which interpolate the first and second color components as different color components in multiple maps. The first and second color components are a Gr component and a Gb component out of G components for reducing Bayer data including R, G, and B components, for example.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The disclosure of this specification relates to an image processing device, an image processing method, and an image processing program. [Background technology]

[0002] For example, Patent Document 1 describes a video compression device for a single frame image of a dual green system having a Bayer array sampling structure. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-63289 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional image processing devices including those in Patent Document 1 have the following features: Corresponding to the G component There is room for improvement in terms of providing optimized image processing.

[0005] In light of the above-mentioned circumstances, an object of one aspect of the present invention is to Corresponding to the G component An object of the present invention is to provide an image processing device, an image processing method, and an image processing program capable of performing optimized image processing. [Means for solving the problem]

[0006] The image processing device according to the first aspect of the present invention determines whether to execute a first interpolation process for interpolating G components in Bayer data using one map, or a second interpolation process for interpolating G components in a row in which R components are arranged and G components in a row in which B components are arranged using separate maps, based on a reduction ratio when reducing the Bayer data. to the value of A control means is provided to determine the control means determines to execute the first interpolation process when the reduction rate is greater than a threshold value as a case where the degree of reduction is small, and determines to execute the second interpolation process when the reduction rate is less than the threshold value as a case where the degree of reduction is large. It is characterized by: Further, in an image processing device according to a second aspect of the present invention, when interpolating pixels for each color component based on Bayer data, the image processing device determines whether to separate the G component into a Gr component as the G component of a row in which an R component is arranged and a Gb component as the G component of a row in which a B component is arranged and to perform pixel interpolation separately for the Gr component and the Gb component, or to perform pixel interpolation for the G component without separating the G component into the Gr component and the Gb component, based on a reduction ratio when reducing the Bayer data. Value of A control means for determining the control means determines, when the degree of reduction is small and the value of the reduction rate is greater than a threshold, to perform pixel interpolation for the G component without separating the G component into the Gr component and the Gb component, and determines, when the degree of reduction is large and the value of the reduction rate is less than the threshold, to separate the G component into the Gr component and the Gb component and to perform pixel interpolation separately for the Gr component and the Gb component. It is characterized by: Further, an image processing method according to a first aspect of the present invention is an image processing method executed by an image processing device, in which a first interpolation process for interpolating G components in Bayer data using one map, or a second interpolation process for interpolating G components in a row in which R components are arranged and G components in a row in which B components are arranged using separate maps, is determined based on a reduction ratio when reducing the Bayer data. Value of Decide based on a control process that determines to execute the first interpolation process when the reduction rate is greater than a threshold value as a case where the degree of reduction is small, and determines to execute the second interpolation process when the reduction rate is less than the threshold value as a case where the degree of reduction is large. It is characterized by: Further, an image processing method according to a second aspect of the present invention is an image processing method executed by an image processing device, and when interpolating pixels for each color component based on Bayer data, a decision is made as to whether to separate the G component into a Gr component as the G component of a row in which an R component is arranged and a Gb component as the G component of a row in which a B component is arranged and to perform pixel interpolation separately for the Gr component and the Gb component, or to perform pixel interpolation for the G component without separating the G component into the Gr component and the Gb component, based on a reduction ratio when reducing the Bayer data. Value of Control processing that makes decisions based on the control process determines, when the degree of reduction is small and the value of the reduction rate is greater than a threshold, to perform pixel interpolation for the G component without separating the G component into the Gr component and the Gb component, and determines, when the degree of reduction is large and the value of the reduction rate is less than the threshold, to separate the G component into the Gr component and the Gb component and to perform pixel interpolation separately for the Gr component and the Gb component. It is characterized by: Further, the image processing program according to the first aspect of the present invention causes a computer to select which of a first interpolation process, which interpolates the G component in the Bayer data with one map, and a second interpolation process, which interpolates the G component in the row in which the R component is arranged and a G component in the row in which the B component is arranged, using separate maps, based on a reduction ratio when reducing the Bayer data. Value of It acts as a control means to make decisions based on and when the degree of reduction is small and the value of the reduction ratio is greater than a threshold value, the control means determines to execute the first interpolation process, and when the degree of reduction is large and the value of the reduction ratio is less than the threshold value, the control means determines to execute the second interpolation process.It is characterized by: Further, an image processing program according to a second aspect of the present invention causes a computer to, when interpolating pixels for each color component based on Bayer data, determine whether to separate a G component into a Gr component as the G component of a row in which an R component is arranged and a Gb component as the G component of a row in which a B component is arranged and to perform pixel interpolation separately for the Gr component and the Gb component, or to perform pixel interpolation for the G component without separating the G component into the Gr component and the Gb component, based on a reduction ratio when reducing the Bayer data. Value of It acts as a control means to make decisions based on and when the degree of reduction is small and the value of the reduction rate is greater than a threshold value, the control means determines to perform pixel interpolation for the G component without separating the G component into the Gr component and the Gb component, and when the degree of reduction is large and the value of the reduction rate is less than the threshold value, the control means determines to separate the G component into the Gr component and the Gb component and perform pixel interpolation separately for the Gr component and the Gb component. It is characterized by: Effect of the Invention

[0007] According to the above aspect, Corresponding to the G component It is possible to provide an image processing device, an image processing method, and an image processing program capable of performing optimized image processing. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is an external view showing an example of a digital camera 1X. [Diagram 2] FIG. 2 is a diagram showing an example of a functional configuration block of a digital camera 1X. [Diagram 3] 2 is a functional block diagram showing an example of the internal configuration of a RAW image reducing unit 3. FIG. [Figure 4] FIG. 2 is a diagram illustrating an example of division processing by a data division unit 100. [Diagram 5] 13 is a diagram showing an example of the interpolation process of the R component divided image D2R by the R component interpolation processing unit 210. FIG. [Figure 6] 13 is a diagram showing an example of the interpolation process of the Gr component divided image D2Gr by the Gr component interpolation processing unit 220. FIG. [Figure 7] 13 is a diagram showing an example of the interpolation process of the Gb component divided image D2Gb by the Gb component interpolation processor 230. FIG. [Figure 8]13 is a diagram showing an example of the interpolation process of the Gr component divided image D2Gr and the Gb component divided image D2Gb by the Gr·Gb component interpolation processing unit 240. FIG. [Figure 9] 13 is a diagram showing an example of the interpolation process of the B component divided image D2B by the B component interpolation processing unit 250. FIG. [Figure 10] FIG. 13 is a diagram illustrating an example of a reduction process when the reduction ratio is 1 / 2. [Figure 11] FIG. 13 is a diagram showing an example of a reduction process when the reduction ratio is 3 / 4. [Figure 12] FIG. 13 is a diagram illustrating an example of resampling processing when the reduction ratio is 1 / 2. [Figure 13] FIG. 13 is a diagram illustrating an example of resampling processing when the reduction ratio is 3 / 4. [Figure 14] FIG. 13 is a diagram showing an example of a combining process when the reduction ratio is 1 / 2. [Figure 15] FIG. 13 is a diagram showing an example of a combining process when the reduction ratio is 3 / 4. [Figure 16] FIG. 11 is a first diagram showing an example of the effect of image processing according to the present embodiment. [Figure 17] FIG. 11 is a second diagram showing an example of the effect of the image processing according to the present embodiment. [Figure 18] 13 is a functional block diagram showing a modified example of the internal configuration of the RAW image reducing section 3. FIG. [Figure 19] 13 is a functional block diagram showing another modified example of the internal configuration of the RAW image reducing section 3. FIG. [Figure 20] 5 is a flowchart illustrating an example of image processing according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The image processing device of this embodiment will be described in detail with reference to Figures 1 to 20. Note that the image processing method and image processing program of this embodiment are realized by causing a computer (CPU: Central Processing Unit) of the image processing device to execute various processing steps.

[0010] The image processing device may be mounted on a fixed photographing device such as a security camera or a surveillance camera, or may be mounted on a portable photographing device such as a digital camera or various terminals. The image processing device may be specialized for performing image processing on images transmitted by wire or wirelessly from a photographing device or other device, or images stored in various storage media. In other words, the configuration of the image processing device has a degree of freedom, and various design changes are possible.

[0011] FIG. 1 is an external view showing an example of a digital camera (photographing device) 1X. In the digital camera 1X, light from a subject is guided into a camera body 11X via a lens 10X, reflected by a mirror 12X, and a real image of the subject is formed on a focusing glass 13X. This real image of the subject is guided to a viewfinder 15X by a pentaprism 14X so that it can be observed. A display device 16X is disposed between the focusing glass 13X and the pentaprism 14X, and displays various information superimposed on the subject image reflected on the focusing glass 13X. The broken lines in FIG. 1 indicate the optical path of the light from the subject that is guided to the viewfinder 15X. In the digital camera 1X, the mirror 12X is raised, the shutter 17X is opened, and the subject light is guided to an image sensor 18X to perform photography, so that the subject image does not reach the display device 16X in the mirror-up state.

[0012] The digital camera 1X may incorporate an image processing device according to an embodiment of the present invention that performs image processing on an image captured by the image sensor 18X, or may have a communication unit (such as a connection terminal or an antenna) that transmits the image captured by the image sensor 18X to an external image processing device according to an embodiment of the present invention by wire or wirelessly. That is, the present invention is not limited to the digital camera 1X having a photographing function, but can also be applied to any device that can obtain an image captured by an external device and apply image processing to it. In other words, the present invention can also be applied to a system composed of multiple devices.

[0013] FIG. 2 is a diagram showing an example of a functional block diagram of a digital camera (photographing device) 1X. The digital camera 1X has a RAW image correction processing section 2 and a RAW image reduction section 3. The RAW image correction processing section 2 performs correction processing (for example, pre-processing such as scratch processing) on ​​a photographed image D1 by an image sensor 18X to produce a RAW image D2, and outputs the RAW image D2 to the RAW image reduction section 3. The RAW image D2 is, for example, Bayer data including each of an R component, a G component (Gr component, Gb component), and a B component. Bayer data means image data in a Bayer array. The RAW image reduction section 3 performs division processing, interpolation processing, reduction processing, resampling processing, and combination processing on the RAW image D2 from the RAW image correction processing section 2, thereby outputting a reduced RAW image D3. The reduced RAW image D3 is also, for example, Bayer data including each of an R component, a G component (Gr component, Gb component), and a B component. The RAW image reducing section 3 constitutes the image processing device of this embodiment, and the details of the RAW image reducing section (image processing device) 3 will be described later.

[0014] The digital camera 1X has a main image generating section 4, a CPU (Central Processing Unit) 5, a key IF section 6, a non-volatile memory I / F section 7, and an external high-speed memory I / F section 8. The RAW image reducing section 3, the main image generating section 4, the CPU 5, the key IF section 6, the non-volatile memory I / F section 7, and the external high-speed memory I / F section 8 are connected to each other by a bus 9.

[0015] The main image generating unit 4 generates a main image based on the reduced RAW image D3 by the RAW image reducing unit 3. The CPU 5 controls the internal components of the digital camera 1X including the RAW image reducing unit 3, the main image generating unit 4, the Key IF unit 6, the non-volatile memory I / F unit 7, and the external high-speed memory I / F unit 8. The Key IF unit 6 executes operation control (operation restriction) according to the Key 6A provided from an external device. The non-volatile memory I / F unit 7 controls writing of various data (e.g., the reduced RAW image D3 by the RAW image reducing unit 3 and the main image by the main image generating unit 4) to the non-volatile memory (SD card) 7A inserted as an external storage medium. The external high-speed memory I / F unit 8 controls writing of various data (e.g., the reduced RAW image D3 by the RAW image reducing unit 3 and the main image by the main image generating unit 4) to the external high-speed memory (DDR memory) 8A.

[0016] 3 is a functional block diagram showing an example of the internal configuration of the RAW image reduction unit (image processing device) 3. As described above, the RAW image reduction unit 3 receives the RAW image D2 from the RAW image correction processing unit 2, and outputs a reduced RAW image D3 obtained by performing division processing, interpolation processing, reduction processing, resampling processing, and combination processing on the RAW image D2 from the RAW image reduction unit 3. Both the RAW image D2 and the reduced RAW image D3 are Bayer data including, for example, an R component, a G component (Gr component, Gb component), and a B component.

[0017] The RAW image reducing unit 3 includes a data dividing unit 100 , an interpolation processing unit 200 , a reduction processing unit 300 , a resampling processing unit 400 , and a data combining unit 500 .

[0018] The data dividing unit 100 divides the RAW image (Bayer data including R, G (Gr, Gb) and B components) D2 from the RAW image correction processing unit 2 into pixel maps for each color component.

[0019] Fig. 4 is a diagram showing an example of the division process by the data division unit 100. As shown in Fig. 4, the RAW image D2 is composed of a total of 16 pixels, including four pixels R00, R02, R20, and R22 of the R component, four pixels Gr01, Gr03, Gr21, and Gr23 of the Gr component, four pixels Gb10, Gb12, Gb30, and Gb32 of the Gb component, and four pixels B11, B13, B31, and B33 of the B component, arranged in a Bayer array.

[0020] The data division unit 100 extracts four pixels R00, R02, R20, and R22 of the R component from the RAW image D2 and divides them into an R component divided image D2R, extracts four pixels Gr01, Gr03, Gr21, and Gr23 of the Gr component and divides them into a Gr component divided image D2Gr, extracts four pixels Gb10, Gb12, Gb30, and Gb32 of the Gb component and divides them into a Gb component divided image D2Gb, and extracts four pixels B11, B13, B31, and B33 of the B component and divides them into a B component divided image D2B.

[0021] The interpolation processing section 200 has an R component interpolation processing section 210 , a Gr component interpolation processing section 220 , a Gb component interpolation processing section 230 , a Gr·Gb component interpolation processing section 240 , and a B component interpolation processing section 250 .

[0022] The RAW image reducing unit 3 has a reduction ratio threshold comparing unit 600, and the input path of the divided image from the data dividing unit 100 to the interpolation processing unit 200 varies depending on the comparison result of the reduction ratio threshold comparing unit 600. The reduction ratio threshold comparing unit 600 compares the reduction ratio A with a threshold S to determine which is larger, and sends switch changeover signals W1, W2 to the switches S1, S2 depending on the comparison result.

[0023] More specifically, when the reduction ratio A is greater than the threshold S, the switches S1 and S2 are switched to the dashed line position in Fig. 3, and the Gr component split image D2Gr and the Gb component split image D2Gb are controlled to be input to the Gr·Gb component interpolation processing unit 240. On the other hand, when the reduction ratio A is smaller than the threshold S, the switches S1 and S2 are switched to the solid line position in Fig. 3, and the Gr component split image D2Gr is input to the Gr component interpolation processing unit 220, and the Gb component split image D2Gb is input to the Gb component interpolation processing unit 230. Note that when the reduction ratio A and the threshold S are the same value, either of the above controls may be performed.

[0024] Furthermore, regardless of the magnitude relationship between the reduction ratio A and the threshold value S (regardless of the comparison result of the reduction ratio threshold comparator 600), the R component split image D2R is input to the R component interpolation processor 210, and the B component split image D2B is input to the B component interpolation processor 250.

[0025] Here, the reduction ratio A may be determined by manual setting by the user, or may be automatically determined based on a predetermined shooting situation (e.g., subject, brightness, etc.). The threshold value S may be set as a fixed value at the time of device manufacture, may be determined by manual setting by the user, or may be automatically determined based on a predetermined shooting situation (e.g., subject, brightness, etc.). For example, in an aspect in which at least one of the reduction ratio A and the threshold value S is automatically determined, it is possible to automatically determine the input path of the Gr component divided image D2Gr and the Gb component divided image D2Gb, and further whether a first interpolation process or a second interpolation process described later is to be executed, without receiving an instruction from the user during shooting with the digital camera (photographing device) 1X.

[0026] For example, in an embodiment of a security or surveillance camera, when no abnormality is detected (under normal circumstances), memory usage can be saved by executing the second interpolation process (interpolation process when the image size after reduction is smaller than when the first interpolation process is executed), and when an abnormality is detected, the process can automatically switch to the first interpolation process (interpolation process when the image size after reduction is larger than when the second interpolation process is executed).

[0027] Hereinafter, as an example, a case where the threshold value S is greater than 1 / 2 and less than 3 / 4 (1 / 2 < S < 3 / 4) will be described. In this case, for example, if the reduction rate A is 1 / 2, the reduction rate A is smaller than the threshold value S, and the second interpolation process described later is executed. On the other hand, for example, if the reduction rate A is 3 / 4, the reduction rate A is greater than the threshold value S, and the first interpolation process described later is executed.

[0028] The R - component interpolation processing unit 210 generates an interpolated - processed image H2R by performing an interpolation process on the R - component divided image D2R from the data division unit 100.

[0029] FIG. 5 is a diagram showing an example of the interpolation process of the R - component divided image D2R by the R - component interpolation processing unit 210. In FIG. 5, the 16 pixels R00, R01, R02, R03, R10, R11, R12, R13, R20, R21, R22, R23, R30, R31, R32, R33 of the R - component after interpolation are calculated by the following formulas based on the 4 pixels R00, R02, R20, R22 of the R - component of the R - component divided image D2R before interpolation. [After interpolation] [Before interpolation] R00 = R00 R01 = (R00 + R02) / 2 R02 = R02 R03 = (R02 + R04) / 2 R10 = (R00 + R20) / 2 R11 = (R00 + R02 + R20 + R22) / 4 R12 = (R02 + R22) / 2 R13 = (R02 + R04 + R22 + R24) / 4 R20 = R20 R21 = (R20 + R22) / 2 R22 = R22 R23 = (R22 + R24) / 2 R30 = (R20 + R40) / 2 R31 = (R20 + R22 + R40 + R42) / 4 R32 = (R22 + R42) / 2 R33 = (R22 + R24 + R42 + R44) / 4

[0030] The Gr component interpolation processing unit 220 performs an interpolation process on the Gr component divided image D2Gr from the data dividing unit 100, thereby generating a post-interpolation processed image H2Gr.

[0031] Fig. 6 is a diagram showing an example of the interpolation process of the Gr component divided image D2Gr by the Gr component interpolation processor 220. In Fig. 6, 16 pixels Gr00, Gr01, Gr02, Gr03, Gr10, Gr11, Gr12, Gr13, Gr20, Gr21, Gr22, Gr23, Gr30, Gr31, Gr32, and Gr33 of the Gr component after interpolation are calculated by the following formula based on four pixels Gr01, Gr03, Gr21, and Gr23 of the Gr component of the Gr component divided image D2Gr before interpolation. [After interpolation] [Before interpolation] Gr00 = (Gr01+Gr01) / 2 Gr01 = Gr01 Gr02 = (Gr01+Gr03) / 2 Gr03 = Gr03 Gr10 = (Gr01+Gr01+Gr21+Gr21) / 4 Gr11 = (Gr01+Gr21) / 2 Gr12 = (Gr01+Gr03+Gr21+Gr23) / 4 Gr13 = (Gr03+Gr23) / 2 Gr20 = (Gr21+Gr21) / 2 Gr21 = Gr21 Gr22 = (Gr21+Gr23) / 2 Gr23 = Gr23 Gr30 = (Gr21+Gr21+Gr41+Gr41) / 4 Gr31 = (Gr21+Gr41) / 2 Gr32 = (Gr21+Gr23+Gr41+Gr43) / 4 Gr33 = (Gr23+Gr43) / 2

[0032] The Gb component interpolation processing unit 230 performs interpolation processing on the Gb component divided image D2Gb from the data dividing unit 100 to generate a post-interpolation processing image H2Gb.

[0033] Fig. 7 is a diagram showing an example of the interpolation process of the Gb component divided image D2Gb by the Gb component interpolation processor 230. In Fig. 7, 16 pixels Gb00, Gb01, Gb02, Gb03, Gb10, Gb11, Gb12, Gb13, Gb20, Gb21, Gb22, Gb23, Gb30, Gb31, Gb32, and Gb33 of the Gb component after interpolation are calculated by the following formula based on the four pixels Gb10, Gb12, Gb30, and Gb32 of the Gb component of the Gb component divided image D2Gb before interpolation. [After interpolation] [Before interpolation] Gb00 = (Gb10+Gb10) / 2 Gb01 = (Gb10+Gb12+Gb10+Gb12) / 4 Gb02 = (Gb12+Gb12) / 2 Gb03 = (Gb12+Gb14+Gb12+Gb14) / 4 Gb10 = Gb10 Gb11 = (Gb10+Gb12) / 2 Gb12 = Gb12 Gb13 = (Gb12+Gb14) / 2 Gb20 = (Gb10+Gb30) / 2 Gb21 = (Gb10+Gb12+Gb30+Gb32) / 4 Gb22 = (Gb12+Gb32) / 2 Gb23 = (Gb12+Gb14+Gb32+Gb34) / 4 Gb30 = Gb30 Gb31 = (Gb30+Gb32) / 2 Gb32 = Gb32 Gb33 = (Gb32+Gb34) / 2

[0034] The Gr / Gb component interpolation processing unit 240 performs an interpolation process on the Gr component divided image D2Gr and the Gb component divided image D2Gb from the data dividing unit 100 to generate a post-interpolation image H2GrGb.

[0035] Fig. 8 is a diagram showing an example of the interpolation process of the Gr component divided image D2Gr and the Gb component divided image D2Gb by the Gr·Gb component interpolation processor 240. In Fig. 8, 16 pixels G00, G01, G02, G03, G10, G11, G12, G13, G20, G21, G22, G23, G30, G31, G32, and G33 of the G component (GrGb component) after interpolation are calculated by the following formula based on four pixels Gr01, Gr03, Gr21, and Gr23 of the Gr component of the Gr component divided image D2Gr before interpolation and four pixels Gb10, Gb12, Gb30, and Gb32 of the Gb component of the Gb component divided image D2Gb before interpolation. [After interpolation] [Before interpolation] G00 = (Gr01+Gr01) / 2 G01 = Gr01 G02 = (Gr01+Gr03) / 2 G03 = Gr03 G10 = Gb10 G11 = (Gb10+Gb12) / 2 G12 = Gb12 G13 = (Gb12+Gb14) / 2 G20 = (Gr21+Gr21) / 2 G21 = Gr21 G22 = (Gr21+Gr23) / 2 G23 = Gr23 G30 = Gb30 G31 = (Gb30+Gb32) / 2 G32 = Gb32 G33 = (Gb32+Gb34) / 2

[0036] In the above example, the Gr·Gb component interpolation process is performed on the pixels to the left and right of the interpolated pixel, but it may be performed on the pixels above, below, left, or right. For example, in the above example, G11=(Gb10+Gb12) / 2, but it may be G11=(Gr01+Gr21+Gb10+Gb12) / 4.

[0037] The B component interpolation processing unit 250 performs an interpolation process on the B component divided image D2B from the data dividing unit 100 to generate a post-interpolation processed image H2B.

[0038] Fig. 9 is a diagram showing an example of the interpolation process of the B component divided image D2B by the B component interpolation processor 250. In Fig. 9, 16 pixels B00, B01, B02, B03, B10, B11, B12, B13, B20, B21, B22, B23, B30, B31, B32, and B33 of the B component after interpolation are calculated by the following formula based on four pixels B11, B13, B31, and B33 of the B component of the B component divided image D2B before interpolation. [After interpolation] [Before interpolation] B00 = (B11+B11+B11+B11) / 4 B01 = (B11+B11) / 2 B02 = (B11+B13+B11+B13) / 4 B03 = (B13+B13) / 2 B10 = (B11+B11) / 2 B11 = B11 B12 = (B11+B13) / 2 B13 = B13 B20 = (B11+B11+B31+B31) / 4 B21 = (B11+B31) / 2 B22 = (B11+B13+B31+B33) / 4 B23 = (B13+B33) / 2 B30 = (B31+B31) / 2 B31 = B31 B32 = (B31+B33) / 2 B33 = B33

[0039] In addition, in the interpolation process, there may be cases where pixel components are not present around the interpolated pixel at the end of the input image. In this case, pixel components of other peripheral parts of the interpolated pixel, or pixel components adjacent to the part where no pixel components are present, may be used. For example, when there is no pixel to the left of the interpolated pixel, the pixel to the right of the interpolated pixel may be used; when there is no pixel to the right of the interpolated pixel, the pixel to the left of the interpolated pixel may be used; when there is no pixel above the interpolated pixel, the pixel below the interpolated pixel may be used; when there is no pixel below the interpolated pixel, the pixel above the interpolated pixel may be used; when there is no pixel to the top left of the interpolated pixel, the pixel to the top right of the interpolated pixel may be used; when there is no pixel to the bottom left of the interpolated pixel, the pixel to the bottom right of the interpolated pixel may be used; when there is no pixel to the top left of the interpolated pixel, the pixel to the bottom left of the interpolated pixel may be used; when there is no pixel to the top right of the interpolated pixel, the pixel to the bottom right of the interpolated pixel may be used. Furthermore, when there are no pixels to the top left, top right, or bottom left of the interpolated pixel, the pixel to the bottom right of the interpolated pixel may be used.

[0040] In this manner, in the image processing of this embodiment, when one of the Gr component and the Gb component of the G component and the other one are defined as "first and second color components", it is determined whether to execute a "first interpolation process (see FIG. 8)" in which the first and second color components are interpolated in one (common) map as the same color component, or a "second interpolation process (see FIG. 6 and FIG. 7)" in which the first and second color components are interpolated in multiple (separate) maps as different color components. That is, by executing either the first interpolation process or the second interpolation process, reduced data of Bayer data including the first and second color components is generated. In addition, in this embodiment, it is determined whether to execute the first interpolation process or the second interpolation process according to the reduction ratio A. Specifically, when the reduction ratio A is greater than the threshold value S, the first interpolation process is executed, and when the reduction ratio A is less than the threshold value S, the second interpolation process is executed. This allows image processing optimized according to the color components to be executed.

[0041] Returning to FIG. 3, the reduction processing unit 300 has an R component reduction processing unit 310, a Gr component reduction processing unit 320, a Gb component reduction processing unit 330, a Gr·Gb component reduction processing unit 340, and a B component reduction processing unit 350.

[0042] The R component reduction processing unit 310, the Gr component reduction processing unit 320, the Gb component reduction processing unit 330, the Gr·Gb component reduction processing unit 340, and the B component reduction processing unit 350 perform reduction processing in accordance with the reduction ratio A. For example, the R component reduction processing unit 310, the Gr component reduction processing unit 320, the Gb component reduction processing unit 330, the Gr·Gb component reduction processing unit 340, and the B component reduction processing unit 350 perform reduction processing using an averaging operation method based on the area ratio of pixels after reduction.

[0043] 10A to 10E are diagrams showing an example of the reduction process when the reduction ratio is 1 / 2.

[0044] Fig. 10A shows the reduction process of the interpolated image H2R by the R component reduction processing unit 310. In Fig. 10A, a map of 16 pixels (4 x 4) is reduced to a map of 4 pixels (2 x 2). In Fig. 10A, the four pixels R00, R01, R10, and R11 of the R component after reduction are calculated by the following formula based on the 16 pixels R00, R01, R02, R03, R10, R11, R12, R13, R20, R21, R22, R23, R30, R31, R32, and R33 of the R component of the interpolated image H2R before reduction. An image composed of the four pixels R00, R01, R10, and R11 of the R component after reduction is called a reduced image S2R. [After reduction] [Before reduction] R00 = (R00+R01+R10+R11) / 4 R01 = (R02+R03+R12+R13) / 4 R10 = (R20+R21+R30+R31) / 4 R11 = (R22+R23+R32+R33) / 4

[0045] FIG. 10B shows the reduction process of the interpolated image H2Gr by the Gr component reduction processing unit 320. In FIG. 10B, a map of 16 pixels (4×4) is reduced to a map of 4 pixels (2×2). In FIG. 10B, the four pixels Gr00, Gr01, Gr10, and Gr11 of the Gr component after reduction are calculated by the following formula based on the 16 pixels Gr00, Gr01, Gr02, Gr03, Gr10, Gr11, Gr12, Gr13, Gr20, Gr21, Gr22, Gr23, Gr30, Gr31, Gr32, and Gr33 of the Gr component of the interpolated image H2Gr before reduction. An image composed of the four pixels Gr00, Gr01, Gr10, and Gr11 of the Gr component after reduction is called the reduced image S2Gr. [After reduction] [Before reduction] Gr00 = (Gr00+Gr01+Gr10+Gr11) / 4 Gr01 = (Gr02+Gr03+Gr12+Gr13) / 4 Gr10 = (Gr20+Gr21+Gr30+Gr31) / 4 Gr11 = (Gr22+Gr23+Gr32+Gr33) / 4

[0046] Fig. 10C shows the reduction process of the interpolated image H2Gb by the Gb component reduction processing unit 330. In Fig. 10C, a map of 16 pixels (4 x 4) is reduced to a map of 4 pixels (2 x 2). In Fig. 10C, the four pixels Gb00, Gb01, Gb10, and Gb11 of the Gb component after reduction are calculated by the following formula based on the 16 pixels Gb00, Gb01, Gb02, Gb03, Gb10, Gb11, Gb12, Gb13, Gb20, Gb21, Gb22, Gb23, Gb30, Gb31, Gb32, and Gb33 of the Gb component of the interpolated image H2Gb before reduction. An image composed of the four pixels Gb00, Gb01, Gb10, and Gb11 of the Gb component after reduction is called a reduced image S2Gb. [After reduction] [Before reduction] Gb00 = (Gb00+Gb01+Gb10+Gb11) / 4 Gb01 = (Gb02+Gb03+Gb12+Gb13) / 4 Gb10 = (Gb20+Gb21+Gb30+Gb31) / 4 Gb11 = (Gb22+Gb23+Gb32+Gb33) / 4

[0047] FIG. 10D shows the reduction process of the interpolated image H2G (H2GrGb) by the Gr·Gb component reduction processing unit 340. In FIG. 10D, a map of 16 pixels (4×4) is reduced to a map of 4 pixels (2×2). In FIG. 10D, the four pixels G00, G01, G10, and G11 of the G component after reduction are calculated by the following formula based on the 16 pixels G00, G01, G02, G03, G10, G11, G12, G13, G20, G21, G22, G23, G30, G31, G32, and G33 of the G component of the interpolated image H2G (H2GrGb) before reduction. An image composed of the four pixels G00, G01, G10, and G11 of the G component after reduction is called the reduced image S2G (S2GrGb). [After reduction] [Before reduction] G00 = (G00+G01+G10+G11) / 4 G01 = (G02+G03+G12+G13) / 4 G10 = (G20+G21+G30+G31) / 4 G11 = (G22+G23+G32+G33) / 4

[0048] FIG. 10E shows the reduction process of the interpolated image H2B by the B component reduction processing unit 350. In FIG. 10E, a map of 16 pixels (4×4) is reduced to a map of 4 pixels (2×2). In FIG. 10E, the four pixels B00, B01, B10, and B11 of the B component after reduction are calculated by the following formula based on the 16 pixels B00, B01, B02, B03, B10, B11, B12, B13, B20, B21, B22, B23, B30, B31, B32, and B33 of the B component of the interpolated image H2B before reduction. An image composed of the four pixels B00, B01, B10, and B11 of the B component after reduction is called a reduced image S2B. [After reduction] [Before reduction] B00 = (B00+B01+B10+B11) / 4 B01 = (B02+B03+B12+B13) / 4 B10 = (B20+B21+B30+B31) / 4 B11 = (B22+B23+B32+B33) / 4

[0049] 11A to 11C are diagrams showing an example of the reduction process when the reduction ratio is 3 / 4.

[0050] Fig. 11A shows the reduction process of the post-interpolation image H2R by the R component reduction processor 310. In Fig. 11A, a map of 16 pixels (4 x 4) is reduced to a map of 9 pixels (3 x 3). In Fig. 11A, the nine pixels R00, R01, R02, R10, R11, R12, R20, R21, and R22 of the R component after reduction are calculated by the following formula based on the 16 pixels R00, R01, R02, R03, R10, R11, R12, R13, R20, R21, R22, R23, R30, R31, R32, and R33 of the R component of the post-interpolation image H2R before reduction. An image consisting of the nine pixels R00, R01, R02, R10, R11, R12, R20, R21, and R22 of the R component after reduction is called a post-reduction image S2R. [After reduction] [Before reduction] R00 = (R00*1*1+R01*(1 / 3)*1+R10*1*(1 / 3)+R11*(1 / 3)*(1 / 3)) / (1*1+1*(1 / 3)+1*(1 / 3)+(1 / 3)*(1 / 3)) R01 = (R01*(2 / 3)*1+R02*(2 / 3)*1+R11*(2 / 3)*(1 / 3)+R12*(2 / 3)*(1 / 3)) / ((2 / 3)*1+(2 / 3)*1+(2 / 3)*(1 / 3)+(2 / 3)*(1 / 3)) R02 = (R02*(1 / 3)*1+R03*1*1+R12*(1 / 3)*(1 / 3)+R13*1*(1 / 3)) / ((1 / 3)*1+1*1+(1 / 3)*(1 / 3)+1*(1 / 3)) R10 = (R10*1*(2 / 3)+R11*(1 / 3)*(2 / 3)+R20*1*(2 / 3)+R21*(1 / 3)*(2 / 3)) / (1*(2 / 3)+(1 / 3)*(2 / 3)+1*(2 / 3)+(1 / 3)*(2 / 3)) R11 = (R11*(2 / 3)*(2 / 3)+R12*(2 / 3)*(2 / 3)+R21*(2 / 3)*(2 / 3)+R22*(2 / 3)*(2 / 3)) / ((2 / 3)*(2 / 3)+(2 / 3*(2 / 3)+(2 / 3)*(2 / 3)+(2 / 3)*(2 / 3)) R12 = (R12*(1 / 3)*(2 / 3)+R13*1*(2 / 3)+R22*(1 / 3)*(2 / 3)+R23*1*(2 / 3)) / ((1 / 3)*(2 / 3)+1*(2 / 3)+(1 / 3)*(2 / 3)+1*(2 / 3)) R20 = (R20*1*(1 / 3)+R21*(1 / 3)*(1 / 3)+R30*1*1+R31*(1 / 3)*1) / (1*(1 / 3)+(1 / 3)*(1 / 3)+1*1+(1 / 3)*1) R21 = (R21*(2 / 3)*(1 / 3)+R22*(2 / 3)*(1 / 3)+R31*(2 / 3)*1+R32*(2 / 3)*1) / ((2 / 3)*(1 / 3)+(2 / 3)*(1 / 3)+(2 / 3)*1+(2 / 3)*1) R22 = (R22*(1 / 3)*(1 / 3)+R23*1*(1 / 3)+R32*(1 / 3)*1+R33*1*1) / ((1 / 3)*(1 / 3)+1*(1 / 3)+(1 / 3)*1+1*1)

[0051] Fig. 11B shows the reduction process of the post-interpolation image H2G (H2GrGb) by the Gr·Gb component reduction processing unit 340. In Fig. 11B, a 4 x 4 map of 16 pixels is reduced to a 3 x 3 map of 9 pixels. In Fig. 11B, the nine pixels G00, G01, G02, G10, G11, G12, G20, G21, and G22 of the G component after reduction are calculated by the following formula based on the 16 pixels Gr00, Gr01, Gr02, Gr03, Gb10, Gb11, Gb12, Gb13, Gr20, Gr21, Gr22, Gr23, Gb30, Gb31, Gb32, and Gb33 of the post-interpolation image H2G (H2GrGb) before reduction. An image consisting of the nine G component pixels G00, G01, G02, G10, G11, G12, G20, G21, and G22 after reduction is called the post-reduction image S2G (S2GrGb). [After reduction] [Before reduction] G00 = (Gr00*1*1+Gr01*(1 / 3)*1+Gb10*1*(1 / 3)+Gb11*(1 / 3)*(1 / 3)) / (1*1+1*(1 / 3)+1*(1 / 3)+(1 / 3)*(1 / 3)) G01 = (Gr01*(2 / 3)*1+Gr02*(2 / 3)*1+Gb11*(2 / 3)*(1 / 3)+Gb12*(2 / 3)*(1 / 3)) / ((2 / 3)*1+(2 / 3)*1+(2 / 3)*(1 / 3)+(2 / 3)*(1 / 3)) G02 = (Gr02*(1 / 3)*1+Gr03*1*1+Gb12*(1 / 3)*(1 / 3)+Gb13*1*(1 / 3)) / ((1 / 3)*1+1*1+(1 / 3)*(1 / 3)+1*(1 / 3)) G10 = (Gb10*1*(2 / 3)+Gb11*(1 / 3)*(2 / 3)+Gr20*1*(2 / 3)+Gr21*(1 / 3)*(2 / 3)) / (1*(2 / 3)+(1 / 3)*(2 / 3)+1*(2 / 3)+(1 / 3)*(2 / 3)) G11 = (Gb11*(2 / 3)*(2 / 3)+Gb12*(2 / 3)*(2 / 3)+Gr21*(2 / 3)*(2 / 3)+Gr22*(2 / 3)*(2 / 3)) / ((2 / 3)*(2 / 3)+(2 / 3*(2 / 3)+(2 / 3)*(2 / 3)+(2 / 3)*(2 / 3)) G12 = (Gb12*(1 / 3)*(2 / 3)+Gb13*1*(2 / 3)+Gr22*(1 / 3)*(2 / 3)+Gr23*1*(2 / 3)) / ((1 / 3)*(2 / 3)+1*(2 / 3)+(1 / 3)*(2 / 3)+1*(2 / 3)) G20 = (Gr20*1*(1 / 3)+Gr21*(1 / 3)*(1 / 3)+Gb30*1*1+Gb31*(1 / 3)*1) / (1*(1 / 3)+(1 / 3)*(1 / 3)+1*1+(1 / 3)*1) G21 = (Gr21*(2 / 3)*(1 / 3)+Gr22*(2 / 3)*(1 / 3)+Gb31*(2 / 3)*1+Gb32*(2 / 3)*1) / ((2 / 3)*(1 / 3)+(2 / 3)*(1 / 3)+(2 / 3)*1+(2 / 3)*1) G22 = (Gr22*(1 / 3)*(1 / 3)+Gr23*1*(1 / 3)+Gb32*(1 / 3)*1+Gb33*1*1) / ((1 / 3)*(1 / 3)+1*(1 / 3)+(1 / 3)*1+1*1)

[0052] Fig. 11C shows the reduction process of the interpolated image H2B by the B component reduction processing unit 350. In Fig. 11C, a map of 16 pixels (4 x 4) is reduced to a map of 9 pixels (3 x 3). In Fig. 11C, the nine pixels B00, B01, B02, B10, B11, B12, B20, B21, and B22 of the B component after reduction are calculated by the following formula based on the 16 pixels B00, B01, B02, B03, B10, B11, B12, B13, B20, B21, B22, B23, B30, B31, B32, and B33 of the B component of the interpolated image H2B before reduction. An image consisting of the nine pixels B00, B01, B02, B10, B11, B12, B20, B21, and B22 of the B component after reduction is called a post-reduction processing image S2B. [After reduction] [Before reduction] B00 = (B00*1*1+B01*(1 / 3)*1+B10*1*(1 / 3)+B11*(1 / 3)*(1 / 3)) / (1*1+1*(1 / 3)+1*(1 / 3)+(1 / 3)*(1 / 3)) B01 = (B01*(2 / 3)*1+B02*(2 / 3)*1+B11*(2 / 3)*(1 / 3)+B12*(2 / 3)*(1 / 3)) / ((2 / 3)*1+(2 / 3)*1+(2 / 3)*(1 / 3)+(2 / 3)*(1 / 3)) B02 = (B02*(1 / 3)*1+B03*1*1+B12*(1 / 3)*(1 / 3)+B13*1*(1 / 3)) / ((1 / 3)*1+1*1+(1 / 3)*(1 / 3)+1*(1 / 3)) B10 = (B10*1*(2 / 3)+B11*(1 / 3)*(2 / 3)+B20*1*(2 / 3)+B21*(1 / 3)*(2 / 3)) / (1*(2 / 3)+(1 / 3)*(2 / 3)+1*(2 / 3)+(1 / 3)*(2 / 3)) B11 = (B11*(2 / 3)*(2 / 3)+B12*(2 / 3)*(2 / 3)+B21*(2 / 3)*(2 / 3)+B22*(2 / 3)*(2 / 3)) / ((2 / 3)*(2 / 3)+(2 / 3*(2 / 3)+(2 / 3)*(2 / 3)+(2 / 3)*(2 / 3)) B12 = (B12*(1 / 3)*(2 / 3)+B13*1*(2 / 3)+B22*(1 / 3)*(2 / 3)+B23*1*(2 / 3)) / ((1 / 3)*(2 / 3)+1*(2 / 3)+(1 / 3)*(2 / 3)+1*(2 / 3)) B20 = (B20*1*(1 / 3)+B21*(1 / 3)*(1 / 3)+B30*1*1+B31*(1 / 3)*1) / (1*(1 / 3)+(1 / 3)*(1 / 3)+1*1+(1 / 3)*1) B21 = (B21*(2 / 3)*(1 / 3)+B22*(2 / 3)*(1 / 3)+B31*(2 / 3)*1+B32*(2 / 3)*1) / ((2 / 3)*(1 / 3)+(2 / 3)*(1 / 3)+(2 / 3)*1+(2 / 3)*1) B22 = (B22*(1 / 3)*(1 / 3)+B23*1*(1 / 3)+B32*(1 / 3)*1+B33*1*1) / ((1 / 3)*(1 / 3)+1*(1 / 3)+(1 / 3)*1+1*1)

[0053] Returning to FIG. 3, the resampling processing section 400 has an R component resampling processing section 410, a Gr component resampling processing section 420, a Gb component resampling processing section 430, a Gr·Gb component resampling processing section 440, and a B component resampling processing section 450.

[0054] The R component resampling processor 410, the Gr component resampling processor 420, the Gb component resampling processor 430, the Gr·Gb component resampling processor 440, and the B component resampling processor 450 perform resampling processing in accordance with the reduction ratio A.

[0055] 12A to 12E are diagrams showing an example of resampling processing when the reduction ratio is 1 / 2.

[0056] In FIG. 12A, the R component resampling processor 410 extracts pixel R00 from among four pixels R00, R01, R10, and R11 of the R component of the post-reduction processing image S2R as a resampled image RS2R.

[0057] In FIG. 12B, the Gr component resampling processor 420 extracts the pixel Gr01 from among the four pixels Gr00, Gr01, Gr10, and Gr11 of the Gr component of the post-reduction processing image S2Gr as a resampled image RS2Gr.

[0058] In FIG. 12C, the Gb component resampling processor 430 extracts pixel Gb10 from the four pixels Gb00, Gb01, Gb10, and Gb11 of the Gb component of the post-reduction processing image S2Gb as a resampled image RS2Gb.

[0059] In FIG. 12D, the Gr / Gb component resampling processor 440 extracts pixels G01 and G10 from the four G component pixels G00, G01, G10, and G11 of the post-reduction processing image S2G (S2GrGb) as a resampled image RS2G (RS2GrGb).

[0060] In FIG. 12E, the B component resampling processor 450 extracts pixel B11 from among four pixels B00, B01, B10, and B11 of the B component of the post-reduction processing image S2B as a resampled image RS2B.

[0061] 13A to 13E are diagrams showing an example of the resampling process when the reduction ratio is 3 / 4.

[0062] In FIG. 13A, the R component resampling processing unit 410 extracts pixels R00, R02, R20, and R22 from the nine pixels R00, R01, R02, R10, R11, R12, R20, R21, and R22 of the R component of the post-reduction processing image S2R as a resampled image RS2R.

[0063] In FIG. 13B, the Gr component resampling processing unit 420 extracts pixels Gr01 and Gr21 from the nine pixels Gr00, Gr01, Gr02, Gr10, Gr11, Gr12, Gr20, Gr21, and Gr22 of the Gr component of the post-reduction processing image S2Gr as a resampled image RS2Gr.

[0064] In FIG. 13C, the Gb component resampling processing unit 430 extracts pixels Gb10 and Gb12 from the nine pixels Gb00, Gb01, Gb02, Gb10, Gb11, Gb12, Gb20, Gb21, and Gb22 of the Gb component of the post-reduction processing image S2Gb as a resampled image RS2Gb.

[0065] In FIG. 13D, the Gr / Gb component resampling processing unit 440 extracts pixels G01, G10, G12, and G21 from the nine pixels G00, G01, G02, G10, G11, G12, G20, G21, and G22 of the G component of the reduced image S2G (S2GrGb) as the resampled image RS2G (RS2GrGb).

[0066] In FIG. 13E, the B component resampling processing unit 450 extracts pixel B11 from the nine pixels B00, B01, B02, B10, B11, B12, B20, B21, and B22 of the B component of the reduction-processed image S2B as a resampled image RS2B.

[0067] The RAW image reducing unit 3 has a reduction ratio threshold comparing unit 600, and the input path of the resampled image from the resampling processing unit 400 to the data combining unit 500 varies depending on the comparison result of the reduction ratio threshold comparing unit 600. The reduction ratio threshold comparing unit 600 compares the reduction ratio A and the threshold S to see which is larger, and sends switch changeover signals W3 and W4 to the switches S3 and S4 depending on the comparison result.

[0068] More specifically, when the reduction ratio A is greater than the threshold S, the switches S3 and S4 are switched to the dashed line position in Fig. 3, and the resampled image RS2G (RS2GrGb) from the Gr / Gb component resampling processing unit 440 is separated into a Gr component (Rs2Gr) and a Gb component (RS2Gb) and input to the data combination unit 500. On the other hand, when the reduction ratio A is smaller than the threshold S, the switches S3 and S4 are switched to the solid line position in Fig. 3, and the resampled image RS2Gr from the Gr component resampling processing unit 420 and the resampled image RS2Gb from the Gb component resampling processing unit 430 are each input to the data combination unit 500. Note that when the reduction ratio A and the threshold S are the same value, either of the above controls may be performed.

[0069] Furthermore, regardless of the magnitude relationship between the reduction ratio A and the threshold value S (regardless of the comparison result of the reduction ratio threshold value comparison unit 600), the resampled image RS2R from the R component resampling processing unit 410 and the resampled image RS2B from the B component resampling processing unit 450 are each input to the data combination unit 500.

[0070] The data combining unit 500 then combines the resampled images input from the resampling processing unit 400 to generate a reduced RAW image D3.

[0071] 14A and 14B are diagrams showing an example of the combining process when the reduction ratio is 1 / 2.

[0072] In Figure 14A, pixel R00 extracted by the R component resampling processing unit 410 as the resampled image RS2R, pixel Gr01 extracted by the Gr component resampling processing unit 420 as the resampled image RS2Gr, pixel Gb10 extracted by the Gb component resampling processing unit 430 as the resampled image RS2Gb, and pixel B11 extracted by the B component resampling processing unit 450 as the resampled image RS2B are combined.

[0073] In FIG. 14B, pixel R00 extracted by the R component resampling processing unit 410 as the resampled image RS2R, pixels G01 and G10 extracted by the Gr / Gb component resampling processing unit 440 as the resampled image RS2G (RS2GrGb), and pixel B11 extracted by the B component resampling processing unit 450 as the resampled image RS2B are combined.

[0074] 15A and 15B are diagrams showing an example of the combining process when the reduction ratio is 3 / 4.

[0075] In Figure 15A, pixels R00, R02, R20, and R22 extracted by the R component resampling processing unit 410 as the resampled image RS2R, pixels Gr01 and Gr21 extracted by the Gr component resampling processing unit 420 as the resampled image RS2Gr, pixels Gb10 and Gb12 extracted by the Gb component resampling processing unit 430 as the resampled image RS2Gb, and pixel B11 extracted by the B component resampling processing unit 450 as the resampled image RS2B are combined.

[0076] In FIG. 15B, pixels R00, R02, R20, and R22 extracted by the R component resampling processing unit 410 as the resampled image RS2R, pixels G01, G10, G12, and G21 extracted by the Gr / Gb component resampling processing unit 440 as the resampled image RS2G (RS2GrGb), and pixel B11 extracted by the B component resampling processing unit 450 as the resampled image RS2B are combined.

[0077] In the image processing of this embodiment, when the reduction ratio A is greater than the threshold value S (close to unity), a first interpolation process is executed in which the Gr and Gb components (first and second color components) of the G components are treated as the same color components and interpolated in one (common) map, and when the reduction ratio A is smaller than the threshold value S, a second interpolation process is executed in which the Gr and Gb components (first and second color components) of the G components are treated as different color components and interpolated in multiple (separate) maps. This makes it possible to perform image processing optimized according to the color components.

[0078] Furthermore, from the viewpoint of effectively reducing the size of the Bayer image (R, Gr, Gb, B), a determination is made as to whether a first reduction process (see FIG. 11) is to be performed when the reduction rate A is greater than a threshold value S (close to life size), or a second reduction process (see FIG. 10) is to be performed when the reduction rate A is less than the threshold value S. This makes it possible to provide reduced images that have been reduced using different methods depending on the output destination.

[0079] In this way, when the reduction ratio A is relatively large (close to life size), a high-quality image with reduced jaggies and improved resolution can be obtained, while when the reduction ratio A is relatively small, a high-quality image with reduced false colors and moire can be obtained.

[0080] The above-mentioned effects will be further described with reference to Fig. 16A and Fig. 16B. Fig. 16A and Fig. 16B are first diagrams showing an example of the effects of the image processing of this embodiment. Fig. 16A corresponds to a first interpolation process in which the Gr component and the Gb component (first and second color components) of the G component are interpolated as the same color component in one (common) map, and Fig. 16B corresponds to a second interpolation process in which the Gr component and the Gb component (first and second color components) of the G component are interpolated as different color components in multiple (separate) maps. Fig. 16B illustrates the Gr component, but the same applies to the Gb component.

[0081] In the first interpolation process of FIG. 16A, of the 16 pixels of 4×4, eight pixels (G01, G03, G21, G23, G10, G12, G30, G32) are included as the source data of interpolation (before interpolation). These eight pixels (G01, G03, G21, G23, G10, G12, G30, G32) as the source data of interpolation (before interpolation) are enclosed in a thick frame and indicated with the code GA. In contrast, in the second interpolation process of FIG. 16B, of the 16 pixels of 4×4, four pixels (Gr01, Gr03, Gr21, Gr23) are included as the source data of interpolation (before interpolation). These four pixels (Gr01, Gr03, Gr21, Gr23) as the source data of interpolation (before interpolation) are enclosed in a thick frame and indicated with the code GrA. In this way, the first interpolation process, which has a larger amount of source (pre-interpolation) data, is more advantageous in terms of reducing jaggies and improving resolution.

[0082] On the other hand, in the second interpolation process of FIG. 16B, the Gr component and the Gb component of the G component are made into separate maps, so that the interpolation can be performed by referring to a wider range. That is, in the first interpolation process of FIG. 16A, the eight pixels GA (G01, G03, G21, G23, G10, G12, G30, G32) as the data to be interpolated (before interpolation) are close to each other, whereas in the second interpolation process of FIG. 16B, the four pixels GrA (Gr01, Gr03, Gr21, Gr23) as the data to be interpolated (before interpolation) are far from each other. For example, when performing interpolation by referring to the left and right pixels, the number of pixels increases by +1, but this actually corresponds to referring to a wider range, and it is possible to obtain a stronger low-pass filter effect as a countermeasure against false colors and moire. When performing the interpolation and the low-pass filter process simultaneously, the number of reference pixels can be reduced as the pixels are farther apart. If the buffer for holding the input data is the same, the number of reference pixels can be reduced as the number of reference pixels is reduced, so that the low-pass filter can be applied over a wider range.

[0083] 17A, 17B, 17C, and 17D are second diagrams showing an example of the effect of the image processing of this embodiment. FIG. 17A and FIG. 17B correspond to a first interpolation process in which the Gr component and the Gb component (first and second color components) of the G component are interpolated as the same color components in one (common) map, and FIG. 17C and FIG. 17D correspond to a second interpolation process in which the Gr component and the Gb component (first and second color components) of the G component are interpolated as different color components in multiple (separate) maps. FIG. 17C and FIG. 17D show the Gr component as an example, but the same applies to the Gb component. In FIG. 17A to FIG. 17D, the area surrounded by a thick solid line indicates the range LPFA of the low-pass filter, and the area surrounded by a thick dashed line indicates the range Din of the input data referred to in the interpolation process and the low-pass filter process.

[0084] In FIG. 17A, the interpolation process is performed by referring to the input data (Din) of 12 pixels, and the low-pass filter process (LPFA) of 9 pixels of 3×3 is performed. In FIG. 17B, the interpolation process is performed by referring to the input data (Din) of 24 pixels, and the low-pass filter process (LPFA) of 25 pixels of 5×5 is performed. In FIG. 17C, the interpolation process is performed by referring to the input data (Din) of 9 pixels, and the low-pass filter process (LPFA) of 9 pixels of 3×3 is performed. In FIG. 17D, the interpolation process is performed by referring to the input data (Din) of 16 pixels, and the low-pass filter process (LPFA) of 25 pixels of 5×5 is performed. In this way, the second interpolation process in FIG. 17C and FIG. 17D achieves a wide range of low-pass filter effects with a smaller number of reference pixels.

[0085] 18 is a functional block diagram showing another modified example of the internal configuration of the RAW image reducing unit 3. In this modified example, an R component low-pass filter processing unit 1210 is added between the R component interpolation processing unit 210 and the R component reduction processing unit 310, a Gr component low-pass filter processing unit 1220 is added between the Gr component interpolation processing unit 220 and the Gr component reduction processing unit 320, a Gb component low-pass filter processing unit 1230 is added between the Gb component interpolation processing unit 230 and the Gb component reduction processing unit 330, a Gr·Gb component low-pass filter processing unit 1240 is added between the Gr·Gb component interpolation processing unit 240 and the Gr·Gb component reduction processing unit 340, and a B component low-pass filter processing unit 1250 is added between the B component interpolation processing unit 250 and the B component reduction processing unit 350 in FIG. The R component low-pass filter processing unit 1210, the Gr component low-pass filter processing unit 1220, the Gb component low-pass filter processing unit 1230, the Gr·Gb component low-pass filter processing unit 1240, and the B component low-pass filter processing unit 1250 perform low-pass filter processing according to the reduction ratio A in addition to the low-pass filter effect associated with the above-mentioned interpolation processing. That is, in addition to the low-pass filter effect associated with the first and second interpolation processing, an optimized additional low-pass filter effect can be obtained by the R component low-pass filter processing unit 1210, the Gr component low-pass filter processing unit 1220, the Gb component low-pass filter processing unit 1230, the Gr·Gb component low-pass filter processing unit 1240, and the B component low-pass filter processing unit 1250. Furthermore, the moire removal performance can be improved by applying a low-pass filter to each pixel map.

[0086] Fig. 19 is a functional block diagram showing yet another modified example of the internal configuration of the RAW image reducing unit 3. In Fig. 3, the interpolation process, reduction process, and resampling process are performed simultaneously for the horizontal and vertical directions of the pixel map (RAW image D2), whereas in this modified example, the interpolation process, reduction process, and resampling process are performed separately for the horizontal and vertical directions of the pixel map (RAW image D2). This makes it possible to execute image processing (for example, first and second interpolation processes) optimized separately for the horizontal and vertical directions of the pixel map (RAW image D2). Furthermore, by dividing the processing steps (reducing the horizontal direction and then reducing the vertical direction), image processing with a small buffer amount becomes possible.

[0087] Specifically, in FIG. 3, the R component interpolation processing unit 210 is divided into an R component horizontal direction interpolation processing unit 210H and an R component vertical direction interpolation processing unit 210V, the R component downsampling processing unit 310 is divided into an R component horizontal direction downsampling processing unit 310H and an R component vertical direction downsampling processing unit 310V, and the R component resampling processing unit 410 is divided into an R component horizontal direction resampling processing unit 410H and an R component vertical direction resampling processing unit 410V.

[0088] Also, in FIG. 3, the Gr component interpolation processing unit 220 is divided into a Gr component horizontal direction interpolation processing unit 220H and a Gr component vertical direction interpolation processing unit 220V, the Gr component downsampling processing unit 320 is divided into a Gr component horizontal direction downsampling processing unit 320H and a Gr component vertical direction downsampling processing unit 320V, and the Gr component resampling processing unit 420 is divided into a Gr component horizontal direction resampling processing unit 420H and a Gr component vertical direction resampling processing unit 420V.

[0089] Also, in FIG. 3, Gb component interpolation processing unit 230 is divided into Gb component horizontal direction interpolation processing unit 230H and Gb component vertical direction interpolation processing unit 230V, Gb component downsampling processing unit 330 is divided into Gb component horizontal direction downsampling processing unit 330H and Gb component vertical direction downsampling processing unit 330V, and Gb component resampling processing unit 430 is divided into Gb component horizontal direction resampling processing unit 430H and Gb component vertical direction resampling processing unit 430V.

[0090] Also, in FIG. 3, the Gr·Gb component interpolation processing unit 240 is divided into a Gr·Gb component horizontal direction interpolation processing unit 240H and a Gr·Gb component vertical direction interpolation processing unit 240V, the Gr·Gb component reduction processing unit 340 is divided into a Gr·Gb component horizontal direction reduction processing unit 340H and a Gr·Gb component vertical direction reduction processing unit 340V, and the Gr·Gb component resampling processing unit 440 is divided into a Gr·Gb component horizontal direction resampling processing unit 440H and a Gr·Gb component vertical direction resampling processing unit 440V.

[0091] Also, in FIG. 3, the B component interpolation processing unit 250 is divided into a B component horizontal direction interpolation processing unit 250H and a B component vertical direction interpolation processing unit 250V, the B component reduction processing unit 350 is divided into a B component horizontal direction reduction processing unit 350H and a B component vertical direction reduction processing unit 350V, and the B component resampling processing unit 450 is divided into a B component horizontal direction resampling processing unit 450H and a B component vertical direction resampling processing unit 450V.

[0092] FIG. 20 is a flowchart showing an example of the image processing of the present embodiment. The main body that executes each processing step of the flowchart in FIG. 20 is, for example, a functional component of the CPU (computer) that constitutes the RAW image reduction unit (image processing apparatus) 3.

[0093] In step ST1, the data division unit 100 divides the RAW image D2 into an R component divided image D2R, a Gr component divided image D2Gr, a Gb component divided image D2Gb, and a B component divided image D2B.

[0094] In step ST2, the magnitude relationship between the reduction rate A and the threshold value S is determined. When the reduction rate A is greater than the threshold value S (A > S), the process proceeds to step ST3, and when the reduction rate A is smaller than the threshold value S (A < S), the process proceeds to step ST7.

[0095] In step ST3, the interpolation processing unit 200 executes a first interpolation process in which the Gr component and the Gb component (first and second color components) of the G component are interpolated in one (common) map as the same color component. In step ST4, the reduction processing unit 300 executes a reduction process according to the reduction ratio A. In step ST5, the resampling processing unit 400 executes a resampling process according to the reduction ratio A. In step ST6, the data combination unit 500 combines the resampled images input from the resampling processing unit 400 to generate a reduced RAW image D3.

[0096] In step ST7, the interpolation processing unit 200 performs a second interpolation process in which the Gr component and Gb component (first and second color components) of the G component are interpolated in multiple (separate) maps as different color components. In step ST8, the reduction processing unit 300 performs a reduction process according to the reduction ratio A. In step ST9, the resampling processing unit 400 performs a resampling process according to the reduction ratio A. In step ST10, the data combination unit 500 combines the resampled images input from the resampling processing unit 400 to generate a reduced RAW image D3.

[0097] In addition, the present invention is not limited to the above-mentioned embodiment, and various modifications can be made in the implementation stage without departing from the gist of the invention. In addition, the functions executed in the above-mentioned embodiment may be implemented in appropriate combinations as much as possible. The above-mentioned embodiment includes various steps, and various inventions can be extracted by appropriate combinations of the multiple components disclosed. For example, if an effect can be obtained even if some components are deleted from all the components shown in the embodiment, the configuration from which the components are deleted can be extracted as an invention.

[0098] For example, in the above embodiment, the Gr component and Gb component of the G component for reducing Bayer data including R, G, and B components are described as the "first and second color components," but the "first and second color components" may be color components other than the Gr component and Gb component.

[0099] The invention as described in the claims of the original application is set forth below. [Appendix 1] An image processing device that determines whether to execute a first interpolation process that interpolates first and second color components as the same color components using one map, or a second interpolation process that interpolates the first and second color components as different color components using multiple maps. [Appendix 2] 2. The image processing device according to claim 1, wherein the first and second color components are Gr and Gb components of the G component for reducing Bayer data including an R component, a G component, and a B component. [Appendix 3] 3. The image processing device according to claim 1, further comprising: a processor for processing the image data based on the determined result; [Appendix 4] The image processing device according to claim 3, wherein, when the reduction ratio is greater than a threshold, the first interpolation process is executed, and, when the reduction ratio is smaller than the threshold, the second interpolation process is executed. [Appendix 5] 5. The image processing device according to claim 1, wherein the image processing device automatically determines which of the processes to execute without receiving an instruction from a user during image capture by the image capture device. [Appendix 6] An image processing device according to any one of claims 1 to 5, which generates reduced data of Bayer data including the first and second color components by executing either the first interpolation process or the second interpolation process. [Appendix 7] A method for causing a computer to execute each process of the image processing device according to any one of appendixes 1 to 6. [Appendix 8] A program for causing a computer to execute each process of the image processing device according to any one of appendixes 1 to 6. [Explanation of symbols]

[0100] 3. RAW image reduction unit (image processing device) 100 Data division section 200 Interpolation processing unit 210 R component interpolation processing unit 220···Gr component interpolation processing section 230···Gb component interpolation processing section 240···Gr·Gb component interpolation processing section 250···B component interpolation processing section 300 Reduction processing unit 310: R component reduction processing unit 320···Gr component reduction processing section 330···Gb component reduction processing section 340···Gr·Gb component reduction processing section 350 B component reduction processing section 400···Resampling processing section 410 R component resampling processing unit 420···Gr component resampling processing section 430···Gb component resampling processing section 440···Gr·Gb component resampling processing section 450 B component resampling processing section 500 Data connection section 600...Reduction ratio threshold comparison section 1210 R component low-pass filter processing section 1220···Gr component low-pass filter processing section 1230···Gb component low-pass filter processing section 1240···Gr·Gb component low-pass filter processing section 1250···B component low-pass filter processing section

Claims

1. a control means for determining whether to execute a first interpolation process for interpolating a G component in the Bayer data using one map or a second interpolation process for interpolating a G component in a row in which an R component is arranged and a G component in a row in which a B component is arranged using separate maps, based on a reduction ratio value when the Bayer data is reduced; the control means determines to execute the first interpolation process when the reduction rate is greater than a threshold value as a case where the degree of reduction is small, and determines to execute the second interpolation process when the reduction rate is less than the threshold value as a case where the degree of reduction is large. Image processing device.

2. A control means is provided which, when interpolating pixels for each color component based on Bayer data, determines whether to separate the G component into a Gr component as the G component of a row in which an R component is arranged and a Gb component as the G component of a row in which a B component is arranged and perform pixel interpolation for the Gr component and the Gb component separately, or to perform pixel interpolation for the G component without separating the G component into the Gr component and the Gb component, based on a value of a reduction rate when reducing the Bayer data; The control means When the degree of reduction is small, that is, when the value of the reduction ratio is greater than a threshold value, it is determined to perform pixel interpolation for the G component without separating the G component into the Gr component and the Gb component; When the degree of reduction is large, that is, when the value of the reduction ratio is smaller than the threshold value, it is determined that the G component is separated into the Gr component and the Gb component, and pixel interpolation is performed separately for the Gr component and the Gb component. Image processing device.

3. An image processing method executed by an image processing device, comprising: a control process for determining whether to execute a first interpolation process for interpolating a G component in the Bayer data using one map or a second interpolation process for interpolating a G component in a row in which an R component is arranged and a G component in a row in which a B component is arranged using separate maps, based on a reduction ratio value when reducing the Bayer data; The control process determines to execute the first interpolation process when the degree of reduction is small and the value of the reduction ratio is greater than a threshold value, and determines to execute the second interpolation process when the degree of reduction is large and the value of the reduction ratio is smaller than the threshold value. Image processing methods.

4. An image processing method executed by an image processing device, comprising: a control process for determining, when a pixel is to be interpolated for each color component based on Bayer data, whether to separate the G component into a Gr component as the G component of a row in which an R component is arranged and a Gb component as the G component of a row in which a B component is arranged and to perform pixel interpolation for the Gr component and the Gb component separately, or to perform pixel interpolation for the G component without separating the G component into the Gr component and the Gb component, based on a value of a reduction ratio when reducing the Bayer data; The control process includes: When the degree of reduction is small, that is, when the value of the reduction ratio is greater than a threshold value, it is determined to perform pixel interpolation for the G component without separating the G component into the Gr component and the Gb component; When the degree of reduction is large, that is, when the value of the reduction ratio is smaller than the threshold value, it is determined that the G component is separated into the Gr component and the Gb component, and pixel interpolation is performed separately for the Gr component and the Gb component. Image processing methods.

5. A computer comprising: functioning as a control means for determining whether to execute a first interpolation process for interpolating the G component in the Bayer data using one map, or a second interpolation process for interpolating the G component in a row in which the R component is arranged and the G component in a row in which the B component is arranged using separate maps, based on a reduction ratio value when reducing the Bayer data; the control means determines to execute the first interpolation process when the reduction rate is greater than a threshold value as a case where the degree of reduction is small, and determines to execute the second interpolation process when the reduction rate is less than the threshold value as a case where the degree of reduction is large. Image processing program.

6. Computer, when interpolating pixels for each color component based on Bayer data, whether to separate the G component into a Gr component as the G component of a row in which an R component is arranged and a Gb component as the G component of a row in which a B component is arranged and perform pixel interpolation for the Gr component and the Gb component separately, or to perform pixel interpolation for the G component without separating the G component into the Gr component and the Gb component, based on a reduction ratio value when reducing the Bayer data; The control means When the degree of reduction is small, that is, when the value of the reduction ratio is greater than a threshold value, it is determined to perform pixel interpolation for the G component without separating the G component into the Gr component and the Gb component; When the degree of reduction is large, that is, when the value of the reduction ratio is smaller than the threshold value, it is determined that the G component is separated into the Gr component and the Gb component, and pixel interpolation is performed separately for the Gr component and the Gb component. Image processing program.

Images