Image processing device and imaging device

Abstract

To provide an image processing device and an imaging apparatus which reduce a circuit scale.SOLUTION: An image processing device 200 being a device formed of an image processing unit that excludes an imaging unit from an imaging apparatus comprises: a selection unit which selects any of a plurality of pieces of image data with different sizes in a time division manner; and a noise removal unit 260 which removes noise from the image data selected by the selection unit in a selection order.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image processing apparatus and an imaging apparatus. [Background technology]

[0002] Conventionally, when removing noise from an image, there is a technique that involves applying a multi-resolution conversion process to the image to generate multiple reduced images of different sizes, and then removing noise from the original image based on the results of noise extraction processing on each reduced image. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2000-224421 [Overview of the project]

[0004] The image processing apparatus of the first disclosed technology includes a selection unit that time-division selects one of a plurality of image data of different sizes, and a noise reduction unit that removes noise from the image data selected by the selection unit in the order of selection.

[0005] The image processing device of the second disclosed technology processes 1 / 2 of the input image data. 2 1 / 2 of the size 2n A reduction unit generates a set of reduced image data up to a size (where n is an integer greater than or equal to 1), and for the reduced image data set reduced by the reduction unit, it processes the image data by 1 / 2 2(n-1) 1 / 2 of the size 2 An enlargement unit that generates a set of enlarged image data up to the size, and the 1 / 2 2n Image data with reduced size and 1 / 2 generated by the enlargement unit. 2(n-1) 1 / 2 of the size 2 A group of enlarged image data up to a certain size, a selection unit that selects any image data from this group in a time-division manner, and a unit that denoises the image data selected by the selection unit in the order of selection, so that the size of the denoised image data is 1 / 2 2The system includes a noise reduction unit that outputs the noise-reduced image data to the enlargement unit until it reaches the correct size.

[0006] The image processing apparatus of the third disclosed technology includes a generation unit that generates a plurality of image data of different sizes based on input image data, and a noise reduction unit that removes noise from selected image data of different sizes generated by the generation unit in a selection order. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 is a block diagram showing an example of the hardware configuration of an imaging device. [Figure 2] Figure 2 is a block diagram showing an example of the hardware configuration of an image processing device. [Figure 3] Figure 3 is an explanatory diagram showing an example of the configuration of the reduced section. [Figure 4] Figure 4 is an explanatory diagram showing an example of 1 / 162 resolution processing. [Figure 5] Figure 5 is an explanatory diagram showing an example of 1 / 82 resolution processing. [Figure 6] Figure 6 is an explanatory diagram showing an example of 1 / 42 resolution processing. [Figure 7] Figure 7 is an explanatory diagram showing an example of 1 / 22 resolution processing. [Figure 8] Figure 8 is an explanatory diagram showing example 1 of time-division noise reduction processing. [Figure 9] Figure 9 is an explanatory diagram showing example 2 of time-division noise reduction processing. [Figure 10] Figure 10 is an explanatory diagram showing example 3 of time-division noise reduction processing. [Figure 11] Figure 11 is an explanatory diagram showing example 4 of time-division noise reduction processing. [Figure 12] Figure 12 is an explanatory diagram showing example 5 of time-division noise reduction processing. [Figure 13] Figure 13 is an explanatory diagram showing example 6 of time-division noise reduction processing. [Figure 14]Figure 14 is an explanatory diagram showing example 7 of time-division noise reduction processing. [Modes for carrying out the invention]

[0008] <Example of hardware configuration for imaging device> Figure 1 is a block diagram showing an example of the hardware configuration of an imaging device. The imaging device 100 is a device capable of capturing still images or videos, and specifically includes, for example, a digital camera, a digital video camera, a smartphone, a tablet, a personal computer, or a game console. In Figure 1, a digital camera is used as an example of an imaging device for explanation.

[0009] The imaging device 100 includes a processor 101, a storage device 102, a drive unit 103, an optical system 104, an image sensor 105, an AFE (Analog Front End) 106, an LSI (Large Scale Integration) 107, an operating device 108, a sensor 109, a display device 110, a communication IF (Interface) 111, and a bus 112. The processor 101, storage device 102, drive unit 103, LSI 107, operating device 108, sensor 109, display device 110, and communication IF 111 are connected to the bus 112.

[0010] The processor 101 controls the imaging device 100. The memory device 102 serves as the work area for the processor 101. The memory device 102 is a non-temporary or temporary recording medium that stores various programs and data. Examples of the memory device 102 include ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), and flash memory. Multiple memory devices 102 may be mounted on the imaging device 100, and at least one of them may be detachable from the imaging device 100.

[0011] The drive unit 103 drives and controls the optical system 104. The drive unit 103 has a drive circuit 103a and a drive source 103b. The drive circuit 103a controls the drive source 103b by instructions from the processor 101. The drive source 103b is, for example, a motor, and under the control of the drive circuit 103a, moves the zooming lens 741b and focusing lens 741c in the optical system 104 in the optical axis direction, and controls the opening and closing of the aperture 742.

[0012] The optical system 104 includes a plurality of lenses (lens 741a, zooming lens 741b, and focusing lens 741c) arranged in the optical axis direction, and an aperture 742. The optical system 104 collects subject light and emits it to the image sensor 105.

[0013] The image sensor 105 receives subject light from the optical system 104 and converts it into an electrical signal. The image sensor 105 may be, for example, an XY addressing solid-state image sensor (e.g., CMOS (Complementary Metal-Oxide Semiconductor)) or a sequential scanning solid-state image sensor (e.g., a CCD (Charge Coupled Device)).

[0014] Multiple light-receiving elements (pixels) are arranged in a matrix on the light-receiving surface of the image sensor 105. Each pixel of the image sensor 105 is fitted with multiple types of color filters, each transmitting light of a different color component, arranged according to a predetermined color arrangement (for example, a Bayer arrangement). Therefore, each pixel of the image sensor 105 outputs an analog electrical signal corresponding to each color component through color separation by the color filters.

[0015] AFE106 is an analog front-end circuit that performs signal processing on the analog electrical signal from the image sensor 105. AFE106 sequentially performs gain adjustment of the electrical signal, analog signal processing (correlated double sampling, black level correction, etc.), A / D conversion processing, and digital signal processing (defective pixel correction, etc.) to generate RAW image data, which is then output to LSI107. The drive unit 103, optical system 104, image sensor 105, and AFE106 described above constitute the imaging unit 720.

[0016] LSI107 is an integrated circuit that performs specific processing on RAW image data from AFE106, such as color interpolation, white balance adjustment, edge enhancement, gamma correction, and gradation conversion, as well as encoding, decoding, and compression / decompression. Specifically, LSI107 may be implemented by a PLD (Programmable Logic Device) such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array).

[0017] The operating device 108 is used to input commands and data. Examples of the operating device 108 include various buttons including a shutter release button, switches, dials, and a touch panel. Sensors are devices that detect information, such as AF (Automatic Focus) sensors, AE (Automatic Exposure) sensors, gyro sensors, accelerometers, and temperature sensors. The display device 110 displays image data and setting screens 300. The display device 110 includes a rear monitor located on the back of the imaging device 100 and an electronic viewfinder. The communication IF 111 connects to the network and sends and receives data.

[0018] The processor 101, memory device 102, LSI 107, operating device 108, sensor 109, display device 110, communication IF 111, and bus 112 are collectively referred to as the image processing unit 130. Furthermore, a device consisting of the image processing unit 130, excluding the imaging unit 120 from the imaging device 100, is referred to as an image processing device.

[0019] <Example Hardware Configuration of Image Processing Apparatus> Figure 2 is a block diagram showing an example hardware configuration of the image processing apparatus. The image processing apparatus 200 executes multi-resolution analysis, for example. The image processing apparatus 200 includes a reduction unit 210, a first selection unit 220, a modification unit 230, an enlargement unit 240, a second selection unit 250, a noise removal unit 260, and a third selection unit 270.

[0020] The reduction unit 210 is a circuit that generates a plurality of pieces of reduced image data each smaller than the size (also referred to as resolution) of input image data and having different sizes from the input image data. That is, the reduction unit 210 is a circuit that functions as a generation unit that generates a plurality of pieces of image data with different sizes based on the input image data. Specifically, for example, the reduction unit 210 reduces the input image data to 1 / 2 2 size to 1 / 2 2n size (where n is an integer of 1 or greater), and generates a reduced image data group up to that size. Figure 2 shows an example where n=4. When n=4, the reduction unit 210 generates 1 / 2 2 size, 1 / 4 2 size, 1 / 8 2 size, and 1 / 16 2 size reduced image data.

[0021] The first selection unit 220 stores the reduced image data group from 1 / 2 2 size to 1 / 2 2(n-1) size, selects reduced image data of any size, and outputs it to the modification unit 230. Specifically, for example, the first selection unit 220 selects the reduced image data in the order they are stored and outputs the selected data to the modification unit 230.

[0022] The first selection unit 220 has (n-1) line buffers each storing the reduced image data from 1 / 2 2 size to 1 / 2 2(n-1) size (where n is an integer of 1 or greater). Since FIG. 2 is an example where n=4, the first selection unit 220 includes a 1 / 2 2 reduction buffer 221 and a 1 / 42 Reduced buffer 222, and 1 / 8 2 It has a reduced buffer 223.

[0023] For example, 1 / 2 2 The reduced buffer 221 is 1 / 2 2 It retains one row of the reduced-size image data, and 1 / 4 2 The reduced buffer 222 is 1 / 4 2 It retains one row of the reduced-size image data, and is 1 / 8 2 The reduced buffer 223 is 1 / 8 2 Each reduction buffer (221-223) holds one line of reduced image data. Note that each reduction buffer may hold more than one line; multiple lines may also be held.

[0024] The modification unit 230 is a circuit that changes the output timing to the enlargement unit 240 of the reduced image data selected from the first selection unit 220 and the noise-reduced image data selected from the third selection unit 270 when the size of the noise-reduced image data selected from the third selection unit 270 is a specific size. Specifically, the modification unit 230 changes the output timing to the enlargement unit 240 so that the selection timing by the second selection unit 250 differs for multiple enlarged image data of different sizes from the enlargement unit 240.

[0025] For example, the modified part 230 is 1 / 2 2(n-m) The enlarged image data was de-noised by the noise reduction unit 260, resulting in 1 / 2 the original size. 2(n-m) The output of the denoised image data of size (m is an integer greater than or equal to 1 satisfying n < 2m-1) to the magnification unit 240 is controlled to be delayed by (n - (2m-1)) rows of the input image data.

[0026] Specifically, for example, 1 / 2 2(n-1) If the image data is enlarged in size, the modified section 230 is 1 / 2 2(n-1) The output of the noise-reduced image data to the enlargement unit 240 is controlled to be delayed by (n-1) rows of the input image data (m=1). If n=4, the modification unit 230 will be 1 / 8 2The output of the noise-reduced image data to the enlargement unit 240 is delayed by three lines of the input image data.

[0027] Also, 1 / 2 2(n-2) If the image data is enlarged in size, the modified section 230 is 1 / 2 2(n-1) The output of the noise-reduced image data to the enlargement unit 240 is controlled to be delayed by (n-3) rows of the input image data (m=2). If n=4, the modification unit 230 will be 1 / 4 2 The output of the noise-reduced image data to the enlargement unit 240 is delayed by one line of the input image data.

[0028] Also, 1 / 2 2(n-3) If the image data is enlarged in size, the modified section 230 is 1 / 2 2(n-3) The output of the denoised image data to the enlargement unit 240 is controlled to be delayed by (n-5) rows of the input image data (m=3). If n=4, m does not satisfy n<2m-1, so the modification unit 230 is 1 / 2 2 The noise-reduced image data is output to the enlargement unit 240 without changing the output timing.

[0029] In this way, the second selection unit 250 can select the enlarged image data group from the enlargement unit 240 in a time-division manner. Therefore, the noise reduction unit 260 can perform noise reduction on each image data of different sizes at different timings.

[0030] Furthermore, if the size of the noise-reduced image data selected from the third selection unit 270 is not a specific size, the modification unit 230 outputs the reduced image data selected from the first selection unit 220 and the noise-reduced image data selected from the third selection unit 270 to the enlargement unit 240 without changing the output timing to the enlargement unit 240.

[0031] The enlargement unit 240 is a circuit that uses the reduced image data selected from the first selection unit 220 and the denoised image data selected from the third selection unit 270 from the modification unit 230 to enlarge the size of the denoised image data selected from the third selection unit 270, generate enlarged image data, and output it to the second selection unit 250. Specifically, for example, the enlargement unit 240 takes the reduced image data group reduced by the reduction unit 210 and measures it by half. 2(n-1) 1 / 2 of the size 2 Generates a set of enlarged image data up to the specified size.

[0032] The second selection unit 250 is a circuit that time-division selects one of several image data of different sizes. Specifically, the second selection unit 250 selects a second-size image data, which is different from the first size, and outputs it to the noise reduction unit 260, between selecting a first-size image data and outputting it to the noise reduction unit 260.

[0033] For example, the second selection section 250 is 1 / 2 of the reduction section 210. 2n A line buffer holds the reduced-size image data, and the 1 / 2 generated by the enlargement unit 240. 2(n-1) 1 / 2 of the size 2 It has a line buffer that holds each of the enlarged image data up to the specified size.

[0034] Figure 2 is an example where n=4, so the second selection unit 250 is 1 / 16 2 Reduced buffer 251, 1 / 8 2 Expanded buffer 252, 1 / 4 2 Expanded buffer 253 and 1 / 2 2 It has an expanded buffer of 254. 1 / 16 2 The reduction buffer 251 is 1 / 2 of the reduction section 210. 2n It stores one row of reduced-size image data, and 1 / 8 2 The expanded buffer 252 is 1 / 8 2 It retains one row of enlarged image data, and 1 / 4 2 The expanded buffer 253 is 1 / 4 2It retains one row of the enlarged image data, and 1 / 2 2 The expanded buffer 254 is 1 / 2 2 It stores one row of the enlarged image data.

[0035] The noise reduction unit 260 is a circuit that removes noise from the image data selected by the second selection unit 250 in the order of selection. Specifically, for example, the noise reduction unit 260 uses a K × K filter (where K is an integer of 2 or more) to remove noise contained in the image data selected by the second selection unit 250 and generates noise-reduced image data.

[0036] For example, when K=5, the noise reduction unit 260, upon accumulating three consecutive lines of image data in the buffer of the second selection unit 250, interpolates the missing two lines of image data by mirroring them from the three consecutive lines of image data, and then performs filtering using a K×K filter.

[0037] Furthermore, when four consecutive lines of image data are accumulated in the buffer of the second selection unit 250, the missing line of image data is interpolated by mirroring it from the four consecutive lines of image data, and filtering is performed using a K×K filter. When five consecutive lines of image data are accumulated in the buffer of the second selection unit 250, filtering is performed using a K×K filter.

[0038] The noise reduction unit 260 reduces the size of the noise-reduced image data to half its original size. 2 The denoised image data is output to the third selection unit 270 until it reaches the correct size.

[0039] In other words, the noise reduction unit 260 is 1 / 2 2 The denoised image data is output to a circuit inside or outside the image processing device 200 (hereinafter referred to as the output circuit). The output circuit uses the input image data to divide it by 1 / 2. 2The denoised image data is enlarged to generate image data of the same size as the input image data. Then, the output circuit generates denoised image data based on the generated image data and the input image data.

[0040] <Example of the configuration of the reduced section 210> Next, an example configuration of the reduction unit 210 will be described. The reduction unit 210 consists of n input buffers and n reduction processing units (1 / 2 2 Shrinkage processing section ~ 1 / 2 2n It has a reduction processing unit and, that is, the i-th (1≦i≦n) input buffer is 1 / 2 2(i-1) The row-direction image data of the reduction processing unit is sequentially stored, and 1 / 2 2i By reading from the reduction processing unit 322, it is reduced to 1 / 2 2i The output is sent to the reduction processing unit. The following explanation uses n=4 as an example and is shown in Figure 3.

[0041] Figure 3 is an explanatory diagram showing an example configuration of the reduction unit 210. The reduction unit 210 acquires row-direction image data of the input image data 300 row by row in ascending order of row number starting from the first row (row 0), and then reduces it to 1 / 2 2 Reduced image data size 302, 1 / 4 2 Reduced image data size 304, 1 / 8 2 Reduced image data size 308, and 1 / 16 2 Generates a reduced-size image data 316.

[0042] Specifically, for example, the reduction unit 210 includes a first input buffer 311, a second input buffer 312, a third input buffer 313, a fourth input buffer 314, and 1 / 2 2 Reduction processing unit 321 and 1 / 4 2 The reduction processing unit 322 and 1 / 8 2 The reduction processing unit 323 and 1 / 16 2 It has a reduction processing unit 324.

[0043] The first input buffer 311 sequentially holds the row-direction image data of the input image data 300, and divides it into 1 / 2 2 By reading from the reduction processing unit 321, it is reduced to 1 / 22 Output to the reduction processing unit 321.

[0044] The second input buffer 312 is 1 / 2 2 The row-direction image data of the reduction processing unit 321 is sequentially stored, and 1 / 4 2 By reading from the reduction processing unit 322, it is reduced to 1 / 4. 2 Output to the reduction processing unit 322.

[0045] The third input buffer 313 is 1 / 4 2 The row-direction image data of the reduction processing unit 322 is sequentially stored, and 1 / 8 2 Reading from the reduction processing unit 323 reduces the size to 1 / 8. 2 Output to the reduction processing unit 323.

[0046] The fourth input buffer 314 is 1 / 8 2 The row-direction image data of the reduction processing unit 323 is sequentially stored, and 1 / 16 2 Reading from the reduction processing unit 324 reduces the size to 1 / 16. 2 Output to the reduction processing unit 323.

[0047] 1 / 2 2 The reduction processing unit 321 reads two consecutive rows of row-direction image data from the input image data 300 from the first input buffer 311, and divides it by 1 / 2. 2 Generates row-direction image data for one row of the reduced image data 302. Specifically, for example, 1 / 2 2 The reduction processing unit 321 takes the average value of the pixel group in a 2x2 grid in two consecutive rows of row-direction image data of the input image data 300 and divides it by 1 / 2. 2 It is generated as the value of one pixel in the row-direction image data for one row of the reduced image data 302.

[0048] 1 / 2 2 The reduction processing unit 321 reduces the generated 1 / 2 2 The row-direction image data for one row of the reduced image data 302 is divided into 1 / 2 of the second input buffer 312 and the first selection unit 220. 2 Write to shrink buffer 221.

[0049] 1 / 42 The reduction processing unit 322 receives 1 / 2 from the second input buffer 312 2 reads two consecutive rows of row-direction image data from reduced image data 302, and generates 1 / 4 2 one row of row-direction image data for reduced image data 304. Specifically, for example, 1 / 4 2 the reduction processing unit 322 calculates, for 1 / 2 2 two consecutive rows of row-direction image data of reduced image data 302, an average value of pixel values of a 2-row×2-column pixel group (corresponding to a 4-row×4-column pixel group of input image data 300), and uses the average value as 1 / 4 2 a pixel value in one row of row-direction image data of reduced image data 304.

[0050] 1 / 4 2 The reduction processing unit 322 outputs the generated 1 / 4 2 one row of row-direction image data of reduced image data 304 to the third input buffer 313 and the 1 / 4 2 reduction buffer 222 of the first selection unit 220, and writes the data therein.

[0051] 1 / 8 2 The reduction processing unit 323 reads 1 / 4 2 two consecutive rows of row-direction image data of reduced image data 304 from the third input buffer 313, and generates 1 / 8 2 one row of row-direction image data of reduced image data 308. Specifically, for example, 1 / 8 2 the reduction processing unit 323 calculates, for 1 / 4 2 two consecutive rows of row-direction image data of reduced image data 304, an average value of pixel values of a 2-row×2-column pixel group (corresponding to an 8-row×8-column pixel group of input image data 300), and uses the average value as 1 / 8 2 a pixel value in one row of row-direction image data of reduced image data 308.

[0052] 1 / 8 2 The reduction processing unit 323 outputs the generated 1 / 8 2 one row of row-direction image data of reduced image data 308 to the third input buffer 313 and the 1 / 8 2 reduction buffer 223 of the first selection unit 220, and writes the data therein.

[0053] 1 / 16 2 The reduction processing unit 324 takes 1 / 8 of the output from the fourth input buffer 314. 2 Read out row-direction image data for two consecutive rows of the reduced image data 308, and reduce it to 1 / 16 2 Generates row-oriented image data for one row of the reduced image data 316. Specifically, for example, 1 / 16 2 The reduction processing unit 324 is 1 / 8 2 The average value of the pixel values ​​in the 2x2 pixel group (corresponding to the 16x16 pixel group of input image data 300) in the row-direction image data of two consecutive rows of reduced image data 308 is divided by 1 / 16. 2 It is generated as the value of one pixel in the row-direction image data for one row of the reduced image data 316.

[0054] 1 / 16 2 The reduction processing unit 324 generates 1 / 16 2 The row-direction image data for one row of the reduced image data 316 is divided into 1 / 16 of the fourth input buffer 314 and the second selection unit 250. 2 Write to shrink buffer 251.

[0055] <1 / 16 2 Resolution processing > Figure 4 is 1 / 16 2 This is an explanatory diagram showing an example of resolution processing. The reduction unit 210 is 1 / 16 2 The reduction processing unit 324 reduces the size to 1 / 16 2 A row-direction image data for one row of the reduced image data 316 is generated, and 1 / 16 of the second selection unit 250 2 Write to shrink buffer 251 (step S401).

[0056] The reduced portion 210 is 1 / 8 2 The reduction processing unit 323 reduces the scale to 1 / 18 2 A row-direction image data for one row of the reduced image data 308 is generated, and 1 / 8 of the first selection unit 220 2 Write to the reduced buffer 223 (step S402).

[0057] The second selection part 250 is 1 / 16 2Select a reduced buffer of 251, 1 / 16 2 The row-direction image data of the reduced image data 316 is output to the noise reduction unit 260 (step S403).

[0058] The noise reduction unit 260 is 1 / 16 2 The row-direction image data for K rows of the reduced image data 316 is subjected to noise reduction processing, and the noise reduction result is 1 / 16 2 NR image data 416 is 1 / 16 of the third selection unit 270 2 Write to NR buffer 271 (step S404). In this way, 1 / 16 2 Resolution processing reduces the scale to 1 / 16 2 NR image data 416 is generated.

[0059] <1 / 8 2 Resolution processing > Figure 5 is 1 / 8 2 This is an explanatory diagram showing an example of resolution processing. The reduction section 210 is 1 / 4 2 The reduction processing unit 322 reduces the size to 1 / 4. 2 A row-direction image data for one row of the reduced image data 304 is generated, and 1 / 4 of the first selection unit 220 2 Write to the reduced buffer 222 (step S501).

[0060] The first selection section 220 is 1 / 8 2 Select the reduced buffer 223, 1 / 8 2 The row-direction image data of the reduced image data 308 is output to the modification unit 230 (step S502).

[0061] The third selection part 270 is 1 / 16 2 Select NR buffer 271, 1 / 16 2 The row-direction image data of the NR image data 416 is output to the modification unit 230 (step S503).

[0062] Modification 230 is 1 / 8 2 Row-oriented image data of reduced image data 308 and 1 / 16 2The output timing of the row-direction image data of the NR image data 416 is changed and output to the magnification unit 240 (step S504). Specifically, for example, the modification unit 230 delays the output of the input image data 300 by three rows and outputs it to the magnification unit 240.

[0063] The enlarged section 240 is 1 / 8 2 Using row-oriented image data of 308 reduced image data, 1 / 16 2 The size of the row-direction image data for NR image data 416 is reduced to 1 / 8. 2 Enlarged to 1 / 8 2 Row-direction image data is generated from the enlarged image data 508, and 1 / 8 of the second selection unit 250 2 Write to the expanded buffer 252 (step S505).

[0064] The second selection part 250 is 1 / 8 2 Select the expanded buffer 252, 1 / 8 2 The row-direction image data of the reduced image data 308 is output to the noise reduction unit 260 (step S506).

[0065] The noise reduction unit 260 is 1 / 8 2 Noise reduction processing was performed on row-direction image data for K rows of the enlarged image data 508, and the noise reduction processing result is 1 / 8 2 The row-direction image data for one row of NR image data 408 is divided into 1 / 8 of the third selection unit 270. 2 Write to NR buffer 272 (step S507). In this way, 1 / 8 2 Resolution processing reduces the image to 1 / 8 2 NR image data 408 is generated.

[0066] <1 / 4 2 Resolution processing > Figure 6 is 1 / 4 2 This is an explanatory diagram showing an example of resolution processing. The reduction unit 210 is 1 / 2 2 The reduction processing unit 321 reduces the size by half. 2 The row-direction image data for one row of the reduced image data 302 is generated, and 1 / 2 of the first selection unit 220 2 Write to the reduced buffer 221 (step S601).

[0067] The first selection unit 220 is 1 / 4 2 Select the reduced buffer 222, 1 / 4 2 The row-direction image data of the reduced image data 304 is output to the modification unit 230 (step S602).

[0068] The third selection part 270 is 1 / 8 2 Select NR buffer 272, 1 / 8 2 The row-direction image data of the NR image data 408 is output to the modification unit 230 (step S603).

[0069] Modification 230 is 1 / 4 2 Row-oriented image data of reduced image data 304 and 1 / 8 2 The output timing of the row-direction image data of the NR image data 408 is changed and output to the magnification unit 240 (step S604). Specifically, for example, the modification unit 230 delays the output of the input image data 300 by one row and outputs it to the magnification unit 240.

[0070] The enlarged section 240 is 1 / 4 2 Using row-oriented image data of reduced image data 304, 1 / 8 2 The size of the row-direction image data for NR image data 408 is reduced to 1 / 4. 2 Enlarged to 1 / 4 2 Row-direction image data is generated from the enlarged image data 504, and 1 / 4 of the second selection unit 250 2 Write to the expanded buffer 253 (step S605).

[0071] The second selection section 250 is 1 / 4 2 Select the expanded buffer 253, 1 / 4 2 The row-direction image data of the reduced image data 304 is output to the noise reduction unit 260 (step S606).

[0072] The noise reduction unit 260 is 1 / 4 2 Noise reduction processing was performed on row-direction image data for K rows of the enlarged image data 504, and the result of the noise reduction processing is 1 / 4 2The row-direction image data for one row of NR image data 404 is divided into 1 / 4 of the third selection unit 270. 2 Write to NR buffer 273 (step S607). In this way, 1 / 4 2 Resolution processing reduces the size to 1 / 4. 2 NR image data 404 is generated.

[0073] <1 / 2 2 Resolution processing > Figure 7 is 1 / 2 2 This is an explanatory diagram showing an example of resolution processing. The first selection unit 220 is 1 / 2 2 Select the reduced buffer 221, 1 / 2 2 The row-direction image data of the reduced image data 302 is output to the modification unit 230 (step S701).

[0074] The third selection section 270 is 1 / 4 2 Select NR buffer 273, 1 / 4 2 The row-direction image data of the NR image data 404 is output to the modification unit 230 (step S702).

[0075] Modified section 230 is 1 / 2 2 Row-direction image data and 1 / 4 of reduced image data 302 2 The NR image data 404 is output to the magnification unit 240 without changing the timing (step S703).

[0076] The enlarged portion 240 is 1 / 2 2 Using the row-direction image data of reduced image data 302, 1 / 4 2 The size of the row-direction image data for NR image data 404 is halved. 2 Enlarged to 1 / 2 2 Row-direction image data is generated from the enlarged image data 502, and 1 / 2 of the second selection unit 250 2 Write to the expanded buffer 254 (step S704).

[0077] The second selection unit 250 is 1 / 2 2 Select the expanded buffer 254, 1 / 2 2The row-direction image data of the enlarged image data 502 is output to the noise reduction unit 260 (step S705).

[0078] The noise reduction unit 260 is 1 / 2 2 The row-direction image data for K rows of the enlarged image data 502 is subjected to noise reduction processing, and the noise reduction processing result is 1 / 2 2 The row-direction image data for one row of NR image data 402 is output (step S706). In this way, 1 / 2 2 Resolution processing reduces the size by half. 2 NR image data 402 is generated.

[0079] <Time-division noise reduction processing> Next, the 1 / 16 shown in Figures 4 to 7 2 Resolution processing, 1 / 8 2 Resolution processing, 1 / 4 2 Resolution processing and 1 / 2 2 The time-division noise reduction process in resolution processing will be explained using Figures 8 to 14.

[0080] Figures 8 to 14 are explanatory diagrams illustrating examples of time-division noise reduction processing. In Figures 8 to 14, the numbers in the cells of each column represent the row number of the image data indicated by that column. For example, row number 0 in the column for input image data 300 indicates the row-direction image data of row 0 of input image data 300.

[0081] Furthermore, if there are row numbers in the same row, it indicates that they are being processed at the same time (for example, if it is the column of input image data 300, it is written to the reduction unit 210; if it is the columns of reduced image data 302, 304, 308, 316, it is reduced by the reduction unit 210; if it is the columns of NR image data 402, 404, 408, 416, it is processed by the noise reduction unit 260; and if it is the columns of enlarged image data 502, 504, 508, it is processed by the enlargement unit 240).

[0082] The image processing device 200 reads the row-direction image data of the input image data 300 in ascending order of row number. In Figure 8, when the row-direction image data group from row numbers 0 to 15 of the input image data 300 is read in ascending order of row number, 1 / 2 2 A set of row-direction image data from row 0 to row 7 of the reduced image data 302 is generated, and 1 / 4 2 A set of row-direction image data from row 0 to row 3 of the reduced image data 304 is generated, and 1 / 8 2 A set of row-direction image data is generated from row 0 to row 1 of the reduced image data 308, and 1 / 16 2 This indicates that row-direction image data for row 0 of reduced image data 316 has been generated.

[0083] Specifically, for example, 1 / 2 2 This indicates that the row-direction image data for row 0 of the reduced image data 302 is generated from the row-direction image data for rows 0 and 1 of the input image data 300. Similarly, 1 / 2 2 Each of the row-direction image data (Figure 14) from row 1 to row 49 of the reduced image data 302 is generated by the row-direction image data of the same row and the previous row of the input image data 300.

[0084] Also, 1 / 4 2 The row-direction image data for row 0 of reduced image data 304 is 1 / 2 2 This indicates that it is generated from the row-direction image data of row 0 and row 1 of the reduced image data 302. Similarly, 1 / 4 2 Each of the row-direction image data (Figure 14) from row 1 to row 24 of the reduced image data 304 is 1 / 2 2 It is generated from the row-direction image data of the same row and the previous row of the reduced image data 302.

[0085] Also, 1 / 8 2 The row-direction image data for row 0 of the reduced image data 308 is 1 / 4 2 This indicates that it is generated from the row-direction image data of row 0 and row 1 of the reduced image data 304. Similarly, 1 / 8 2Each of the row-direction image data (Figure 14) from row 1 to row 12 of the reduced image data 308 is 1 / 4 2 It is generated from the row-direction image data of the same row and the previous row of the reduced image data 304.

[0086] Also, 1 / 16 2 The row-direction image data for row 0 of reduced image data 316 is 1 / 8 2 This indicates that it is generated from the row-direction image data of row 0 and row 1 of the reduced image data 308. Similarly, 1 / 16 2 Each of the row-direction image data (Figure 14) from row 1 to row 6 of the reduced image data 316 is 1 / 8 2 It is generated from the row-direction image data of the same row and the previous row of the reduced image data 308.

[0087] In Figures 10 to 12, the noise reduction unit 260 generates noise-reduced image data through noise reduction processing. For example, in Figure 10, 1 / 16 2 When three rows of reduced image data 316 (row 0 (Figure 8), row 1 (Figure 9), row 2) are generated, the noise reduction unit 260 mirrors the row-direction image data of row 1 and row 2 to reduce it to 1 / 16 2 The reduced image data 316 is converted into a row-oriented image data group consisting of 5 rows. The noise reduction unit 260 reduces the image to 1 / 16. 2 By filtering the row-oriented image data of 5 rows of the reduced image data 316 with a 5x5 filter, the size is reduced to 1 / 16. 2 Generate row-direction image data for row 0 of NR image data 416.

[0088] Similarly, in Figure 11, 1 / 16 2 When four rows of reduced image data 316 are generated (row 0 (Figure 8), row 1 (Figure 9), row 2 (Figure 10), row 3), the noise reduction unit 260 mirrors the row-direction image data of row 3 and reduces it to 1 / 16 2 The reduced image data 316 is converted into a row-oriented image data group consisting of 5 rows. The noise reduction unit 260 reduces the image to 1 / 16. 2By filtering the row-oriented image data of 5 rows of the reduced image data 316 with a 5x5 filter, the size is reduced to 1 / 16. 2 Generates row-direction image data for the first row of NR image data 416.

[0089] Also, in Figure 12, 1 / 16 2 When five rows of reduced image data 316 (row 0 (Figure 8), row 1 (Figure 9), row 2 (Figure 10), row 3 (Figure 11), row 4) are generated, the noise reduction unit 260 reduces the image to 1 / 16 2 By filtering the row-oriented image data of 5 rows of the reduced image data 316 with a 5x5 filter, the size is reduced to 1 / 16. 2 The row-direction image data for the second row of NR image data 416 is generated. Similarly, in Figures 13 and 14, the 1 / 16 ratio is calculated. 2 Row-direction image data for the 3rd and 4th rows of NR image data 416 is generated.

[0090] Also, 1 / 16 2 When the NR image data 416 is generated, the enlargement unit 240 is delayed by three lines of the input image data 300 by the modification unit 230, and then scaled down to 1 / 8. 2 Generate enlarged image data 508.

[0091] Specifically, for example, in Figure 10, 1 / 16 2 When the row-direction image data for the 0th row of the NR image data 416 is generated, the enlargement unit 240 is delayed by 3 rows of the input image data 300 by the modification unit 230, as shown in Figure 11, and scaled to 1 / 8. 2 Generate row-direction image data for row 0 of enlarged image data 508.

[0092] Similarly, in Figure 11, 1 / 16 2 When the row-direction image data for the first row of the NR image data 416 is generated, the enlargement unit 240 is delayed by three rows of the input image data 300 by the modification unit 230, as shown in Figure 12, and scaled to 1 / 8 2 Generates row-direction image data for the second row of enlarged image data 508.

[0093] Similarly, in Figure 12, 1 / 16 2 When the row-direction image data for the second row of the NR image data 416 is generated, the enlargement unit 240 is delayed by three rows of the input image data 300 by the modification unit 230, as shown in Figure 13, and scaled to 1 / 8. 2 Generates row-direction image data for the 4th row of enlarged image data 508.

[0094] Similarly, in Figure 13, 1 / 16 2 When the row-direction image data for the third row of the NR image data 416 is generated, the enlargement unit 240 is delayed by three rows of the input image data 300 by the modification unit 230, as shown in Figure 14, and scaled to 1 / 8. 2 Generate row-direction image data for the 6th row of the enlarged image data 508.

[0095] Note: 1 / 8 2 The row-direction image data for odd-numbered rows of enlarged image data 508 is 1 / 8 2 The enlarged image data is generated between the generation of row-direction image data for consecutive even-numbered rows of 508 images.

[0096] Furthermore, in Figures 12 to 14, the noise reduction unit 260 generates noise-reduced image data through noise reduction processing. For example, in Figure 12, 1 / 8 2 When three rows of enlarged image data 508 are generated (row 0 (Figure 11), row 1 (Figure 11), row 2), the noise reduction unit 260 mirrors the row-direction image data of row 1 and row 2 to reduce it to 1 / 8 2 The enlarged image data 508 is converted into a row-direction image data group consisting of 5 rows. The noise reduction unit 260 is 1 / 8 2 By filtering the row-oriented image data of 5 rows of the enlarged image data 508 with a 5x5 filter, the image is reduced to 1 / 8. 2 Generates row-direction image data for row 0 of NR image data 408.

[0097] Similarly, in Figure 12, 1 / 8 2 When four rows of enlarged image data 508 are generated (row 0 (Figure 11), row 1 (Figure 11), row 2, row 3), the noise reduction unit 260 mirrors the row-direction image data of row 3 and reduces it to 1 / 82 The enlarged image data 508 is converted into a row-direction image data group consisting of 5 rows. The noise reduction unit 260 is 1 / 8 2 By filtering the row-oriented image data of 5 rows of the enlarged image data 508 with a 5x5 filter, the image is reduced to 1 / 8. 2 Generates row-direction image data for the first row of NR image data 408.

[0098] Similarly, in Figure 13, 1 / 8 2 When five rows of enlarged image data 508 are generated (row 0 (Figure 11), row 1 (Figure 11), row 2 (Figure 12), row 3 (Figure 12), row 4), the noise reduction unit 260 reduces the image by 1 / 8. 2 By filtering the row-oriented image data of 5 rows of the enlarged image data 508 with a 5x5 filter, the image is reduced to 1 / 8. 2 The row-direction image data for the second row of NR image data 408 is generated. Similarly, in Figures 13 and 14, 1 / 8 each are generated. 2 The row-direction image data for rows 3 through 5 of NR image data 408 is generated.

[0099] Also, 1 / 8 2 When the NR image data 408 is generated, the enlargement unit 240 is delayed by one line of the input image data 300 by the modification unit 230, and is reduced to 1 / 4. 2 Generate enlarged image data 504.

[0100] Specifically, for example, in Figure 12, 1 / 8 2 When the row-direction image data for the 0th row of the NR image data 408 is generated, the enlargement unit 240 is delayed by one row of the input image data 300 by the modification unit 230, and is reduced to 1 / 4. 2 Generate row-direction image data for row 0 of enlarged image data 504.

[0101] Similarly, in Figure 12, 1 / 8 2 When the first row of the NR image data 408 is generated, the enlargement unit 240 is delayed by one row of the input image data 300 by the modification unit 230, and is reduced to 1 / 4. 2 Generate row-direction image data for the second row of enlarged image data 504.

[0102] Similarly, in Figure 13, 1 / 8 2 When the row-direction image data for the second row of NR image data 408 is generated, the enlargement unit 240 is delayed by one row of the input image data 300 by the modification unit 230, and is reduced to 1 / 4. 2 Generate row-direction image data for the 4th row of enlarged image data 504.

[0103] Similarly, in Figure 13, 1 / 8 2 When the row-direction image data for the third row of the NR image data 408 is generated, the enlargement unit 240 is delayed by one row of the input image data 300 by the modification unit 230, and is reduced to 1 / 4. 2 Generate row-direction image data for the 6th row of enlarged image data 504.

[0104] Similarly, in Figure 14, 1 / 8 2 When the row-direction image data for the fourth row of the NR image data 408 is generated, the enlargement unit 240 is delayed by one row of the input image data 300 by the modification unit 230, and is reduced to 1 / 4. 2 Generate row-direction image data for row 8 of enlarged image data 504.

[0105] Similarly, in Figure 14, 1 / 8 2 When the row-direction image data for the 5th row of the NR image data 408 is generated, the enlargement unit 240 is delayed by one row of the input image data 300 by the modification unit 230, and is reduced to 1 / 4. 2 Generate row-direction image data for the 10th row of enlarged image data 504.

[0106] Note 1 / 4 2 The row-direction image data for odd-numbered rows of enlarged image data 504 is 1 / 4 2 The enlarged image data 504 is generated between the generation of row-direction image data for consecutive even-numbered rows.

[0107] Furthermore, in Figures 12 to 14, the noise reduction unit 260 generates noise-reduced image data through noise reduction processing. For example, in Figure 12, 1 / 4 2When three rows of enlarged image data 504 (row 0, row 1, row 2) are generated, the noise reduction unit 260 mirrors the row-direction image data of row 1 and row 2, reducing it to 1 / 4. 2 The enlarged image data 504 is converted into a row-direction image data group consisting of 5 rows. The noise reduction unit 260 is 1 / 4 2 By filtering the row-oriented image data of 5 rows of the enlarged image data 504 with a 5x5 filter, the image is reduced to 1 / 4. 2 Generates row-direction image data for row 0 of NR image data 404.

[0108] Similarly, in Figure 13, 1 / 4 2 When four rows of enlarged image data 504 are generated (row 0 (Figure 12), row 1 (Figure 12), row 2 (Figure 12), row 3), the noise reduction unit 260 mirrors the row-direction image data of row 3 and reduces it to 1 / 4. 2 The enlarged image data 504 is converted into a row-direction image data group consisting of 5 rows. The noise reduction unit 260 is 1 / 4 2 By filtering the row-oriented image data of 5 rows of the enlarged image data 504 with a 5x5 filter, the image is reduced to 1 / 4. 2 Generates row-direction image data for the first row of NR image data 404.

[0109] Similarly, in Figure 13, 1 / 4 2 When five rows of enlarged image data 504 are generated (row 0 (Figure 12), row 1 (Figure 12), row 2 (Figure 12), row 3, row 4), the noise reduction unit 260 reduces the noise by 1 / 4. 2 By filtering the row-oriented image data of 5 rows of the enlarged image data 504 with a 5x5 filter, the image is reduced to 1 / 4. 2 The row-direction image data for the second row of NR image data 404 is generated. Similarly, in Figures 13 and 14, 1 / 4 each are generated. 2 Row-direction image data for rows 3 through 8 of NR image data 404 is generated.

[0110] Also, 1 / 4 2 When the NR image data 404 is generated, the enlargement unit 240 is reduced to 1 / 2 without any timing change by the modification unit 230. 2Generate enlarged image data 502.

[0111] Specifically, for example, in Figure 12, 1 / 4 2 When row-direction image data for row 0 of NR image data 404 is generated, the magnification unit 240 is 1 / 2 2 Generate row-direction image data for row 0 of enlarged image data 502.

[0112] Similarly, in Figure 13, 1 / 4 2 When the row-direction image data for the first row of NR image data 404 is generated, the magnification unit 240 is 1 / 2 2 Generate row-direction image data for the second row of enlarged image data 502.

[0113] Similarly, in Figure 13, 1 / 4 2 When the row-direction image data for the second row of the NR image data 404 is generated, the magnification unit 240 is 1 / 2 2 Generate row-direction image data for the 4th row of enlarged image data 502.

[0114] Similarly, in Figure 13, 1 / 4 2 When the row-direction image data for the third row of NR image data 404 is generated, the magnification unit 240 is 1 / 2 2 Generate row-direction image data for the 6th row of enlarged image data 502.

[0115] Similarly, in Figure 13, 1 / 4 2 When the row-direction image data for the fourth row of the NR image data 404 is generated, the magnification unit 240 is 1 / 2 2 Generate row-direction image data for row 8 of enlarged image data 502.

[0116] Similarly, in Figure 14, 1 / 4 2 When the row-direction image data for the 5th row of NR image data 404 is generated, the magnification unit 240 is 1 / 2 2 Generate row-direction image data for the 10th row of enlarged image data 502.

[0117] Similarly, in Figure 14, 1 / 4 2When row-direction image data for the 6th row of NR image data 404 is generated, the magnification unit 240 is 1 / 2 2 Generate row-direction image data for the 12th row of enlarged image data 502.

[0118] Similarly, in Figure 14, 1 / 4 2 When row-direction image data for the 7th row of NR image data 404 is generated, the magnification unit 240 is 1 / 2 2 Generate row-direction image data for row 14 of enlarged image data 502.

[0119] Similarly, in Figure 14, 1 / 4 2 When row-direction image data for the 8th row of NR image data 404 is generated, the magnification unit 240 is 1 / 2 2 Generate row-direction image data for row 16 of enlarged image data 502.

[0120] Note 1 / 2 2 The row-direction image data for odd-numbered rows of enlarged image data 502 is 1 / 2 2 The enlarged image data 502 is generated between the generation of row-direction image data for consecutive even-numbered rows.

[0121] Furthermore, in Figures 13 and 14, the noise reduction unit 260 generates noise-reduced image data through noise reduction processing. For example, in Figure 13, 1 / 2 2 When three rows of enlarged image data 502 (row 0 (Figure 12), row 1, row 2) are generated, the noise reduction unit 260 mirrors the row-direction image data of row 1 and row 2 to 1 / 2 2 The enlarged image data 502 is converted into a row-direction image data group consisting of 5 rows. The noise reduction unit 260 is 1 / 2 2 By filtering the row-oriented image data of 5 rows of the enlarged image data 502 with a 5x5 filter, the image is reduced to 1 / 2. 2 Generate row-direction image data for row 0 of NR image data 402.

[0122] Similarly, in Figure 13, 1 / 2 2When four rows of enlarged image data 502 (row 0 (Figure 12), row 1, row 2, row 3) are generated, the noise reduction unit 260 mirrors the row-direction image data of row 3 and divides it by 1 / 2. 2 The enlarged image data 502 is converted into a row-direction image data group consisting of 5 rows. The noise reduction unit 260 is 1 / 2 2 By filtering the row-oriented image data of 5 rows of the enlarged image data 502 with a 5x5 filter, the image is reduced to 1 / 2. 2 Generates row-direction image data for the first row of NR image data 402.

[0123] Similarly, in Figure 13, 1 / 2 2 When five rows of enlarged image data 502 (row 0 (Figure 12), row 1, row 2, row 3, row 4) are generated, the noise reduction unit 260 reduces the noise by half. 2 By filtering the row-oriented image data of 5 rows of the enlarged image data 502 with a 5x5 filter, the image is reduced to 1 / 2. 2 The row-direction image data for the second row of NR image data 402 is generated. Similarly, in Figures 13 and 14, 1 / 2 each are generated. 2 Row-direction image data for rows 3 through 14 of NR image data 402 is generated.

[0124] Referring to the noise reduction process in Figures 11 to 14, in the input image data 300, from line 63 to line 111, no NR image data is generated on the same line. In this way, a single noise reduction unit 260 can perform time-division noise reduction processing on multiple image data of different sizes (resolutions).

[0125] Thus, the image processing device 200 does not need to provide a noise reduction unit 260 and an enlargement unit 240 for each of the multiple reduction processing units 321 to 324, and the noise reduction unit 260 and the enlargement unit 240 can be shared among the multiple reduction processing units 321 to 324. In other words, the number of noise reduction units 260 is less than the number of multiple reduction processing units 321 to 324 (the number of types of reduced image data sizes). Therefore, the circuit size of the image processing device 200 can be reduced and costs can be reduced by reducing the number of parts. Compared to simply performing noise reduction on the first size image data and then on the second size image data, the processing time can be shortened.

[0126] It should be noted that the present invention is not limited to the above, and may be any combination thereof. Furthermore, other embodiments that can be conceivable within the scope of the technical idea of ​​the present invention are also included in the scope of the present invention. [Explanation of symbols]

[0127] 100 Imaging device, 200 Image processing device, 210 Reduction unit, 220 First selection unit, 230 Modification unit, 240 Enlargement unit, 250 Second selection unit, 260 Noise reduction unit, 270 Third selection unit

Claims

1. 1 / 2 of the input image data ２ 1 / 2 of the size ２ｎ A reduction unit that generates a set of reduced image data up to the size (where n is an integer of 1 or more), For the group of reduced image data reduced by the reduction unit, 1 / 2 ２（ｎ－１） 1 / 2 of the size ２ An enlargement unit that generates a set of enlarged image data up to the specified size, The aforementioned 1 / 2 ２ｎ Image data with reduced size and 1 / 2 scale generated by the enlargement unit. ２（ｎ－１） 1 / 2 of the size ２ A selection unit that selects one of the image data from a set of enlarged image data up to the specified size using time-division multiplexing, A noise reduction unit that performs noise reduction processing on the image data selected by the selection unit in the order of selection, wherein the size of the image data after the noise reduction processing is 1 / 2 ２ A noise reduction unit outputs the image data that has undergone the noise reduction process to the enlargement unit until it reaches the desired size. An image processing device having

2. An image processing apparatus according to claim 1, 1 / 2 ２（ｎ－ｍ） 1 / 2 size enlarged image data that has been subjected to the noise removal processing by the noise removal unit ２（ｎ－ｍ） a changing unit configured to control to delay the output of noise-removed image data of size (where m is an integer of 1 or more satisfying n < 2m-1) to the enlargement unit by (n-(2m-1)) lines of the input image data; An image processing device having

3. An image processing apparatus according to claim 1 or claim 2, The selection unit is an image processing device having (n-1) buffers that hold each of the reduced image data groups.

4. An image processing apparatus according to claim 3, The selection unit is an image processing apparatus having (n-1) line buffers that hold each of the reduced image data groups.

5. An image processing apparatus according to any one of claims 1 to 4, The noise reduction unit is an image processing device that performs the noise reduction process using a K × K filter (where K is an integer of 2 or more).

6. An imaging apparatus having an image processing apparatus according to any one of claims 1 to 5.

7. The imaging device according to claim 6, It has an image sensor in which multiple light-receiving elements are arranged, The reduction unit is an imaging device that inputs image data based on the signal output from the image sensor as the input image data.

8. The imaging apparatus according to claim 7, An imaging device having a drive unit that controls the drive of an optical system that emits light to the image sensor.

9. The imaging apparatus according to claim 8, An imaging device having the aforementioned optical system.

Images