Imaging device

Abstract

To achieve high accuracy in noise reduction.SOLUTION: An imaging apparatus includes an imaging unit that images a subject and outputs first image data and second image data that is temporally later than the first image data, a calculation unit that calculates a first weight of the first image data on the basis of first noise-removed image data from which noise has been removed from the first image data and the second image data, an afterimage reduction processing unit that reduces afterimages in the second image data, and a removing unit that removes noise from the second image data that has been subjected to afterimage reduction processing by the afterimage reduction processing unit, on the basis of the first noise-removed image data and the first weight calculated by the calculation unit.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to an imaging device.

Background Art

[0002] As a noise removal method for video, there is a weighted average process of a past frame and a current frame (see, for example, Patent Document 1 below). However, the noise reduction system of Patent Document 1 must hold more past frames in an image memory to improve accuracy, but there is also a limit to the amount of memory used in the image memory. On the other hand, if the amount of memory used is limited, an improvement in the accuracy of noise reduction cannot be expected.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

[0004] An imaging device according to a first aspect of the invention disclosed in the present application includes an imaging unit that images a subject and outputs first image data, second image data that is temporally later than the first image data, and third image data that is temporally later than the second image data; a calculation unit that calculates a first weight of the first image data based on first noise-removed image data obtained by removing noise from the first image data and the second image data, calculates a second weight of the second image data based on second noise-removed image data obtained by removing noise from the second image data and the third image data, and calculates a third weight based on the first weight and the second weight; and a removal unit that removes noise from the third image data based on the third weight calculated by the calculation unit. Then, the afterimage reduction processing unit synthesizes the first image data and the second image data, and interpolates between the subject image in the first image data and the subject image in the second image data in the synthesized image data.

[0005] The imaging device according to the second aspect of the invention disclosed in the present application includes an imaging unit that images a subject and outputs first image data and second image data that is temporally later than the first image data, and noise is removed from the first image data Output to the display device The first noise-removed image data and the second image data To Based on this, a calculation unit that calculates a first weight of the first image data; An afterimage reduction processing unit that reduces the afterimage in the second image data; Based on the first noise-removed image data and the first weight calculated by the calculation unit, After the afterimage reduction processing by the afterimage reduction processing unit A removal unit that removes noise from the second image data.

[0006] The imaging device according to the third aspect of the invention disclosed in the present application includes an imaging unit that images a subject and outputs first image data and second image data that is temporally later than the first image data, and a calculation unit that calculates a first weight of the first image data based on the first noise-removed image data obtained by removing noise from the first image data and the second image data , the Based on the first noise-removed image data and the first weight calculated by the calculation unit, , the A removal unit that removes noise from the second image data; An afterimage reduction processing unit that reduces the afterimage in the second noise-removed image data obtained by removing noise from the second image data by the removal unit; It has.

[0007] The imaging device according to the fourth aspect of the invention disclosed in the present application includes an imaging unit that images a subject and outputs first image data and second image data that is temporally later than the first image data, and noise is removed from the first image data Output to the display device The first noise-removed image data and a calculation unit that calculates a first weight of the first image data based on the second image data, and a removal unit that removes noise from the second image data based on the first noise-removed image data and the first weight calculated by the calculation unit, and a residual image reduction processing unit that reduces a residual image in the second noise-removed image data obtained by removing noise from the second image data by the removal unit.

[0008] The imaging device according to the fifth aspect of the invention disclosed in the present application includes an input unit that receives an input of a frame rate when imaging a subject, the A first corresponding to the frame rate input by the input unit From the light amount information The first Light amount information Rather than Large A second Light amount information An adjustment unit that adjusts to, and a second adjusted by the frame rate and the adjustment unit Light amount information Based on this, an imaging unit that images the subject and outputs first image data and second image data that is temporally later than the first image data, and noise is removed from the first image data Output to the display device A calculation unit that calculates a first weight of the first image data based on the obtained first noise-removed image data and the second image data, and based on the first noise-removed image data and the first weight calculated by the calculation unit, a removal unit that removes noise from the second image data.

[0009] An image processing apparatus according to a sixth aspect of the invention disclosed in the present application includes an acquisition unit that acquires first image data, second image data captured temporally after the first image data, and third image data captured temporally after the second image data; a calculation unit that calculates a first weight of the first image data based on the first noise-removed image data obtained by removing noise from the first image data and the second image data, calculates a second weight of the second image data based on the second noise-removed image data obtained by removing noise from the second image data and the third image data, and calculates a third weight based on the first weight and the second weight; and a removal unit that removes noise from the third image data based on the third weight calculated by the calculation unit.

[0010] An image processing apparatus according to a seventh aspect of the invention disclosed in the present application includes an acquisition unit that acquires first image data and second image data captured later in time than the first image data, an extraction unit that extracts first spatial information indicating a feature of a spatial structure included in an image from at least one of first noise-removed image data obtained by removing noise from the first image data and the second image data, a calculation unit that calculates a first weight of the first image data based on the first spatial information extracted by the extraction unit, and a removal unit that removes noise from the second image data based on the first noise-removed image data and the first weight calculated by the calculation unit.

[0011] An image processing apparatus according to an eighth aspect of the invention disclosed in the present application includes an acquisition unit that acquires first image data and second image data captured later in time than the first image data, a calculation unit that calculates a first weight of the first image data based on the first noise-removed image data obtained by removing noise from the first image data and the second image data, a residual image reduction processing unit that reduces a residual image in the second image data, and a removal unit that removes noise from the second image data that has been subjected to the residual image reduction processing by the residual image reduction processing unit based on the first noise-removed image data and the first weight calculated by the calculation unit.

[0012] An image processing apparatus according to a ninth aspect of the invention disclosed in the present application includes an acquisition unit that acquires first image data and second image data captured later in time than the first image data, a calculation unit that calculates a first weight of the first image data based on the first noise-removed image data obtained by removing noise from the first image data and the second image data, a removal unit that removes noise from the second image data based on the first noise-removed image data and the first weight calculated by the calculation unit, and a residual image reduction processing unit that reduces a residual image in second noise-removed image data obtained by removing noise from the second image data by the removal unit.

[0013] The image processing program according to the tenth aspect of the invention disclosed in the present application causes a processor to execute an acquisition process of acquiring first image data, second image data captured temporally after the first image data, and third image data captured temporally after the second image data, a calculation process of calculating a first weight of the first image data based on first noise-removed image data obtained by removing noise from the first image data and the second image data, calculating a second weight of the second image data based on second noise-removed image data obtained by removing noise from the second image data and the third image data, and calculating a third weight based on the first weight and the second weight, and a removal process of removing noise from the third image data based on the third weight calculated by the calculation process.

[0014] The image processing program according to the eleventh aspect of the invention disclosed in the present application causes a processor to execute an acquisition process of acquiring first image data and second image data captured temporally after the first image data, an extraction process of extracting first spatial information indicating a feature of a spatial structure included in an image from at least one of the first noise-removed image data obtained by removing noise from the first image data and the second image data, a calculation process of calculating a first weight of the first image data based on the first spatial information extracted by the extraction process, and a removal process of removing noise from the second image data based on the first noise-removed image data and the first weight calculated by the calculation process.

[0015] The image processing program according to the twelfth aspect of the invention disclosed in the present application causes a processor to execute an acquisition process of acquiring first image data and second image data captured later in time than the first image data, a calculation process of calculating a first weight of the first image data based on the first noise-removed image data obtained by removing noise from the first image data and the second image data, a residual image reduction process of reducing a residual image in the second image data, and a removal process of removing noise from the second image data that has been subjected to the residual image reduction process by the residual image reduction process, based on the first noise-removed image data and the first weight calculated by the calculation process.

[0016] The image processing program according to the thirteenth aspect of the invention disclosed in the present application causes a processor to execute an acquisition process of acquiring first image data and second image data captured later in time than the first image data, a calculation process of calculating a first weight of the first image data based on the first noise-removed image data obtained by removing noise from the first image data and the second image data, a removal process of removing noise from the second image data based on the first noise-removed image data and the first weight calculated by the calculation process, and a residual image reduction process of reducing a residual image in the second noise-removed image data obtained by removing noise from the second image data by the removal process.

Brief Description of Drawings

[0017]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

DETAILED DESCRIPTION OF THE INVENTION

Example

[0018] Example 1 will be described. As a noise removal method for video, there is a weighted average process between noise-removed past image data and current image data that has not been noise-removed. In order to improve the noise removal accuracy, it is necessary to hold a large number of past frame numbers in the memory, but the memory usage will increase.

[0019] Therefore, in Example 1, in order to suppress the number of noise-removed image data to the minimum necessary, the imaging device recursively uses the weights used for noise removal to improve the noise removal accuracy and reduce the memory usage.

[0020] The weight is the reliability of the preceding image data with respect to the succeeding image data for each region when the temporally consecutive preceding image data and succeeding image data in a moving image are compared in the same region. A region is a set of pixels of one or more pixels. The region may be the entire image data.

[0021] The weight takes, for example, a range from 0.0 to 1.0, and the larger the value, the higher the reliability. The more similar the same regions of the preceding image data and the succeeding image data are, the larger the value of the weight becomes. By recursively using the weight, the value of the weight becomes smaller for the afterimage of the past image data, and the afterimage is reduced. Hereinafter, Example 1 will be described in detail.

[0022] <Hardware configuration example of the imaging device> FIG. 1 is a block diagram showing a hardware configuration example of the imaging device. The imaging device 100 is a device capable of shooting video. Specifically, for example, it is a digital camera, a digital video camera, a smartphone, a tablet, a personal computer, or a game machine. In FIG. 1, a digital camera is taken as an example of the imaging device for description.

[0023] The imaging device 100 includes a processor 101, a memory device 102, a drive unit 103, an optical system 104, an imaging element 105, an AFE (Analog Front End) 106, an LSI (Large Scale Integration) 107, an operation device 108, a sensor 109, a display device 110, a communication IF (Interface) 111, and a bus 112. The processor 101, the memory device 102, the drive unit 103, the LSI 107, the operation device 108, the sensor 109, the display device 110, and the communication IF 111 are connected to the bus 112.

[0024] The processor 101 controls the imaging device 100. The memory device 102 serves as a working area for the processor 101. Also, the memory device 102 is a non-temporary or temporary recording medium that stores various programs and data. Examples of the memory device 102 include a ROM (Read Only Memory), a RAM (Random Access Memory), an HDD (Hard Disk Drive), and a flash memory. A plurality of memory devices 102 may be mounted on the imaging device 100, and at least one of them may be detachable from the imaging device 100.

[0025] The drive unit 103 drives and controls the optical system 104. The drive unit 103 includes a drive circuit 103a and a drive source 103b. The drive circuit 103a controls the drive source 103b according to an instruction from the processor 101. The drive source 103b is, for example, a motor, and under the control of the drive circuit 103a, it moves the zooming lens 141b and the focusing lens 141c in the optical axis direction within the optical system 104, or controls the opening and closing of the aperture 142.

[0026] The optical system 104 includes a plurality of lenses (lens 141a, zooming lens 141b, and focusing lens 141c) arranged in the optical axis direction and an aperture 142. The optical system 104 collects subject light and emits it to the imaging element 105.

[0027] The imaging device 105 receives subject light from the optical system 104 and converts it into an electrical signal. The imaging device 105 may be, for example, a solid-state imaging device of the XY address type (e.g., CMOS (Complementary Metal-Oxide Semiconductor)), or may be a solid-state imaging device of the sequential scanning type (e.g., CCD (Charge Coupled Device)).

[0028] On the light-receiving surface of the imaging device 105, a plurality of light-receiving elements (pixels) are arranged in a matrix. And for the pixels of the imaging device 105, a plurality of types of color filters that transmit light of different color components are arranged according to a predetermined color arrangement (e.g., Bayer arrangement). Therefore, each pixel of the imaging device 105 outputs an analog electrical signal corresponding to each color component by color separation with the color filter.

[0029] AFE 106 is an analog front-end circuit that performs signal processing on the analog electrical signal from the imaging device 105. AFE 106 sequentially executes gain adjustment of the electrical signal, analog signal processing (correlated double sampling, black level correction, etc.), A / D conversion processing, digital signal processing (defective pixel correction, etc.) to generate RAW image data and output it to the LSI. The above-described driving unit 103, optical system 104, imaging device 105, and AFE 106 constitute the imaging unit 120.

[0030] LSI 107 is an integrated circuit that performs specific processing such as image processing (color interpolation, white balance adjustment, edge enhancement, gamma correction, tone conversion, etc.), encoding processing, decoding processing, compression / decompression processing, etc. on the RAW image data from AFE 106. Specifically, LSI 107 may be realized by a PLD (Programmable Logic Device) such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array), for example.

[0031] The operation device 108 inputs commands and data. Examples of the operation device 108 include various buttons including a release button, switches, dials, and touch panels. A sensor is a device that detects information, and examples thereof include an AF (Automatic Focus) sensor, an AE (Automatic Exposure) sensor, a gyro sensor, an acceleration sensor, and a temperature sensor. The display device 110 displays image data and a setting screen. The display device 110 includes a rear monitor on the back of the imaging device 100 and an electronic viewfinder. The communication IF 111 connects to a network and transmits and receives data.

[0032] <Functional configuration example of the imaging device 100> FIG. 2 is a block diagram showing a functional configuration example of the imaging device 100 according to Embodiment 1. The imaging device 100 includes an acquisition unit 201, a calculation unit 202, a removal unit 203, an image storage unit 204, and a first memory 205. The acquisition unit 201, the calculation unit 202, the removal unit 203, the image storage unit 204, and the first memory 205 constitute an image processing device 200.

[0033] The acquisition unit 201 acquires a series of temporally continuous image data captured by the imaging unit 120. The acquisition unit 201 is, for example, a buffer memory which is one of the storage devices 102. When the imaging device 100 is an image processing device 200 without the imaging unit 120, a series of image data is acquired from the storage device 102 or the communication IF 111. The acquisition unit 201 is, for example, a buffer memory which is one of the storage devices 102.

[0034] Specifically, the calculation unit 202, the removal unit 203, and the image storage unit 204 are realized, for example, by causing the processor 101 to execute a program stored in the storage device 102 shown in FIG. 1 or by the LSI 107. The first memory 205 is one of the storage devices 102.

[0035] Here, a series of image data will be described by taking the consecutive first to third image data as an example. It is assumed that the imaging unit 120 captures a subject and outputs the first to third image data in this order. The first image data is the oldest image data output from the imaging unit 120 among the consecutive first to third image data in terms of time. The second image data is the image data output from the imaging unit 120 after the first image data in terms of time. The third image data is the image data output from the imaging unit 120 after the second image data in terms of time.

[0036] The calculation unit 202 includes a weight calculation unit 221, a weight adjustment unit 222, a weight storage unit 223, and a second memory 224. The weight calculation unit 221 calculates a first weight of the first image data based on the first noise-removed image data obtained by removing noise from the first image data by the removal unit 203 and the second image data. When calculating the first weight, the calculation unit 202 reads out the first noise-removed image data from the first memory 205 as past image data preceding the second image data. The first image data is the preceding image data of the second image data, and the second image data is the subsequent image data of the first image data.

[0037] In addition, the calculation unit 202 calculates a second weight of the second image data based on the second noise-removed image data obtained by removing noise from the second image data by the removal unit 203 and the third image data. When calculating the second weight, the calculation unit 202 reads out the second noise-removed image data from the first memory 205 as past image data preceding the third image data. The second image data is the preceding image data of the third image data, and the third image data is the subsequent image data of the second image data.

[0038] As described above, the weight is the reliability of the preceding image data with respect to the subsequent image data for each region when the preceding and subsequent image data that are temporally continuous are compared in the same region. A region is a set of pixels of one or more pixels. The region may be the entire image data.

[0039] Here, the image data that has been noise-removed before time t is denoted as Iold, and the image data at time t is denoted as It. When not distinguishing between Iold and It, it is simply denoted as image data I. The image data I is, for example, a matrix corresponding to the values of each pixel in a matrix-like pixel group. Therefore, the weights are also expressed by a matrix and denoted as W. Each element of the weight W corresponds to the region (x, y) in the image data I, and the weight w of that region takes a value in the range of, for example, 0.0 or more and 1.0 or less.

[0040] In a certain region, the higher the weight (reliability) w, the less blurring of the image, that is, the fewer afterimages, between the preceding image data and the subsequent image data in that region, and the lower the weight (reliability), the more blurring, that is, the more afterimages, between the preceding image data and the subsequent image data in that region.

[0041] The weight w of the region in the preceding image data Iold with respect to the subsequent image data at time t is calculated by the following formula (1).

[0042] w = f(It, Iold) ··· (1)

[0043] The function f(·) is a function for obtaining the weight w of the region in the preceding image data Iold. If the region is the entire preceding image data Iold, the weight w becomes one value. Also, the weight calculation unit 221 may obtain a plurality of weights W according to the number of channels of the preceding image data Iold. The weight w of the region in the preceding image data Iold should be made larger as the similarity of the subsequent image data It is larger. As an example of how to obtain the weight w of the region, there is the following formula (2).

[0044]

Equation

[0045] The function f(x, y) means calculating the weight w of the region (x, y). x indicates the position in the column direction of the region of the previous image data Iold, and y indicates the position in the row direction. Also, the function φ(x, y) is a function for obtaining partial image data of the neighborhood region centered on the region (x, y). Also, ||·|| means the L2 norm. Also, σ is a parameter for adjusting the calculation result of the weight. σ is a parameter for adjusting the sensitivity to the difference of the image which is the value of the numerator of the above formula (2), and when the value of σ is increased, even if the difference of the image is large, the weight will be calculated large.

[0046] The weight adjustment unit 222 adjusts the weight W calculated by the weight calculation unit 221 based on the latest weight Wold stored in the second memory 224. Let the adjusted weight W be the weight W'. The weight storage unit 223 stores the weight W before adjustment by the weight adjustment unit 222 as Wold in the second memory 224 (see formula (4)). The second memory 224 is constituted by the storage device 102.

[0047] The adjusted weight W' is calculated by the following formula (3).

[0048] W' = g(W, Wold) ··· (3)

[0049] The function g(·) is a function for adjusting the weight W. Wold is updated by the weight W before adjustment as shown in the following formula (4). Here, the weight W is directly updated to Wold, but the size of the weight W may be reduced and updated to Wold. The weight is referred to as the past weight Wold.

[0050] Wold = W ··· (4)

[0051] Also, the right side of formula (3) is expressed, for example, by the following formula (5).

[0052]

Equation

[0053] Further, an upper limit or a lower limit may be set for the adjusted weight W', or a uniform adjustment coefficient may be multiplied. Further, the right side of Equation (3) may be expressed, for example, by the following Equation (6).

[0054] g(W, Wold) = min(W, Wold) ··· (6)

[0055] In the above Equation (6), the weight W and the past weight Wold are compared element by element, and the weight w with the smaller value is adopted. In this way, the weight w' in the region of the adjusted weight W' becomes smaller than the weight w before adjustment.

[0056] Here, taking the above-described first image data to third image data as an example, the adjustment of the weight W will be described. When the first image data is used as the preceding image data and the first noise-removed image data obtained by removing noise from the first image data is used as the past image Iold, the weight W is defined as the weight W1. When the second image data is used as the preceding image data and the second noise-removed image data obtained by removing noise from the second image data is used as the past image Iold, the weight W is defined as the weight W2. When the third image data is used as the preceding image data and the third noise-removed image data obtained by removing noise from the third image data is used as the past image Iold, the weight W is defined as the weight W3.

[0057] When the first image data is the first image data of the moving image data, since there is no image data preceding the first image data, the weight W1 is not adjusted by the weight adjustment unit 222 but is stored as the past weight Wold in the second memory 224 by the weight storage unit 223. When the weight W2 is calculated using the first image data and the second image data, the weight adjustment unit 222 reads the past weight Wold (= weight W1) from the second memory 224 and multiplies it by the weight W2 according to the above Equation (5) to calculate the adjusted weight W2'. The weight storage unit 223 stores the weight W2 before adjustment as the past weight Wold in the second memory 224.

[0058] Also, when the weight W3 is calculated from the second image data and the third image data, the weight adjustment unit 222 reads the weight Wold (= weight W2) from the second memory 224, multiplies it by the weight W3 according to the above formula (5), and calculates the adjusted weight W3'. The weight storage unit 223 stores the weight W3 before adjustment as Wold in the second memory 224.

[0059] The removal unit 203 removes the noise from the image data to be denoised based on the adjusted weight W', and outputs denoised image data. Specifically, for example, the removal unit 203 removes the noise from the image data to be denoised by weighted average using the adjusted weight W'. The weighted average is expressed, for example, by the following formula (7).

[0060]

Equation

[0061] In the above formula (7), Iout is the denoised image data (denoised image data).

[0062] The image storage unit 204 stores the denoised image data Iout output from the removal unit 203 as past image data Iold in the first memory 205. The first memory 205 is constituted by the storage device 102.

[0063] <Recursive Noise Removal Process> FIG. 3 is a flowchart showing an example of a recursive noise removal procedure by the imaging device 100 according to the first embodiment. FIG. 4 is an explanatory diagram showing an example of recursive noise removal using the adjusted weight W'. As a comparative example of FIG. 4, FIG. 5 is illustrated. FIG. 5 is an explanatory diagram showing an example of recursive noise removal without applying the adjusted weight W'. In FIGS. 4 and 5, the step numbers of FIG. 3 are assigned to the portions corresponding to the steps of FIG. 3. In FIG. 5, since the adjusted weight W' is not applied, steps S304 to S306 do not exist. In FIGS. 4 and 5, the imaging device 100 has a fixed imaging direction, and the case of imaging moving image data in which the subject image moves from left to right will be described as an example.

[0064] First, the left side of the central dotted line in FIGS. 4 and 5 will be described with reference to FIG. 3. The imaging device 100 acquires and reads the current image data by the acquisition unit 201 (step S301), and reads the past image data from the first memory 205 (step S302). Here, the current image data is the subsequent image data I(t - 1) at time t - 1, and the past image data is the noise-removed image data obtained by removing noise from the preceding image data at time (t - 2) before time (t - 1), that is, Iold. It is assumed that the subject image OB(t - 2) exists in Iold.

[0065] The imaging device 100 calculates the weight W(t - 1) by giving the past image data Iold and the current image data I(t - 1) to the formula (2) by the weight calculation unit 221 (step S303). In the past image data Iold and the current image data I(t - 1), the more similar the images in the same region are, the larger the weight becomes, and the less similar they are, the smaller the weight becomes. Therefore, in the past image data Iold and the current image data I(t - 1), the regions where neither the subject images OB(t - 2) nor OB(t - 1) exist have almost the same image, and the weight w(t - 1) of the region becomes, for example, 1.0, which is the maximum value.

[0066] On one hand, in the past image data Iold and the current image data I(t - 1), the region where either one of the subject images OB(t - 2) and OB(t - 1) exists becomes a different image, and the weight w(t - 1) of this region becomes low. Here, as an example, the weight w(t - 1) of this region is set to 0.0. Note that the region at the boundary between the subject images OB(t - 2) and OB(t - 1) is more similar than the region where either one of the subject images OB(t - 2) and OB(t - 1) exists. Here, as an example, the weight w(t - 1) of this region is set to 0.5.

[0067] Next, the imaging device 100 reads the past weight Wold from the second memory 224 (step S304). Taking this past weight Wold as an example, it is a matrix with the value of all elements being "1.0". The imaging device 100 provides the past weight Wold from the second memory 224 and the weight W(t - 1) calculated in step S303 to the weight adjustment unit 222 according to Equation (3), and outputs the adjusted weight W´ (step S305).

[0068] The adjusted weight W´ is obtained, for example, by the product of the past weight Wold and the weight W(t), as shown in Equation (5). Since the past weight Wold is a matrix with the value of all elements being "1.0", the adjusted weight W´ is the same as the weight W(t - 1) calculated for the weight.

[0069] Then, the imaging device 100 stores the weight W(t - 1) before adjustment in the second memory 224 as a new past weight Wold (step S306).

[0070] The imaging device 100 uses the removal unit 203 to input the past image data Iold, the current image data I(t - 1), and the adjusted weight W' into Equation (7) to remove noise from the current image data I(t - 1) (step S307), and outputs the noise-removed image data Iout(t - 1) (step S308). The output noise-removed image data Iout(t - 1) is displayed on the display device 110. In the noise-removed image data Iout(t - 1), the subject image OB(t - 1) exists at the same position as the current image data I(t - 1), and the afterimage A(t - 2) exists at the same position as the subject image OB(t - 2) in the past image data Iold. The afterimage A(t - 2) mainly becomes image data with prominent outlines of the subject image OB(t - 2).

[0071] Also, the imaging device 100 outputs the noise-removed image data Iout(t - 1) to the first memory 205 by the image storage unit 204, and updates the past image data Iold (step S309). After that, the imaging device 100 determines whether to end the image processing, for example, based on an input of an end operation for shooting the user's movement (step S310). If not ended (step S310: No), the imaging device 100 updates the read position of the buffer memory in the acquisition unit 201 (step S311), and returns to step S301.

[0072] Next, the right side of the center dotted line in FIGS. 4 and 5 will be described with reference to FIG. 3. The imaging device 100 acquires and reads the current image data I(t) at time t by the acquisition unit 201 (step S301), and reads the past image data Iold from the first memory 205 (step S302). Here, the past image data Iold is the noise-removed image data Iout(t - 1) obtained in step S309.

[0073] The imaging device 100 calculates the weight W(t) by providing the past image data Iold and the current image data I(t) to Equation (2) using the weight calculation unit 221 (step S303). In the past image data Iold and the current image data I(t), the more similar the images in the same region are, the larger the weight becomes, and the less similar they are, the smaller the weight becomes. Therefore, in the past image data Iold and the current image data I(t), the regions where neither the subject images OB(t - 1) nor OB(t) exist result in almost the same image, and the weight w(t) of the region becomes, for example, 1.0, which is the maximum value.

[0074] On the other hand, in the past image data Iold and the current image data I(t), the regions where either one of the subject images OB(t - 1) or OB(t) exists result in different images, and the weight w(t) of the region becomes low. Here, as an example, the weight w(t) of the region is set to 0.0. Regarding the region at the boundary between the subject images OB(t - 1) and OB(t), it is more similar than the regions where either one of the subject images OB(t - 1) or OB(t) exists. Here, as an example, the weight w(t) of the region is set to 0.5. Also, the weight of the region where the afterimage A(t - 2) exists in the past image data Iold (the noise-removed image data Iout(t - 1)) is set to 0.75.

[0075] Next, the imaging device 100 reads the past weight Wold from the second memory 224 (step S304). This past weight Wold is the weight W(t - 1) calculated previously in step S306. The imaging device 100 provides the past weight Wold from the second memory 224 and the weight W(t) calculated in weight calculation in step S303 to Equation (3) using the weight adjustment unit 222, and outputs the adjusted weight W´ (step S305).

[0076] The adjusted weight W´ is obtained, for example, by the product of the past weight Wold and the weight W(t) as shown in Equation (5). In this case, since the past weight Wold is the weight W(t-1) calculated previously in step S306, the weight w(t) inside the region where the afterimage A(t-2) exists becomes 0, and the weight w(t) of the contour becomes 0.375 (=0.75×0.5). Also, the weight w(t) inside the region where the afterimage A(t-1) exists remains 0 (=0.0×0.0), but the weight w(t) of the contour is 0.25 (=0.5×0.5).

[0077] Then, the imaging device 100 stores the weight W(t) before adjustment in the second memory 224 and sets it as a new past weight Wold (step S306).

[0078] The imaging device 100 provides the past image data Iold, the current image data I(t), and the adjusted weight W´ to the removing unit 203 according to Equation (7) to remove noise from the current image data I(t) (step S307), and outputs the noise-removed image data Iout(t) (step S308). The output noise-removed image data Iout(t) has the subject image OB(t) at the same position as the current image data I(t), the afterimage A(t-1) at the same position as the subject image OB(t-1) of the past image data Iold, and further, the afterimage A(t-2) at the same position as the subject image OB(t-2).

[0079] Also, the imaging device 100 outputs the noise-removed image data Iout(t) to the first memory 205 by the image storage unit 204 and updates the past image data Iold (step S309). After that, the imaging device 100 determines whether to end the image processing, for example, based on an input of an end operation for user operation shooting (step S310). If it is to end (step S310: Yes), the imaging device 100 ends the image processing.

[0080] In this way, in the region where there is no conversion between the past image data Iold and the current image data I(t-1), I(t), the weights W(t-1), W(t) for referring to the past image data Iold become large. As a result, it is possible to increase the noise removal rate while reducing the memory used.

[0081] However, in the case of the example in FIG. 5, if there is a region where the calculation of the weights between the past image data Iold and the current image data I(t-1), I(t) is not successful, that is, a failed region, the result is also retained. Therefore, failures remain over a long period of time. This causes the image quality to deteriorate significantly as an afterimage.

[0082] For example, like the afterimage A(t-2) of the contour of the subject image OB(t-2) indicated by the thick arrow 5A1 in FIG. 5, the weight calculation result in step S303 in the next frame (image data I(t)) has become as large as the thick arrow 5a1 (w(t-1)=0.75), and as a result, the afterimage A(t-2) of the failed contour remains as shown by the thick arrow 5A2.

[0083] On the other hand, the imaging device 100 of the first embodiment holds the past image data Iold in a state where the weights calculated at that time are attached and reuses it. As a result, the weights W(t-1), W(t) for referring to the past image data Iold tend to be small in regions with low reliability where afterimages occur. Therefore, it is possible to suppress the occurrence of afterimages.

[0084] For example, while the weight w(t-1) of the contour of the subject image OB(t-2) indicated by the thick arrow 4B1 in FIG. 4 is as large as the thick arrow 4b1 (0.75), the weight w(t) has become small in the adjusted thick arrow 4b2 (0.375 = 0.5×0.75), and as a result, the remaining condition of the afterimage A(t-2) of the failed contour as shown by the thick arrow 5B2 has been reduced.

[0085] Thus, in the case of FIG. 5, since the noise removal accuracy is lower than that in FIG. 4, in order to achieve the same noise removal accuracy as in FIG. 4, it is necessary to store a larger number of past image data in the first memory 205 than in the case of FIG. 4, resulting in an increase in memory usage. In contrast, in FIG. 4, since it is only necessary to store the most recent past image data in the first memory 205, it is possible to achieve both an improvement in noise removal accuracy and memory savings.

Embodiment

[0086] Embodiment 2 is an example of suppressing the occurrence of afterimages by using the spatial information included in the image data. In Embodiment 2, the description will focus on the differences from Embodiment 1, and the same reference numerals will be used for the common parts with Embodiment 1, and the description thereof will be omitted.

[0087] <Functional configuration example of imaging device 100> FIG. 6 is a block diagram showing a functional configuration example of the imaging device 100 according to Embodiment 2. The calculation unit 602 of Embodiment 2 includes a spatial information extraction unit 621 and a weight calculation unit 622. The spatial information extraction unit 621 extracts spatial information from the image data to be extracted. The image data to be extracted is at least one of the past image data Iold or the current image data It. When using both image data Iold and It, the spatial information extraction unit 621 may extract spatial information for each image data I, calculate the average of the two pieces of spatial information, select the maximum value, or calculate an α blend.

[0088] Spatial information is information indicating the characteristics of the spatial structure included in the image data to be extracted. Specifically, for example, there are spatial frequency components. In the case of spatial frequency components, for example, the spatial information extraction unit 621 can extract the low-frequency component Slow and the high-frequency component Shigh using two types of low-pass filters of different sizes as shown in the following formulas (8) to (13).

[0089]

Equation

[0090] In equations (8) and (9), I is the image data to be extracted. Also, the function abs(·) in equation (11) is a function that calculates the absolute value for each region (x, y). Equation (13) is substituted into equations (8) and (9). When substituted into equation (8), equation (13) becomes Hσ1(x, y), and σ becomes σ1. Similarly, when substituted into equation (9), equation (13) becomes Hσ2(x, y), and σ becomes σ2.

[0091] Regarding the image data I to be extracted, in order to facilitate the extraction of spatial information, the image data I may be pre-transformed. Further, the spatial information extraction unit 621 may calculate frequency components of a plurality of bands, not just the two components of the low-frequency component Slow and the high-frequency component Shigh.

[0092] The weight calculation unit 622 calculates the weight W of the past image data based on the current image data, the past image data, and the spatial information extracted by the spatial information extraction unit 621. Regions where afterimages are likely to be prominent are often bright regions or like areas near edges (the contours of the subject image in the past image data). Therefore, it is advisable to reduce the weights of such regions.

[0093] Therefore, first, in order to determine bright regions, the weight calculation unit 622 executes processing as in the following equations (14) and (15) using the low-frequency component Slow.

[0094]

Equation

[0095] In equations (14), Slow, It, Slow, Iold respectively represent the low-frequency components Slow of the current image data It and the past image data Iold. The max(·) in equation (14) is a function that calculates the maximum value for each region (x, y). S´low in equation (15) is the maximum value of the low-frequency components Slow of the current image data It and the past image data Iold, but it may also be the average value.

[0096] Also, σ(·) is a sigmoid function that calculates a value for each region (x, y), and the larger the input value, the closer it gets to 1. Tlow is a parameter representing a threshold value. Thus, the adjustment value Klow by the low-frequency component Slow in Equation (15) becomes a smaller value as the low-frequency component Slow is larger, that is, as the region is brighter.

[0097] Next, in order to determine the vicinity of the edge, the weight calculation unit 622 executes processing as in the following equations (16) and (17) using the high-frequency component Shigh.

[0098] [Number]

[0099] Shigh, It, Shigh, and Iold in Equation (16) respectively represent the high-frequency components Shigh of the current image data It and the past image data Iold. max(·) in Equation (16) is a function that calculates the maximum value for each region (x, y). S´high in Equation (17) is the maximum value of the high-frequency components Shigh of the current image data It and the past image data Iold, but it may also be an average value.

[0100] Also, σ(·) is a sigmoid function that calculates a value for each region (x, y), and the larger the input value, the closer it gets to 1. Thigh is a parameter representing a threshold value. Thus, the adjustment value Khigh by the high-frequency component Shigh in Equation (17) becomes a smaller value as the high-frequency component Shigh is larger, that is, as it is closer to the edge.

[0101] Using the obtained adjustment value Klow by the low-frequency component Slow and the adjustment value Khigh by the high-frequency component Shigh, the finally updated weight Wnew is obtained by the following equation (18). The removal unit 203 removes noise from the current image data It by the same processing as in the first embodiment using the updated weight Wnew, and outputs it as noise-removed image data. Also, the image storage unit 204 stores the noise-removed image data as the past image data Iold in the first memory 205.

[0102]

Number

[0103] In Equation (18), W on the right side is a weight for referring to past image data Iold. By updating the weight W to the weight Wnew in Equation (18), the weights in bright regions and near edges, which are regions where afterimages are likely to be prominent, become smaller, so that the occurrence of afterimages can be suppressed. Here, the low-frequency component Slow and the high-frequency component Shigh of the current image data It and the past image data Iold are used, but only one of the low-frequency component Slow and the high-frequency component Shigh may be used.

[0104] <Recursive noise removal process> FIG. 7 is a flowchart showing an example of a recursive noise removal procedure by the imaging device 100 according to the second embodiment. In the second embodiment, compared with FIG. 3, steps S303 to S306 do not exist, and after step S302, the imaging device 100 executes a spatial information extraction process by the spatial information extraction unit 621 (step S701). Next, the imaging device 100 calculates an adjusted weight Wnew by the weight calculation unit 622 (step S702). After that, the imaging device 100 removes noise from the current image data It using the adjusted weight Wnew by the removal unit 203 (step S307), and outputs the noise-removed image data Iout to the display device 110 (step S308).

[0105] The imaging device 100 stores the noise-removed image data Iout as past image data Iold in the first memory 205 by the image storage unit 204 (step S309). By recursively executing such processing until imaging is completed, the weight in the region where the afterimage is likely to be prominent is adjusted to be low, and as a result, the occurrence of the afterimage can be suppressed.

Embodiment

[0106] Example 3 is an example in which the spatial information extraction unit 621 of Example 2 is applied in Example 1. In Example 3, the description will focus on the differences from Examples 1 and 2, and the same reference numerals will be used for the common parts with Examples 1 and 2, and the description thereof will be omitted.

[0107] <Functional configuration example of imaging device 100> FIG. 8 is a block diagram showing a functional configuration example of the imaging device 100 according to Example 3. In the case of Example 3, the calculation unit 802 includes a spatial information extraction unit 621, a weight calculation unit 622, a weight adjustment unit 222, a weight storage unit 223, and a second memory 224. The weight adjustment unit 222 adjusts the weight Wnew updated by the weight calculation unit 622 based on the latest weight Wold stored in the second memory 224.

[0108] Let the adjusted updated weight Wnew be the adjusted weight W'. The weight storage unit 223 stores the weight Wnew before adjustment by the weight adjustment unit 222 as the past weight Wold in the second memory 224. Specifically, for example, the weight adjustment unit 222 adjusts the weight Wnew by replacing W in formulas (3) to (7) used by the weight adjustment unit 222 with Wnew.

[0109] <Recursive noise removal process> FIG. 9 is a flowchart showing an example of a recursive noise removal procedure by the imaging device 100 according to Example 3. In Example 3, when compared with FIG. 3, step S303 does not exist. After step S302, the imaging device 100 executes a spatial information extraction process by the spatial information extraction unit 621 (step S701). Next, the imaging device 100 calculates the weight Wnew by the weight calculation unit 622 (step S702).

[0110] The imaging device 100 reads the past weight Wold from the second memory 224 (step S304), adjusts the weight Wnew with the past weight Wold by the weight adjustment unit 222, outputs the adjusted weight W' to the removal unit 203, and outputs the weight Wnew before adjustment to the weight storage unit 223 (step S305). Then, the weight storage unit 223 stores the weight Wnew before adjustment in the second memory 224 and updates the past weight Wold (step S306).

[0111] After that, the imaging device 100 removes noise from the current image data It using the adjusted weight W' by the removal unit 203 (step S307), and outputs the noise-removed image data Iout to the display device 110 (step S308). The imaging device 100 stores the noise-removed image data Iout as the past image data Iold in the first memory 205 by the image storage unit 204 (step S309). By recursively executing such processing until imaging is completed, the weight in the area where the afterimage is likely to be prominent is adjusted to be low, and as a result, the occurrence of afterimages can be suppressed.

[0112] In this way, by extracting spatial information, calculating weights using the spatial information, and then adjusting the weights, it is possible to suppress afterimages before and after weight adjustment, and improve the efficiency of suppressing afterimage generation.

Example

[0113] Example 4 is an example in which, prior to noise removal from the current image data using weights in the removal unit 203, a residual image reduction process is performed on the current image data. The removal unit 203 performs noise removal using weights from the current image data after the residual image reduction process. The imaging device 100 of Example 4 aims to suppress the occurrence of afterimages by reducing the afterimage of the current image data before noise removal. In Example 4, the description will focus on the differences from Examples 1 to 3, and the description of the common parts with Examples 1 to 3 will be omitted using the same reference numerals.

[0114] <Functional configuration example> FIG. 10 is a block diagram showing a functional configuration example of the imaging apparatus 100 according to Example 4. In the case of Example 4, the removal unit 203 acquires weights from the calculation unit 202. The weights from the calculation unit 202 are the adjusted weights W' in the cases of Examples 1 and 3, and the updated weights Wnew in the case of Example 2. Further, when the calculation unit 202 has the configuration shown in FIG. 5, the weights from the calculation unit 202 are the weights W calculated by the weight calculation unit 221 by applying the past image data Iold and the current image data I to the above formula (2). In Example 4, unless otherwise distinguished, these weights are collectively referred to simply as weight W.

[0115] The removal unit 203 includes a ghost reduction processing unit 1001 and a noise removal processing unit 1002. The ghost reduction processing unit 1001 executes a process of reducing the ghost of the current image data. Specifically, for example, the ghost reduction processing unit 1001 blurs the contour of a moving subject image that becomes a ghost by applying a low-pass filter to the current image data. Thereby, the contour of the moving subject image is made less prominent. Further, the ghost reduction processing unit 1001 blurs the contour of a moving subject image that becomes a ghost by interpolating between two pieces of image data I(t - 1) and I(t) that are continuous in the time direction (interpolation processing).

[0116] The noise removal processing unit 1002 removes the noise of the image data to be subjected to noise removal based on the weight W from the calculation unit 202 for the image data subjected to the ghost reduction processing by the ghost reduction processing unit 1001, and outputs the noise-removed image data. Specifically, for example, the removal unit 203 substitutes the weight W from the calculation unit 202 for W' in the above formula (7) to remove the noise of the image data to be subjected to noise removal by weighted average. The image data to be subjected to noise removal is the image data subjected to the low-pass filter or the image data generated by the interpolation processing.

[0117] <Interpolation processing example> FIG. 11 is an explanatory diagram showing an interpolation process example by the afterimage reduction processing unit 1001 according to Example 4. In FIG. 11, the rectangle is a group of image data continuous in the time direction, that is, moving image data. (A) shows the unprocessed image data Ia(t - 3), Ia(t - 2), Ia(t - 1), Ia(t) that has not been subjected to afterimage reduction processing and is input from the acquisition unit 201 to the afterimage reduction processing unit 1001. In the image data Ia(t - 3), Ia(t - 2), Ia(t - 1), Ia(t), it is assumed that the subject images OBa(t - 3), OBa(t - 2), OBa(t - 1), OBa(t) are moving from left to right.

[0118] (B) shows the synthesis of image data. The afterimage reduction processing unit 1001 synthesizes two pieces of image data continuous in the time direction to detect a motion vector. Specifically, for example, the afterimage reduction processing unit 1001 synthesizes the image data Ia(t - 3), Ia(t - 2) to generate synthesized image data Ib(t - 2), and detects the motion vector mv(t - 2) between the subject images OBa(t - 3), OBa(t - 2).

[0119] Similarly, the afterimage reduction processing unit 1001 synthesizes the image data Ia(t - 2), Ia(t - 1) to generate synthesized image data Ib(t - 1), and detects the motion vector mv(t - 1) between the subject images OBa(t - 2), OBa(t - 1). The afterimage reduction processing unit 1001 synthesizes the image data Ia(t - 1), Ia(t) to generate synthesized image data Ib(t), and detects the motion vector mv(t) between the subject images OBa(t - 1), OBa(t).

[0120] (C) shows the interpolation of the synthesized image data Ib(t - 2), Ib(t - 1), Ib(t). Specifically, for example, in the synthesized image data Ib(t - 2), the afterimage reduction processing unit 1001 replicates the contour (left - hand half - arc) on the terminal side of the motion vector mv(t - 2) among the contours of the older subject image OBa(t - 3) in the moving direction to interpolate between the subject images OBa(t - 3), OBa(t - 2). Thereby, the afterimage reduction processing unit 1001 generates interpolated synthesized image data Ic(t - 2) including the interpolated subject image OBc(t - 2).

[0121] Similarly, the afterimage reduction processing unit 1001 interpolates between the subject images OBa(t - 2) and OBa(t - 1) by replicating, in the moving direction, the contour (left - hand half arc) on the terminal side of the motion vector mv(t - 1) among the contours of the older subject image OBa(t - 2) in the composite image data Ib(t - 1). Thereby, the afterimage reduction processing unit 1001 generates interpolated composite image data Ic(t - 1) including the interpolated subject image OBc(t - 1).

[0122] Also, the afterimage reduction processing unit 1001 interpolates between the subject images OBa(t - 1) and OBa(t) by replicating, in the moving direction, the contour (left - hand half arc) on the terminal side of the motion vector mv(t) among the contours of the older subject image OBa(t - 1) in the composite image data Ib(t). Thereby, the afterimage reduction processing unit 1001 generates interpolated composite image data Ic(t) including the interpolated subject image OBc(t). In FIG. 11, the contour to be replicated is the contour of the older subject image, but it may be the contour of the newer subject image instead.

[0123] In this way, the imaging device 100 converts the afterimages that discretely appear in each image data into a moving trajectory of a subject image as if the shutter speed were slow by the interpolation process, and displays them continuously. Thereby, the afterimage can be reduced by showing it to the user as if there were no afterimage.

[0124] <Recursive noise removal processing> FIG. 12 is a flowchart showing an example of a recursive noise removal procedure by the imaging device 100 according to Embodiment 4. In Embodiment 4, after step S302, the imaging device 100 calculates weights according to any one of Embodiments 1 to 3 and FIG. 5 (step S1203). Next, the imaging device 100 executes afterimage reduction processing by a low-pass filter or interpolation processing by the afterimage reduction processing unit 1001 (step S1204). Then, the imaging device 100 removes noise from the image data after the afterimage reduction processing using the weight W calculated in step S1203 by the noise removal processing unit 1002 (step S307).

[0125] After that, the imaging device 100 outputs the noise-removed image data Iout to the display device 110 (step S308), and the image storage unit 204 stores the noise-removed image data Iout as past image data Iold in the first memory 205 (step S309). By recursively executing such processing until imaging is completed, the afterimage can be continuously made less noticeable, and as a result, the occurrence of afterimages can be suppressed.

Embodiment

[0126] In Embodiment 4, an example in which afterimage reduction processing is executed on the current image data prior to noise removal from the current image data using weights in the removal unit 203 has been described. Conversely, in Embodiment 5, afterimage reduction processing is executed on the noise-removed image data after noise removal from the current image data using weights in the removal unit 203. The imaging device 100 of Embodiment 5 aims to suppress the occurrence of afterimages by reducing the afterimage of the current image data after noise removal. In Embodiment 5, the description will focus on the differences from Embodiment 4, and the description of the common parts with Embodiments 1 to 4 will be omitted using the same reference numerals.

[0127] <Functional configuration example> FIG. 13 is a block diagram showing a functional configuration example of the imaging device 100 according to Embodiment 5. The removal unit 203 acquires weights from the calculation unit 202. The weights from the calculation unit 202 are the adjusted weights W' in the cases of Embodiments 1 and 3, and the updated weights Wnew in the case of Embodiment 2. Further, when the calculation unit 202 has the configuration shown in FIG. 5, the weights from the calculation unit 202 are the weights W calculated by the weight calculation unit 221 by applying the past image data Iold and the current image data I to the above formula (2). In Embodiment 5, unless otherwise distinguished, these weights are collectively referred to simply as the weight W.

[0128] The removal unit 203 includes a noise removal processing unit 1002 and an afterimage reduction processing unit 1001. The noise removal processing unit 1002 removes the noise of the image data to be subjected to noise removal based on the weight W from the calculation unit 202 for the image data from the acquisition unit 201, and outputs noise-removed image data. Specifically, for example, the removal unit 203 substitutes the weight W from the calculation unit 202 for W' in the above formula (7) to remove the noise of the image data by weighted average.

[0129] The afterimage reduction processing unit 1001 executes a process of reducing the afterimage of the noise-removed image data. Specifically, for example, the afterimage reduction processing unit 1001 blurs the contour of the moving subject image that becomes an afterimage by applying a low-pass filter to the noise-removed image data. Thereby, the contour of the moving subject image is made less conspicuous. Further, as shown in FIG. 11, the afterimage reduction processing unit 1001 blurs the contour of the moving subject image that becomes an afterimage by interpolating between two pieces of noise-removed image data I(t - 1) and I(t) that are continuous in the time direction (interpolation process).

[0130] <Recursive Noise Removal Processing> FIG. 14 is a flowchart showing an example of a recursive noise removal procedure by the imaging device 100 according to Embodiment 5. In Embodiment 5, after step S302, the imaging device 100 calculates weights according to any one of Embodiments 1 to 3 and FIG. 5 (step S1203). Next, the imaging device 100 uses the weight W calculated in step S1203 by the noise removal processing unit 1002 to remove noise from the noise removal image data (step S307). Then, the imaging device 100 executes afterimage reduction processing by a low-pass filter or interpolation processing by the afterimage reduction processing unit 1001 (step S1204).

[0131] After that, the imaging device 100 outputs the noise removal image data Iout to the display device 110 (step S308), and the image storage unit 204 stores the noise removal image data Iout as past image data Iold in the first memory 205 (step S309). By recursively executing such processing until imaging is completed, the afterimage can be continuously made less noticeable, and as a result, the occurrence of afterimages can be suppressed. Also, by performing noise removal prior to the afterimage reduction processing, blurring and interpolation of noise can be suppressed.

Embodiment

[0132] Embodiment 6 is an example in which a region where an afterimage occurs (afterimage occurrence region) is detected from the image data, and the afterimage is removed from the detected afterimage occurrence region (detection region). Thereby, the range for reducing the afterimage can be narrowed down to the afterimage occurrence region, and the afterimage reduction processing can be efficiently executed. In Embodiment 6, the description will be centered on the differences from Embodiment 5, and the description of the common parts with Embodiments 1 to 5 will be omitted using the same reference numerals.

[0133] FIG. 15 is an explanatory diagram showing an example of afterimage reduction processing according to Example 6. In FIG. 15, the rectangle indicates image data. The shaded area within the rectangle indicates noise N. Although the noise may be uniformly present throughout the image data, here, for simplicity of explanation, it is assumed that the noise N is present at a specific position. (A) shows an example where the subject image does not move, and (B) shows an example where the subject image moves.

[0134] (A), the image data Ia(t - 1), Ia(t) are a series of image data continuous in the time direction. The image data Ia(t - 1), Ia(t) both have the noise N and the subject images OB(t - 1), OB(t) at the same position. The noise-removed image data NRIa(t - 1) is the image data obtained by removing the noise N from the image data Ia(t - 1) based on the noise-removed image data at time t - 2 (past image data Iold). The noise-removed image data NRIa(t) is the image data obtained by removing the noise N from the image data Ia(t) based on the noise-removed image data NRIa(t - 1) at time t - 1 (past image data Iold).

[0135] The imaging device 100 generates the noise-removed image data NRIa(t) in which the noise N is removed from the image data Ia(t) by weighted averaging the image data Ia(t) at time t, which is the current image data, and the noise-removed image data NRIa(t - 1) at time t - 1 (past image data Iold). Also, the difference image data DIa(t) is the image data obtained by taking the difference between the image data Ia(t) and the noise-removed image data NRIa(t). In this case, the subject image OB(t) and its background are canceled out, and the noise N remains as the difference.

[0136] (B), the image data Ib(t - 1), Ib(t) are a series of image data continuous in the time direction. The image data Ib(t - 1), Ib(t) have the noise N at the same position, but the positions of the subject images OB(t - 1), OB(t) are different. That is, it shows that the subject is moving from left to right.

[0137] The noise-removed image data NRIb(t - 1) is image data in which noise N has been removed from the image data Ib(t - 1) based on the noise-removed image data at time t - 2 (past image data Iold). Assume that the noise-removed image data NRIb(t - 1) has a residual image A(t - 2) of the subject image that existed in the image data at time t - 2.

[0138] The noise-removed image data NRIb(t) is image data in which noise N has been removed from the image data Ib(t) based on the noise-removed image data NRIb1(t - 1) at time t - 1 (past image data Iold). However, in the noise-removed image data NRIb1(t), a residual image A(t - 1) exists at the position of the subject image OB(t - 1) in the noise-removed image data NRIb(t - 1).

[0139] The imaging device 100 generates noise-removed image data NRIb1(t) in which noise N has been removed from the image data Ib(t) by weighted averaging the image data Ib(t) at time t, which is the current image data, and the noise-removed image data NRIb(t - 1) at time t - 1 (past image data Iold). Also, the difference image data DIb(t) is image data obtained by taking the difference between the image data Ib(t) and the noise-removed image data NRIb1(t). In this case, the subject image OB(t) and its background are canceled out, and noise N and the residual image A(t - 1) remain as the difference.

[0140] The imaging device 100 refers to the difference image data DIb(t) and specifies the region of the residual image A(t - 1) as the residual image generation region 1500. Next, the imaging device 100 extracts image data 1501 at the same position as the image data of the residual image generation region (in this case, the contour residual image A(t - 1)) from the image data Ib(t) at time t. Then, the imaging device 100 replaces the residual image A(t - 1) in the noise-removed image data NRIb1(t) with the extracted image data 1501. Thereby, the imaging device 100 can generate noise-removed image data NRIb2(t) from which the residual image A(t - 1) has been removed from the noise-removed image data NRIb1(t).

[0141] Note that the imaging device 100 extracts the image data 1501 at the same position as the image data of the afterimage generation region 1500 (in this case, the afterimage A(t - 1) of the contour) from the image data Ib(t) at time t. However, the extraction range is not limited to the same position as the afterimage generation region 1500, and may be a region including the afterimage generation region 1500 and its vicinity. Also, here, since the afterimage is often the contour of the subject image, the afterimage generation region 1500 is set as the region corresponding to the contour of the subject image. However, the inside surrounded by the contour may also be set as the afterimage generation region 1500. Thereby, it is possible to reduce the omission of afterimage removal.

[0142] <Functional configuration example> FIG. 16 is a block diagram showing a functional configuration example of the imaging device 100 according to Example 6. As shown in FIG. 15, the afterimage reduction unit acquires noise removal image data from the noise removal processing unit 1002. Also, as shown in FIG. 15, the afterimage reduction processing unit 1001 acquires the current image data Ib(t) from, for example, the acquisition unit 201. Further, the afterimage reduction processing unit 1001 acquires the past image data Iold from the first memory 205. The past image data Iold is, for example, the noise removal image data NRIb(t - 1) at the previous time t - 1.

[0143] As shown in FIG. 15, the afterimage reduction processing unit 1001 generates the noise removal image data NRIb(t) for the current image data Ib(t) by weighted averaging the current image data Ib(t) and the past image data Iold, and generates the difference image data DIb(t) between the current image data Ib(t) and the generated noise removal image data NRIb(t).

[0144] Then, the afterimage reduction processing unit 1001 detects the afterimage generation region 1500 from the difference image data DIb(t), and specifies the image data 1501 at the same position as the afterimage generation region 1500 from the current image data Ib(t). Then, the afterimage reduction processing unit 1001 generates the noise removal image data NRIb2(t) in which the afterimage A(t - 1) is removed by synthesizing the image data 1501 in the noise removal image data NRIb1(t).

[0145] The afterimage reduction processing unit 1001 outputs the generated noise-removed image data NRIb2(t) to the display device 110 and also outputs it to the image storage unit 204. The image storage unit 204 stores the acquired noise-removed image data NRIb2(t) in the first memory 205 as new past image data Iold.

[0146] Note that the afterimage reduction processing unit 1001 may apply the low-pass filter and interpolation processing shown in Example 5 to Example 6. Specifically, for example, the afterimage reduction processing unit 1001 may apply a low-pass filter or interpolation processing to the current image data Ib(t), and generate difference image data DIb(t) between the current image data Ib(t) after the application and the generated noise-removed image data NRIb(t).

[0147] Also, the afterimage reduction processing unit 1001 may apply a low-pass filter or interpolation processing to the generated noise-removed image data NRIb2(t), and output the noise-removed image data NRIb2(t) after the application to the display device 110 and the image storage unit 204.

[0148] <Recursive Noise Removal Processing> FIG. 17 is a flowchart showing an example of a recursive noise removal procedure by the imaging device 100 according to Example 6. In Example 6, after step S302, the imaging device 100 calculates a weight according to any one of Examples 1 to 3 and FIG. 5 (step S1203). Next, the imaging device 100 removes noise from the noise-removed image data using the weight W calculated in step S1203 by the noise removal processing unit 1002 (step S307). Then, the imaging device 100 executes afterimage reduction processing by a low-pass filter or interpolation processing by the afterimage reduction processing unit 1001 (steps S1771 to S1773).

[0149] Specifically, for example, the afterimage reduction processing unit 1001 generates difference image data DIb(t) between the current image data Ib(t) and the noise-removed image data NRIb1(t) (step S1771), and detects the afterimage generation region 1500 from the difference image data DIb(t) (step S1772).

[0150] Then, the afterimage reduction processing unit 1001 extracts the image data 1501 at the same position as the afterimage generation region 1500 from the current image data Ib(t), and synthesizes it with the noise-removed image data NRIb1(t) to generate the noise-removed image data NRIb2(t) and remove the afterimage A(t - 1) (step S1773). After that, the afterimage reduction processing unit 1001 outputs the noise-removed image data NRIb2(t) to the display device 110 as the output image data Iout (step S308), and outputs it to the image storage unit 204 (step S309).

[0151] By recursively executing such processing until the imaging is completed, the afterimage can be continuously made less noticeable, and as a result, the generation of the afterimage can be suppressed. In this way, the range for reducing the afterimage can be narrowed down to the afterimage generation region, and the afterimage reduction processing can be efficiently executed. In other words, since it is not necessary to execute the afterimage reduction processing for the region where no afterimage has occurred, it is possible to avoid the afterimage reduction processing for the region where the afterimage does not need to be reduced, and the original image can be maintained.

Example

[0152] Example 7 is an example in which the condition for imaging a moving image in which an afterimage is unlikely to occur is adjusted to make the afterimage less noticeable.

[0153] <Moving image imaging example> FIG. 18 is an explanatory diagram showing an example of video imaging according to Example 7. FIG. 18 shows an example of imaging a certain subject as a video, and the rectangle represents a series of image data that is continuous in the time direction, that is, video image data. For the sake of convenience in explanation, the frame rate is set to 3 [fps] as an example. Among the times t1 to t3, time t1 is the oldest time and time t3 is the latest time. (A) is an example of video imaging where the imaging conditions are not adjusted in Example 7, and (B) is an example of video imaging where the imaging conditions are adjusted in Example 7. Therefore, in both (A) and (B), one image data is generated at 1 / 3 [fps].

[0154] Note that at the end of the reference signs, (t1) to (t3) are attached to distinguish the times t1 to t3. When the times t1 to t3 are not distinguished, it is simply denoted as (t). Even when the times t1 to t3 are not distinguished, it is simply denoted as time t.

[0155] In (A), et(t) is the exposure time. gp(t) is the gap time between consecutive exposure times. The exposure time is the time during which the imaging element 105 is exposed to the subject light from the lens system from the open state to the closed state of the shutter (for example, a rolling shutter).

[0156] The imaging device 100 images the subject at times t1 to t3 and generates image data Ia(t1), Ia(t2), Ia(t3). The image data Ia(t1), Ia(t2), Ia(t3) respectively include subject images OBa(t1), OBa(t2), OBa(t3).

[0157] When imaging the subject, first, the imaging device 100 exposes for the exposure time et(t) starting from time t within 1 / 3 [s] starting from time t. Next, the imaging device 100 reads out the analog electrical signal from the imaging element 105 during the gap time gp(t) after the elapse of the exposure time et(t), and the AFE 106 executes signal processing. Thereby, the image data Ia(t) is generated.

[0158] The imaging device 100 recursively performs noise removal for each time t. For example, at time t3, the imaging device 100 performs noise removal on image data Ia(t3) obtained at time t3. Specifically, for example, the imaging device 100 generates noise-removed image data NRIa(t3) in which noise has been removed from the image data Ia(t3) by taking a weighted average of the noise-removed image data in which noise has been removed from the image data Ia(t2) and the image data Ia(t3).

[0159] In the noise-removed image data NRIa(t3), the afterimages Aa(t2) and Aa(t1) are image data inherited from the subject images OBa(t2) and OBa(t1) in the image data Ia(t2) and Ia(t1) through recursive noise removal, and the older the subject images OBa(t2) and OBa(t1), the fainter they are.

[0160] In addition, in the noise-removed image data NRIa(t3), the subject image OBa(t3) and the afterimages Aa(t2) and Aa(t1) appear discretely in the direction of the subject's movement, making the subject's movement appear unnatural. In other words, if the exposure time et(t) is short, the gap time gp(t), which is the non-exposure time, becomes relatively long, causing a shift between the previous and next images and making the discontinuity noticeable.

[0161] In (B), ET(t) is the exposure time. GP(t) is the gap time between successive exposure times. The exposure time ET(t) is longer than the exposure time et(t) due to the imaging condition setting according to this embodiment 7, and therefore the gap time GP(t) is shorter than the gap time gp(t).

[0162] Imaging device 100 captures an image of a subject from time t1 to time t3 to generate image data Ib(t1), Ib(t2), and Ib(t3). Image data Ib(t1), Ib(t2), and Ib(t3) include subject images OBb(t1), OBb(t2), and OBb(t3), respectively.

[0163] When imaging a subject, first, the imaging device 100 exposes for an exposure time ET(t) starting from time t within 1 / 3 [s] starting from time t. Next, after the exposure time ET(t) has elapsed, the imaging device 100 reads an analog electrical signal from the imaging element 105 during a gap time GP(t), and the AFE 106 executes signal processing. Thereby, image data Ib(t) is generated.

[0164] Since the exposure time ET(t) is longer than the exposure time et(t) in (A), it is possible to image the state in which the subject is moving. In this case, the image data Ib(t) includes a subject image OBb(t) and a trajectory TJ(t) following the subject image OBb(t) as the subject moves from left to right. The trajectory TJ(t) is a time-series afterimage of the subject image OBb(t) obtained as the exposure time ET(t) elapses.

[0165] The imaging device 100 recursively executes noise removal at each time t. For example, at time t3, the imaging device 100 executes noise removal on the image data Ib(t3) obtained at time t3. Specifically, for example, the imaging device 100 generates noise-removed image data NRIb(t3) with noise removed from the image data Ib(t3) by taking a weighted average of the noise-removed image data with noise removed from the image data Ib(t2) and the image data Ib(t3).

[0166] In the noise-removed image data NRIb(t3), the afterimages Ab(t2), Ab(t1) are image data inherited from the trajectories TJ(t2), TJ(t1) of the subject images OBa(t2), OBa(t1) in the image data Ib(t2), Ib(t1) by recursive noise removal, and the older subject images OBb(t2), OBb(t1) become thinner.

[0167] In the noise-removed image data NRIb(t3), since the subject image OBb(t3), its trajectory TJ(t3), and the afterimages Ab(t2), Ab(t1) appear continuously in the moving direction of the subject, the movement of the subject becomes a natural video. That is, when the exposure time ET(t) is shorter than the exposure time et(t), the gap time GP(t), which is the time when no exposure is performed, becomes relatively short, and the displacement between the front and rear images is reduced and the discontinuity becomes less noticeable.

[0168] <Functional configuration example of imaging device 100> FIG. 19 is a block diagram showing a functional configuration example of the imaging device 100 according to Example 7. The imaging device 100 includes a conversion table 1900, an input unit 1901, a setting unit 1902, an adjustment unit 1903, an imaging unit 120, and an image processing unit 1905. Specifically, the conversion table 1900 is realized by, for example, the storage device 102 shown in FIG. 1. Further, the input unit 1901, the setting unit 1902, the adjustment unit 1903, and the image processing unit 1905 are specifically realized by, for example, causing the processor 101 to execute a program stored in the storage device 102 shown in FIG. 1, or by the LSI 107.

[0169] The conversion table 1900 is a table that converts the first imaging condition to the second imaging condition. The first imaging condition 1952 is the default imaging condition, for example, the exposure time corresponding to the frame rate 1951. In the imaging device 100, for example, when the frame rate 1951 is set to "FRi", the exposure time is set to "eti". In the example of FIG. 18, in (A), when the frame rate 1951 is 3 [fps], the exposure time is et(t).

[0170] The second imaging condition 1953 is an imaging condition for reducing afterimages and corresponds to the first imaging condition 1952. For example, for FCi which is the second imaging condition 1953, the exposure time ETi is set to be longer than the default exposure time eti when the frame rate 1951 is FRi. Also, in order to suppress overexposure due to the extension of the exposure time from eti to ETi, FCi which is the second imaging condition 1953 may include the amount of decrease in ISO sensitivity (it may also be the ISO sensitivity after the decrease).

[0171] The input unit 1901 receives the input of the frame rate 1951 from the operation device 108 when imaging a subject. Note that the input unit 1901 may receive the frame rate 1951 defined in the shooting mode when the shooting mode is selected from the operation device 108. Note that depending on the shooting mode, the exposure time may be different even for the same frame rate 1951.

[0172] The setting unit 1902 sets the second imaging condition 1953 for reducing afterimages based on the frame rate 1951 input by the input unit 1901 and the exposure time corresponding to the frame rate 1951. Specifically, for example, the setting unit 1902 reads the second imaging condition 1953 corresponding to the frame rate 1951 from the input unit 1901 from the conversion table 1900. When only the frame rate 1951 is given from the input unit 1901, the exposure time of the first imaging condition 1952 is used as the default exposure time, and the setting unit 1902 reads the corresponding second imaging condition 1953 from the conversion table 1900.

[0173] Also, when the frame rate 1951 and the exposure time are specified by the shooting mode from the input unit 1901, the setting unit 1902 reads the second imaging condition 1953 corresponding to the frame rate 1951 and the exposure time from the conversion table 1900. Also, when the second imaging condition 1953 includes the amount of decrease in ISO sensitivity (or the ISO sensitivity after the decrease), the setting unit 1902 also reads the amount of decrease in ISO sensitivity (or the ISO sensitivity after the decrease).

[0174] The adjustment unit 1903 adjusts the exposure time corresponding to the frame rate 1951 input by the input unit 1901 to an exposure time longer than the exposure time. Specifically, for example, as shown in FIG. 18, the adjustment unit 1903 changes the exposure time et(t) to the exposure time ET(t) obtained from the conversion table 1900 when the frame rate 1951 is 3 [fps] and the exposure time is et(t).

[0175] The imaging unit 120 images the subject based on the frame rate 1951 and the exposure time adjusted by the adjustment unit 1903, and outputs an image data sequence continuous in the time direction. Specifically, for example, the imaging unit 120 generates image data Ib(t) as shown in FIG. 18(B).

[0176] The image processing unit 1905 performs image processing on the image data Ib(t) from the imaging unit 120. Specifically, for example, the image processing unit 1905 includes an acquisition unit 201, a calculation unit 202, a removal unit 203, an image storage unit 204, and a first memory 205 shown in Embodiments 1 to 6. Therefore, by the recursive noise processing shown in Embodiment 1, noise removal image data NRIb(t3) as shown in FIG. 18(B) is generated.

[0177] <Example of imaging processing procedure> FIG. 20 is a flowchart showing an example of an imaging processing procedure by the imaging device 100. The imaging device 100 receives the input of the frame rate 1951 by the input unit 1901 (step S2001), and sets the second imaging condition 1953 by the setting unit 1902 (step S2002).

[0178] The imaging device 100 adjusts the exposure time under the second imaging condition 1953 by the adjustment unit 1903 (step S2003), and the imaging unit 120 images a subject to generate an image data sequence (step S2004). The imaging device 100 executes image processing including the recursive noise removal processing (Figs. 3, 7, 9, 12, 14, 17) shown in Examples 1 to 6 by the image processing unit 1905 (step S2005). Note that the image processing (step S2005) may include the recursive noise removal processing as shown in Fig. 5.

[0179] In the above example, the exposure time, which is the light amount information, is the adjustment target. However, the light amount information to be adjusted is not limited to the exposure time, and may be the F-number or the ISO sensitivity. When the F-number is the adjustment target, the exposure time of the first imaging condition 1952 becomes the F-number, and the exposure time of the second imaging condition 1953 becomes an F-number with a value smaller than the F-number of the first imaging condition 1952. Also in this case, in order to reduce overexposure, the second imaging condition 1953 may include a value that makes it shorter than the default exposure time or makes the ISO sensitivity lower.

[0180] When the ISO sensitivity is the adjustment target, the exposure time of the first imaging condition 1952 becomes the ISO sensitivity, and the exposure time of the second imaging condition 1953 becomes an ISO sensitivity higher than the ISO sensitivity of the first imaging condition 1952. Also in this case, in order to reduce overexposure, the second imaging condition 1953 may include a value that makes it shorter than the default exposure time or makes the F-number larger.

[0181] As described above, according to Example 7, an image data sequence in which the afterimage is less noticeable can be generated. Further, by applying these image data sequences to the image processing unit 1905, noise removal results as in Examples 1 to 6 can be obtained recursively.

Explanation of Signs

[0182] 100 Imaging device, 101 Processor, 102 Memory device, 105 Image sensor, 120 Imaging unit, 200 Image processing device, 201 Acquisition unit, 202 Calculation unit, 203 Removal unit, 204 Image storage unit, 205 First memory, 221 Weight calculation unit, 222 Weight adjustment unit, 223 Weight storage unit, 224 Second memory, 602 Calculation unit, 621 Spatial information extraction unit, 622 Weight calculation unit, 802 Calculation unit, 1001 Afterimage reduction processing unit, 1002 Noise removal processing unit, 1900 Conversion table, 1901 Input unit, 1902 Setting unit, 1903 Adjustment unit, 1905 Image processing unit

Claims

1. An imaging unit that images a subject and outputs first image data and second image data that is temporally later than the first image data; A calculation unit that calculates a first weight of the first image data based on the first noise-removed image data obtained by removing noise from the first image data and the second image data; An afterimage reduction processing unit that reduces an afterimage in the second image data; A removal unit that removes noise from the second image data that has been subjected to the afterimage reduction processing by the afterimage reduction processing unit based on the first noise-removed image data and the first weight calculated by the calculation unit; The afterimage reduction processing unit synthesizes the first image data and the second image data, and interpolates between the subject image in the first image data and the subject image in the second image data in the synthesized image data; An imaging device.

2. An imaging unit that images a subject and outputs first image data and second image data that is temporally later than the first image data; A calculation unit that calculates a first weight of the first image data based on the first noise-removed image data obtained by removing noise from the first image data and output to a display device and the second image data; An afterimage reduction processing unit that reduces an afterimage in the second image data; A removal unit that removes noise from the second image data that has been subjected to the afterimage reduction processing by the afterimage reduction processing unit based on the first noise-removed image data and the first weight calculated by the calculation unit; An imaging device having the above.

3. The imaging device according to claim 1 or 2, The afterimage reduction processing unit uses a low-pass filter to reduce an afterimage in the second image data. An imaging device.

4. The imaging device according to claim 3, The imaging unit outputs third image data that is temporally later than the second image data, The calculation unit calculates a second weight of the second image data based on the second noise-removed image data obtained by removing noise from the second image data by the removal unit and the third image data, and calculates a third weight based on the first weight and the second weight. The afterimage reduction processing unit reduces the afterimage in the third image data. The removal unit removes noise from the third image data that has been subjected to afterimage reduction processing by the afterimage reduction processing unit based on the second noise-removed image data and the third weight calculated by the calculation unit. Imaging device.

5. The imaging device according to claim 1 or 2, An extraction unit that extracts first spatial information indicating a feature of a spatial structure included in an image from at least one of the first noise-removed image data and the second image data. The calculation unit calculates the first weight based on the first spatial information extracted by the extraction unit. Imaging device.

6. An imaging unit that captures a subject and outputs first image data and second image data that is temporally later than the first image data. A calculation unit that calculates a first weight of the first image data based on the first noise-removed image data obtained by removing noise from the first image data and the second image data. A removal unit that removes noise from the second image data based on the first noise-removed image data and the first weight calculated by the calculation unit. An afterimage reduction processing unit that reduces an afterimage in the second noise-removed image data obtained by removing noise from the second image data by the removal unit. The afterimage reduction processing unit synthesizes the first noise-removed image data and the second noise-removed image data, and interpolates between the subject image in the first noise-removed image data and the subject image in the second noise-removed image data in the synthesized image data. Imaging device.

7. The imaging device according to claim 6, An imaging device in which the afterimage reduction processing unit reduces an afterimage in the second noise-removed image data using a low-pass filter.

8. An imaging unit that images a subject and outputs first image data and second image data that is temporally later than the first image data, A calculation unit that calculates a first weight of the first image data based on the first noise-removed image data obtained by removing noise from the first image data and output to a display device and the second image data, A removal unit that removes noise from the second image data based on the first noise-removed image data and the first weight calculated by the calculation unit, An afterimage reduction processing unit that reduces an afterimage in the second noise-removed image data obtained by removing noise from the second image data by the removal unit, An imaging device having the same.

9. The imaging device according to claim 7 or 8, having an adjustment unit that adjusts a weight, the imaging unit outputs third image data that is temporally later than the second image data, the calculation unit calculates a second weight of the second image data based on the second noise-removed image data that has been subjected to afterimage reduction processing by the afterimage reduction processing unit and the third image data, and calculates a third weight based on the first weight and the second weight, the removal unit removes noise from the third image data based on the second noise-removed image data that has been subjected to afterimage reduction processing and the third weight calculated by the calculation unit, An imaging device in which the afterimage reduction processing unit reduces an afterimage in the third noise-removed image data obtained by removing noise from the third image data by the removal unit.

10. The imaging device according to claim 6, having an extraction unit that extracts first spatial information indicating a feature of a spatial structure included in an image from at least one of the first noise-removed image data and the second image data, An imaging device, wherein the calculation unit calculates the first weight based on the first spatial information extracted by the extraction unit.

11. The imaging device according to claim 6, wherein the afterimage reduction processing unit detects image data of an afterimage existing in the second noise-removed image data based on a difference between the first noise-removed image data and the second image data, and replaces the image data of the afterimage with image data of a region in the second image data corresponding to the detection region of the afterimage.

12. an input unit that receives an input of a frame rate when imaging a subject; an adjustment unit that adjusts the first light amount information to second light amount information greater than the first light amount information from the first light amount information corresponding to the frame rate input by the input unit; an imaging unit that images the subject based on the frame rate and the second light amount information adjusted by the adjustment unit, and outputs first image data and second image data that is temporally later than the first image data; a calculation unit that calculates a first weight of the first image data based on the first noise-removed image data obtained by removing noise from the first image data and output to a display device and the second image data; a removal unit that removes noise from the second image data based on the first noise-removed image data and the first weight calculated by the calculation unit; and an imaging device having the above.

13. The imaging device according to claim 12, having a setting unit that sets imaging conditions for reducing afterimages based on the frame rate and the first light amount information, wherein the imaging unit images the subject based on the frame rate, the second light amount information, and the imaging conditions set by the setting unit, and outputs the first image data and the second image data.

14. The imaging device according to claim 12, having an adjustment unit for adjusting weights, the imaging unit outputs third image data that is temporally later than the second image data, the calculation unit calculates a second weight of the second image data based on the second noise-removed image data from which noise has been removed by the removal unit and the third image data, and calculates a third weight based on the first weight and the second weight, the removal unit removes noise from the third image data based on the second noise-removed image data and the third weight calculated by the calculation unit, an imaging device.

Images