Imaging device, image processing device, image processing method, and storage medium

Abstract

To provide an imaging device or the like capable of generating a composite image with suppressed deterioration of image quality even when there is an image portion that is not good at block matching.SOLUTION: An imaging device 1 includes a lens 11, an imaging element 23, and a main body side microcomputer 41. The computer 41 causes the imaging element 23 to acquire a plurality of pieces of first image data and second image data whose exposure time is longer than that of the first image data, detect a motion area using the first image data and the second image data, and acquire motion detection information using the plurality of pieces of first image data. The computer 41 changes a composition method according to whether the area is a motion area, and generates one sheet of composite image data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an imaging device, an image processing device, an image processing method, and a storage medium that detects the movement of a subject image in image data and synthesizes a plurality of image data. [Background technology]

[0002] Techniques for synthesizing multiple image data to generate a single synthetic image have been proposed in the past. Image synthesis techniques are used, for example, to generate an image with a higher resolution than a captured image, an image with a wider dynamic range than a captured image, and the like.

[0003] Incidentally, when combining multiple images, if there is camera shake or a moving subject in the images, it is necessary to perform motion detection and align the images based on the detection results before combining. In general, block matching (BM) is often used to detect the motion of a subject image in image data. Since the BM method uses, for example, the difference in pixel values, it is preferable to perform the detection between two images with the same exposure time in order to increase the accuracy of position detection.

[0004] For example, JP 2014-160987 A describes a method for synthesizing a high dynamic range (HDR) image from two images with different exposure times, in which two short-exposure images are captured within the period in which one long-exposure image is captured, and motion detection is performed based on the two short-exposure images. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2014-160987 A Summary of the Invention [Problem to be solved by the invention]

[0006] The BM method is a technology that searches for a partial image in a second image that has the smallest difference from a reference partial image in a first image, and detects the movement between the partial images. For this reason, the BM method may erroneously detect or fail to detect movement in subjects with repeating patterns (mesh patterns, brick walls, etc.) or random textures (moving water surfaces, swaying trees, etc.). When multiple images are synthesized based on erroneously detected movement or results where movement cannot be detected, artifacts may appear in the synthesized image, which may actually degrade image quality.

[0007] The present invention has been made in consideration of the above-mentioned circumstances, and aims to provide an imaging device, an image processing device, an image processing method, and a storage medium that can generate a composite image in which degradation of image quality is suppressed even if there are image parts where block matching is difficult. [Means for solving the problem]

[0008] An imaging device according to one aspect of the present invention includes a lens that forms an optical image of a subject, an imaging element that exposes the optical image to generate image data, a first motion detection unit that performs frequency analysis on a plurality of pieces of image data with different exposure times to detect a motion area in the image data, a second motion detection unit that performs block matching on a plurality of pieces of image data with different exposure times to obtain motion detection information of the subject image in the image data, an image synthesis unit that synthesizes the plurality of pieces of image data to generate one piece of synthesized image data, and a control unit that controls the imaging element, the first motion detection unit, the second motion detection unit, and the image synthesis unit, and the control unit controls the imaging element, the first motion detection unit, the second motion detection unit, and the image synthesis unit. The unit causes the image sensor to perform exposure for a first exposure time multiple times to acquire multiple first image data with different exposure times, causes the image sensor to perform exposure for a second exposure time longer than the first exposure time to acquire second image data, causes the first motion detection unit to detect the motion area using the first image data and the second image data, causes the second motion detection unit to acquire the motion detection information using the multiple first image data, and causes the image synthesis unit to change a method of generating the synthetic image data from the multiple first image data based on the motion detection information depending on whether or not it is the motion area.

[0009] An image processing device according to one embodiment of the present invention includes a memory that stores a plurality of first image data having a first exposure time with different exposure times and a second image data having a second exposure time longer than the first exposure time, a first motion detection unit that performs frequency analysis on the first image data and the second image data to detect a motion area in the image data, a second motion detection unit that performs block matching on the plurality of first image data to obtain motion detection information of a subject image in the image data, and an image synthesis unit that changes a method of generating a single piece of synthetic image data based on the motion detection information depending on whether or not the image data is a motion area, and synthesizes the plurality of first image data to generate the synthetic image data.

[0010] An image processing method according to one embodiment of the present invention performs frequency analysis on first image data with a first exposure time and second image data with a second exposure time longer than the first exposure time to detect motion areas in the image data, performs block matching on multiple first image data pieces with the first exposure time having different exposure times to obtain motion detection information of a subject image in the image data, changes a method of generating a single piece of composite image data based on the motion detection information depending on whether or not it is a motion area, and generates the composite image data by combining the multiple first image data pieces.

[0011] A storage medium according to one embodiment of the present invention is a non-transitory storage medium readable by a computer that stores a computer program to cause a computer to perform frequency analysis on first image data having a first exposure time and second image data having a second exposure time longer than the first exposure time to detect motion areas in the image data, perform block matching on multiple first image data pieces having the first exposure time with different exposure times to obtain motion detection information of a subject image in the image data, change a method of generating a single piece of composite image data based on the motion detection information depending on whether or not it is a motion area, and synthesize the multiple first image data pieces to generate the composite image data. Effect of the Invention

[0012] According to the imaging device, image processing device, image processing method, and storage medium of the present invention, a composite image can be generated in which degradation of image quality is suppressed even if there are image portions that are difficult to process using block matching. [Brief description of the drawings]

[0013] [Figure 1] 1 is a block diagram showing the configuration of an imaging apparatus according to a first embodiment of the present invention. [Diagram 2] 6 is a flowchart showing a process of generating a composite image in the imaging device according to the first embodiment. [Diagram 3] 11 is a flowchart showing a first motion detection process in the first embodiment. [Figure 4] 10 is a flowchart showing a second motion detection process in the first embodiment. [Diagram 5] 6 is a flowchart showing processing for arranging the short-exposure images FS(n) in the first embodiment. [Figure 6] 5 is a timing chart showing the order in which a short-exposure image and a long-exposure image are captured in the first embodiment. [Figure 7] FIG. 2 is a diagram for explaining block matching in the first embodiment. [Figure 8] 4 is a diagram for explaining an example of an area unsuitable for block matching in the first embodiment. [Figure 9] 5 shows an example of frequency analysis on a short-exposure image and a long-exposure image in the first embodiment. [Figure 10] FIG. 4 shows an example of a filter used for frequency analysis in the first embodiment. [Figure 11] 8A and 8B are diagrams showing an example of an image sensor capable of varying exposure times according to lines in a second embodiment of the present invention. [Figure 12] 13 is a timing chart showing the order of shooting short-exposure images and long-exposure images according to lines in the second embodiment. [Figure 13] 13 is a flowchart showing a process of generating a composite image in the imaging device according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the embodiment described below. [First embodiment]

[0015] 1 to 10 show a first embodiment of the present invention, and FIG. 1 is a block diagram showing the configuration of an image pickup apparatus 1. FIG.

[0016] In this embodiment, a digital camera is used as an example of the imaging device 1, but the imaging device 1 is not limited to a digital camera and may be any device that has an imaging function. Also, although an interchangeable lens type digital camera is used as an example, a lens-integrated type digital camera may also be used.

[0017] The imaging device 1 is configured by detachably connecting an interchangeable lens 2 and a camera body 3 using a lens mount or the like.

[0018] The interchangeable lens 2 includes a lens 11, an optical aperture 12, a lens driver 13, an aperture driver 14, a lens-side microcomputer 15, and a flash memory 16.

[0019] The lens 11 is configured as a photographing optical system including, for example, one or more optical lenses, and is a lens unit that forms an optical image of a subject (subject image) on an image sensor 23 of the camera body 3, which will be described later.

[0020] The optical diaphragm 12 adjusts the amount of light of the subject image formed by the lens 11 by controlling the passing range of the light flux traveling from the lens 11 to the image sensor 23 .

[0021] The lens driver 13 drives the lens 11 based on a command from the lens side microcomputer 15 to adjust the focus position.

[0022] The aperture driver 14 drives the optical aperture 12 to change the aperture diameter based on a command from the lens side microcomputer 15. By driving the optical aperture 12, the brightness of the subject image changes, and the degree of blur also changes.

[0023] The flash memory 16 is a recording medium for recording the control programs executed by the lens side microcomputer 15 and various information related to the interchangeable lens 2.

[0024] The lens side microcomputer 15 is connected to the lens driver 13, the aperture driver 14, the flash memory 16, and an interface (I / F) 27, which will be described later. The lens side microcomputer 15 communicates with a body side microcomputer 41, which will be described later, via the interface 27, and upon receiving commands from the body side microcomputer 41, reads / writes information recorded in the flash memory 16 and controls the lens driver 13 and the aperture driver 14. Furthermore, the lens side microcomputer 15 transmits various types of information related to the interchangeable lens 2 to the body side microcomputer 41.

[0025] Next, the camera body 3 includes a mechanical shutter 21, a shutter driver 22, an image sensor 23, an image sensor driver 24, an image blur correction mechanism 25, an image blur correction driver 26, an interface (I / F) 27, an analog processing circuit 28, an analog / digital converter (A / D converter) 29, a bus 30, a DRAM (Dynamic Random Access Memory) 31, an AF (Autofocus) processing circuit 32, an AE (Auto Exposure) processing circuit 33, an image processing circuit 34, an LCD (Liquid Crystal Display) driver 35, an LCD 36, a memory interface (memory I / F) 37, a recording medium 38, an operation device 39, a flash memory 40, and a body-side microcomputer 41.

[0026] The mechanical shutter 21 is disposed between the lens 11 and the imaging element 23, and controls the time at which the light beam from the lens 11 reaches the imaging element 23. The mechanical shutter 21 is, for example, an optical shutter configured to move a shutter curtain.

[0027] The shutter driver 22 drives the mechanical shutter 21 based on a command from the main body side microcomputer 41 to open and close the mechanical shutter 21 .

[0028] The shutter driver 22 drives the mechanical shutter 21 in response to a command from the body side microcomputer 41, for example, when taking a still image, and controls the arrival time of the light beam at the image sensor 23, that is, the exposure time of the subject by the image sensor 23.

[0029] On the other hand, the shutter driver 22 maintains the mechanical shutter 21 in an open state, for example, during moving image shooting or live view, in response to a command from the body-side microcomputer 41. At this time, each frame image is acquired by the electronic shutter function of the image sensor 23. However, the electronic shutter may of course also be used when shooting still images.

[0030] The image sensor 23 is an imaging section that has a pixel section in which a plurality of pixels are arranged two-dimensionally at a predetermined pixel pitch, and that exposes a subject image using the plurality of pixels to generate electrical image data. That is, the image sensor 23 photoelectrically converts the subject image formed by the lens 11 and the optical aperture 12 under the control of the body side microcomputer 41, which is an imaging control section, to generate an analog image signal.

[0031] The imaging element 23 of this embodiment is configured as a color imaging element equipped with color filters, and includes multiple types of pixels with different filter colors. Specifically, there are three types of pixels: G pixels with G (green) filters, R pixels with R (red) filters, and B pixels with B (blue) filters. Furthermore, the G pixels are classified into two types: Gr pixels arranged on the same line as the R pixels, and Gb pixels arranged on the same line as the B pixels.

[0032] More specifically, the color filters of the imaging element 23 are arranged in a mosaic pattern of R filters, G filters, and B filters in a primary color Bayer array.

[0033] As is well known, the primary color Bayer array is a basic array of multiple types of pixels with (2,2) pixels, with G pixels arranged at diagonal positions of the basic array, and R and B pixels arranged at the remaining diagonal positions (see FIG. 11, described later). In this case, the G pixels arranged on the same line as the R pixels are Gr pixels, and the G pixels arranged on the same line as the B pixels are Gb pixels, as described above.

[0034] The image sensor 23 is configured such that a basic array of (2,2) pixels is repeated periodically in two dimensions.

[0035] The pixel arrangement of the image sensor 23 is not limited to the Bayer arrangement, and may be any other pixel arrangement. Therefore, the basic arrangement is not limited to (2,2) pixels.

[0036] The imaging element driver 24 drives the imaging element 23 based on a command from the main body side microcomputer 41, and causes the imaging element 23 to capture an image.

[0037] The image blur correction mechanism 25 relatively shifts the positions of the subject image and the imaging element 23 within a predetermined movable range in a direction perpendicular to the optical axis of the lens 11. The image blur correction mechanism 25 shown in Fig. 1 shifts the imaging element 23 to relatively shift the positions of the subject image and the imaging element 23, thereby suppressing image blur. However, the image blur correction mechanism 25 is not limited to this configuration, and may be an image blur correction mechanism 25 that corrects image blur by moving at least a part of the lens 11, or may be an image blur correction mechanism 25 that has both a configuration for shifting the imaging element 23 and a configuration for moving at least a part of the lens 11.

[0038] Specifically, image blur correction mechanism 25 includes, for example, a Hall element and a voice coil motor, and an imaging unit including image sensor 23 is suspended in midair by the magnetic force of the voice coil motor, and its position is moved by controlling the magnetic force while detecting the position by the Hall element. However, image blur correction mechanism 25 is not limited to a configuration using a voice coil motor and a Hall element, and may employ an appropriate configuration using other drive sources and position detection units.

[0039] The image blur correction driver 26 is an image blur correction control unit that controls the image blur correction mechanism 25 to perform image blur correction to reduce movement of the subject image formed on the image sensor 23, and also performs pixel shifting to generate a high-resolution image. The image blur correction driver 26 receives feedback of the control state of the image blur correction mechanism 25 based on a control command from the main body side microcomputer 41, and controls the image blur correction mechanism 25.

[0040] Here, pixel shifting is a process in which, when pixel shift super-resolution shooting mode is set, the relative position between the subject image and the image sensor 23 is shifted by a pixel pitch or a movement unit smaller than the pixel pitch (for example, 1 pixel pitch unit, 0.5 pixel pitch unit, 0.3 pixel pitch unit, etc.).

[0041] Furthermore, the body-side microcomputer 41 generates a control command for controlling the image blur correction driver 26 based on, for example, the output of a blur amount detection sensor (not shown) for detecting the amount of blur such as camera shake occurring in the imaging device 1. Then, based on the control command from the body-side microcomputer 41, the image blur correction driver 26 performs image blur correction by controlling the image blur correction mechanism 25 to shift the imaging element 23 so as to cancel the detected image blur (that is, in the direction opposite to the direction of the detected image blur, by the shift amount corresponding to the magnitude of the detected image blur).

[0042] The interface 27 connects the lens side microcomputer 15 and the body side microcomputer 41 so as to enable two-way communication between them.

[0043] The analog processing circuit 28 reduces reset noise and the like on the analog image signal read out from the image sensor 23, performs waveform shaping, and further performs gain-up to achieve a target brightness.

[0044] The A / D converter 29 converts the analog image signal output from the analog processing circuit 28 into a digital image signal (RAW (low) image data).

[0045] The bus 30 is a transfer path for transferring various data and control signals generated at a certain location in the imaging device 1 to other locations in the imaging device 1. The bus 30 in this embodiment is connected to the A / D converter 29, the DRAM 31, the AF processing circuit 32, the AE processing circuit 33, the image processing circuit 34, the LCD driver 35, the memory I / F 37, and the main body side microcomputer 41.

[0046] The RAW image data output from the A / D converter 29 is transferred via a bus 30 and temporarily stored in a DRAM 31 .

[0047] The DRAM 31 is a storage element that temporarily stores various data such as the above-mentioned RAW image data, image data processed in the image processing circuit 34, and the like.

[0048] The AF processing circuit 32 extracts high-frequency component signals from the RAW image data input via the bus 30, and acquires a focus evaluation value through AF (autofocus) integration processing. The focus evaluation value acquired here is used for AF driving of the lens 11. Of course, AF is not limited to this type of contrast AF, and it may be configured to perform phase difference AF using, for example, a dedicated AF sensor (or AF pixels on the image sensor 23).

[0049] The AE processing circuit 33 extracts luminance components from the RAW image data input via the bus 30, and calculates appropriate exposure conditions (e.g., values ​​such as Tv, Av, and Sv that provide appropriate exposure) based on the extracted luminance components. The appropriate exposure conditions calculated here are used for automatic exposure (AE) control. Specific examples of AE control include control of the optical diaphragm 12 based on the Av value, control of the mechanical shutter 21 based on the Tv value, or exposure timing control of the image sensor 23 based on the Tv value (so-called electronic shutter control), gain control of the analog processing circuit 28 based on the Sv value (or digital gain control of the image processing circuit 34, etc.), etc.

[0050] The image processing circuit 34 performs various image processing such as OB subtraction, white balance (WB) gain, demosaic, noise reduction, color conversion, gamma conversion, enlargement / reduction, etc., on the RAW image data input via the bus 30. Note that data compression / decompression when recording still images or moving images on the recording medium 38 or when reading them from the recording medium 38 may be performed by this image processing circuit 34, or a dedicated compression / decompression circuit may be provided to perform the data compression / decompression.

[0051] The LCD driver 35 is a display device driving unit that reads out image data stored in the DRAM 31 after image processing by the image processing circuit 34, converts the read image data into a video signal, and drives and controls the LCD 36 to display an image based on the video signal on the LCD 36.

[0052] The LCD 36 is a display device, and displays images and various information related to the imaging device 1 through drive control by the LCD driver 35 as described above. Here, the LCD 36 is used as the display device, but the present invention is not limited to this, and other display devices such as an organic EL (organic electroluminescence) device may also be used.

[0053] The memory I / F 37 is a recording control unit that controls the recording of image data onto a recording medium , and also reads out the image data from the recording medium .

[0054] The recording medium 38 is a recording unit that records image data in a non-volatile manner, and is configured, for example, by a memory card that can be attached to and detached from the camera body 3. However, the recording medium 38 is not limited to a memory card, and may be a disk-shaped recording medium or any other recording medium. The recording medium 38 may also be of a type that is built into the camera body 3. Thus, the recording medium 38 is not limited to being a configuration that is unique to the imaging device 1.

[0055] The operation device 39 is used to input various operations to the imaging device 1, and includes operation buttons such as a power button, a release button, a playback button, a menu button, a cross key, and an OK button. The power button is used to turn the power of the imaging device 1 on and off. The release button is used to instruct the start of image capture. The release button is configured as, for example, a two-stage operation button, and includes a 1st (first) release switch and a 2nd (second) release switch. The playback button is used to play back recorded images. The menu button is used to set the imaging device 1, etc. The cross key is used, for example, to select an item. The OK button is used, for example, to confirm a selected item.

[0056] Here, items that can be set using the menu button, cross key, OK button, etc. include the shooting mode (still image shooting mode, pixel shift super-resolution shooting mode, video shooting mode, etc.), recording mode, playback mode, on / off of the image stabilization function, etc. When an operation is performed on the operation device 39, the operation device 39 outputs a signal according to the operation content to the main body side microcomputer 41.

[0057] The flash memory 40 is a non-transient computer-readable recording medium that non-volatilely records processing programs (including a photographing program for photographing an image, an image processing program for synthesizing a high-resolution image based on the photographed image, etc.) executed by the main body side microcomputer 41 and various information related to the imaging device 1. Here, the information recorded in the flash memory 40 includes, for example, the model name and serial number for identifying the imaging device 1, parameters used in image processing, and setting values ​​set by the user. The information recorded in the flash memory 40 is read by the main body side microcomputer 41.

[0058] The body-side microcomputer 41 constitutes a control unit that comprehensively controls the imaging device 1. The body-side microcomputer 41 controls each unit in the camera body 3, and also transmits commands to the lens-side microcomputer 15 via the interface 27 to control the interchangeable lens 2. When a user inputs an operation from the operation device 39, the body-side microcomputer 41 analyzes the input information, reads parameters required for processing from the flash memory 40 according to the processing program recorded in the flash memory 40, and executes various sequences according to the operation content while performing various arithmetic processing, etc.

[0059] Furthermore, the body-side microcomputer 41 controls the optical diaphragm 12 based on the Av value calculated by the AE processing circuit 33. This control is performed by the lens-side microcomputer 15 and the lens driver 13 via the lens driver 13. The body-side microcomputer 41 performs gain control of the analog processing circuit 28 (or digital gain control of the image processing circuit 34) based on the Sv value calculated by the AE processing circuit 33.

[0060] Furthermore, during still image shooting and pixel shift super-resolution shooting, the body side microcomputer 41 controls the mechanical shutter 21 (or the above-mentioned electronic shutter) based on the Tv value calculated by the AE processing circuit 33 via the shutter driver 22 (or the image sensor driver 24).

[0061] When the image blur correction function is set to ON and the pixel shift super-resolution shooting mode is set in the imaging device 1, the image blur correction mechanism 25 performs image blur correction and pixel shift under the control of the body side microcomputer 41 via the image blur correction driver 26.

[0062] When the pixel shift super-resolution shooting mode is set in the imaging device 1, the body side microcomputer 41 synthesizes a plurality of pieces of image data acquired while performing pixel shifting in accordance with an image processing program for synthesizing a high-resolution image stored in the flash memory 40 to generate one piece of synthetic image data (high-resolution image data). Here, the high-resolution image is an image with a higher resolution than one frame of image obtained from the imaging element 23. Therefore, the body side microcomputer 41, the flash memory 40, and the DRAM 31 that stores the image data are a processor that functions as an image synthesis unit.

[0063] In this embodiment, the processor including the main body microcomputer 41 functions as the image synthesis unit, but the present invention is not limited to this, and the image processing circuit 34 may function as the image synthesis unit, or another processor (not shown) may function as the image synthesis unit. The processor may be configured as a computer in which a CPU performs processing according to a computer program (software) stored in a memory, or may be configured with an FPGA (Field Programmable Gate Array) or an electronic circuit dedicated to processing.

[0064] In the configuration example of this embodiment, a processor including a main body side microcomputer 41 functions as a first motion detection unit, and performs frequency analysis on multiple pieces of image data with different exposure times to detect motion areas in the image data.

[0065] In addition, the processor including the main body side microcomputer 41 functions as a second motion detection unit, and performs block matching (BM) on multiple image data with different exposure times to obtain motion detection information of the subject image in the image data.

[0066] Furthermore, the processor including the main body side microcomputer 41 functions as a control unit that controls the image sensor 23, the first motion detection unit, the second motion detection unit, and the image synthesis unit.

[0067] Fig. 2 is a flowchart showing the process for generating a composite image in the imaging device 1. The process shown in Fig. 2 is performed under the control of the main body side microcomputer 41 which constitutes the processor.

[0068] When taking a plurality of images for generating a high-resolution image, if the image stabilization function is off, the subjects in the peripheral parts of the plurality of images do not match, and a high-resolution image cannot be generated in the peripheral parts of the images. Therefore, in the pixel shift super-resolution shooting mode, it is preferable that the image stabilization function is on as a basic setting. For this reason, the following description will be given assuming that the image stabilization function is set to on and the pixel shift super-resolution shooting mode is set. Note that, when shooting with a camera fixed to a tripod, the image stabilization function may be turned off to suppress error factors caused by the image stabilization function.

[0069] When the processing starts, a short exposure time S (first exposure time) and a long exposure time L (second exposure time longer than the first exposure time) are set based on the appropriate exposure conditions set by the AE processing circuit 33, and the number of composite images N based on the pixel shift super-resolution shooting mode is set (step S1).

[0070] Set 0 to a counter n for counting the number of captured images (step S2), and determine whether n < N (step S3).

[0071] Here, if it is determined that n < N, capture a short-exposure image FS(n) (first image data) with a short exposure time S (step S4). Then, after incrementing n (step S5), return to step S3. The processes of step S4 and step S5 are repeatedly performed until it is determined in step S3 that n ≥ N. Thus, the main body side microcomputer 41 constituting the processor causes the image sensor 23 to perform exposure with the short exposure time S a plurality of times, and acquires a plurality of short-exposure images FS(0) to FS(N - 1) with different exposure times. Here, FIG. 6 is a timing chart showing the shooting order of the short-exposure images FS(0) to FS(N - 1) and the long-exposure image FL.

[0072] In step S3, if it is determined that n ≥ N, the main body side microcomputer 41 constituting the processor causes the image sensor 23 to perform exposure with a long exposure time L, and acquires a long-exposure image FL (second image data) (step S6). That is, as shown in FIG. 6, the long-exposure image FL is captured next to the short-exposure image FS(N - 1). At this time, the main body side microcomputer 41 controls the optical aperture 12 so that the exposure amount of the long-exposure image FL is equal to the exposure amounts of the short-exposure images FS(0) to FS(N - 1).

[0073] Note that the control of the exposure time when capturing the short-exposure image FS(n) and the long-exposure image FL is performed by, for example, the mechanical shutter 21. In this case, since the curtain speed is faster than that of the electronic shutter, distortion of a moving subject (so-called rolling shutter distortion) can be reduced.

[0074] Alternatively, the exposure time may be controlled by, for example, an electronic shutter of the image sensor 23. An electronic shutter can shorten the time lag between exposures because it does not require a shutter charge like the mechanical shutter 21. This can reduce the amount of movement of a moving subject between the first short-exposure image FS(0) and the last short-exposure image FS(N-1).

[0075] 6, the long-exposure image FL is taken last, but it is also possible to take the long-exposure image FL first and then take multiple short-exposure images. In this case, the first motion detection is performed using the long-exposure image FL and the short-exposure image taken first (i.e., the short-exposure image taken closest to the long-exposure image FL and highly correlated with it). Furthermore, the long-exposure image FL may be taken in any other order, without being limited to these examples.

[0076] Also, in the example shown in FIG. 6, only one long exposure image FL is captured, but two or more images may be captured to increase the amount of motion detection information.

[0077] After the long exposure image FL is captured, the processor including the main body side microcomputer 41 performs a first motion detection process (step S7). Figure 3 is a flowchart showing the first motion detection process.

[0078] When the processing of FIG. 3 begins, a frequency analysis filter operation (filter processing) is performed on a region of interest of the long exposure image FL shown in column 2A of FIG. 9, for example while moving the region of interest in raster scan order, to generate a long exposure filtered image WL shown in column 2B of FIG. 9 (step S21).

[0079] Fig. 9 is a diagram showing an example of frequency analysis of a short-exposure image FS(N-1) and a long-exposure image FL. Fig. 10 is a diagram showing an example of a filter used in the frequency analysis. The filter shown in Fig. 10 is a two-dimensional Laplacian filter, which is one of the basic filters that can extract two-dimensional edges. Note that the filter used in the frequency analysis is not limited to a two-dimensional Laplacian filter, nor is it limited to an edge extraction filter.

[0080] Next, a frequency analysis filter operation is performed on the region of interest of the short-exposure image FS(N-1) shown in column 1A of Fig. 9 while moving the region of interest in raster scan order, for example, to generate a short-exposure filtered image WS(N-1) shown in column 1B of Fig. 9 (step S22). Here, the reason for using the short-exposure image FS(N-1) among the multiple short-exposure images FS(0) to FS(N-1) is that the image capture time is closest to that of the long-exposure image FL and has the highest correlation with the long-exposure image FL (see Fig. 6).

[0081] In the long-exposure image FL, compared to the short-exposure image FS(N-1), the image portions of the moving subject (the image portions of the truck in the illustrated example) are blurred and the edges are unclear (see the white framed areas in columns 1B and 2B in FIG. 9). For this reason, in the long-exposure filtered image WL, compared to the short-exposure filtered image WS(N-1), fewer edge components are detected in the image portions of the moving subject. Therefore, a difference image is generated by subtracting the long-exposure filtered image WL, which has fewer edge components, from the short-exposure filtered image WS(N-1), which has more edge components (step S23).

[0082] Furthermore, in order to reduce the effect of noise in the difference image, a smoothing filter is applied to the difference image to generate a smoothed difference image (step S24). Note that, if the effect of noise does not need to be taken into consideration, the process of step S24 may be omitted.

[0083] In the smoothed differential image, pixel regions in which edge components equal to or greater than a certain threshold remain (pixel regions in which edge components have decreased in the long-exposure image) are set as motion regions (step S25), and the process returns to the process shown in Fig. 2. Note that, as a method of frequency analysis, in addition to a method using a filter, frequency components may be extracted by Fourier transform and the difference in frequency components between the long-exposure image FL and the short-exposure image FS(N-1) may be analyzed.

[0084] After performing the process of the first motion detection in step S7, the process proceeds to arranging each pixel of the short-exposure image FS(n) in the composite image P having more pixels than the image of one frame. First, each pixel of the first-captured short-exposure image FS(0) is arranged in the composite image P (step S8).

[0085] Next, 1 is set to the counter n (step S9), and it is determined whether n < N (step S10).

[0086] Here, when it is determined that n < N, the processor including the main body side microcomputer 41 performs the process of the second motion detection (step S11). FIG. 4 is a flowchart showing the process of the second motion detection.

[0087] When entering the process of FIG. 4, a template region R0 is set in the short-exposure image FS(0) (step S31). FIG. 7 is a diagram for explaining block matching. Column A of FIG. 7 shows a state where a template region R0 composed of a plurality of pixels is set in a search region RA within the short-exposure image FS(0). Note that the search region RA is a region larger than the template region R0 and including the template region R0.

[0088] Next, as shown in column B of FIG. 7, a target region R1 is set in a search region RB within the short-exposure image FS(n) where n ≧ 1 (step S32). The target region R1 is a region having the same size as the template region R0. The search region RB in the short-exposure image FS(n) is set at the same position as the search region RA in the short-exposure image FS(0) and with the same size as the search region RA. The initial position of the target region R1 is set, for example, at the same position as the template region R0.

[0089] Then, for example, the integrated value SAD (Sum of Absolute Difference) of the absolute difference of each pixel between the template region R0 and the target region R1 is SAD = Σ|R1(i,j) - R0(i,j)| (Step S33). Here, (i, j) are pixel coordinates in the target region R1 and the template region R0, and Σ is taken for (i, j) in the region. The absolute value is calculated in order to evaluate both the strength of the positive edge component and the strength of the negative edge component.

[0090] Although SAD, which requires a small amount of calculation, may be calculated instead of SAD, which requires a larger amount of calculation, such as Sum of Squared Difference (SSD), Normalized Cross-Correlation (NCC), or Zero-means Normalized Cross-Correlation (ZNCC).

[0091] Next, it is determined whether the movement of the target region R1 in the search region RB has been completed (step S34). If it is determined that the movement has not been completed, the target region R1 is moved in the search region RB (step S35), and the process returns to step S33 to calculate the SAD.

[0092] Thus, when it is determined in step S34 that the movement of target region R1 has ended, the position of target region R1 where SAD is the smallest value (the position of target region R1 shown by the dashed line in column B of Figure 7) is searched for as the position to which template region R0 has moved, and a movement vector (motion detection information) as indicated by the white arrow in column B of Figure 7 is detected (step S36).

[0093] If the motion vector is not detected correctly in step S36, information to the effect that motion detection has failed for the template region R0 that was set at the time of performing the processing of step S36 is stored in the DRAM 31, etc. Examples of cases in which the motion vector is not detected correctly include a case in which the minimum value of the SAD is greater than a predetermined threshold value (i.e., a case in which the consistency between the subject in the target region R1 and the subject in the template region R0 is lower than a predetermined value), a case in which the minimum value of the SAD is equal to or less than a predetermined threshold value, but multiple target regions R1 with the minimum SAD are found, etc.

[0094] Next, it is determined whether the movement of the template region R0 in the short-exposure image FS(0) has been completed (step S37), and if it is determined that the movement has not been completed, the template region R0 is moved in the short-exposure image FS(0) (step S38). The template region R0 is moved in the short-exposure image FS(0) in, for example, a raster scan order. After that, the process returns to step S32 and the above-mentioned process is performed.

[0095] If it is determined in step S37 that the movement of the template region R0 has ended, the process returns to the process shown in FIG.

[0096] Fig. 8 is a diagram for explaining an example of an area that is not suitable for block matching. In the example of Fig. 8, the image includes an image portion of a truck moving to the right, and the image portion of a container includes a repeating pattern (repeated design).

[0097] As shown in column A of FIG. 8, a template region R0 is set in the image portion of the container in the short-exposure image FS(0), a target region R1 is set in the short-exposure image FS(n) where n≧1, and the target region R1 is moved within the search region RB. Then, for example, the minimum value of SAD is calculated in target regions R1-1 and R1-2 located at multiple positions. Even if the target region where the correct movement vector should be detected is R1-1, if the minimum value of SAD is also calculated in target region R1-2, it is not possible to determine which region, R1-1 or R1-2, gives the correct movement vector. Then, the correct movement vector cannot be detected, and it is not possible to determine whether there is movement in the image portion of the container.

[0098] Taking these points into consideration, the processor including the main body side microcomputer 41 performs processing for FS(n) arrangement following step S11 (step S12). Fig. 5 is a flowchart showing processing for arranging the short-exposure image FS(n).

[0099] When the process in FIG. 5 begins, one of the pixels in the short-exposure image FS(n) is set as a target pixel, and it is determined whether or not the target pixel is a pixel within a moving area (step S41).

[0100] If it is determined that the pixel is within the motion area, it is further determined whether or not the pixel is one for which the second motion detection has been successful (step S42).

[0101] If it is determined in step S41 that the pixel is not in a motion area, or if it is determined in step S42 that the second motion detection was successful, the target pixel is placed in the composite image P according to the motion vector obtained by block matching in the second motion detection in step S11 (step S43). That is, the main body side microcomputer 41 places pixels in the composite image data based on the motion detection information for areas in the motion area where acquisition of motion detection information was successful and areas other than the motion area.

[0102] On the other hand, if it is determined in step S42 that the pixel is one for which the second motion detection has failed, the process of step S43 is skipped. That is, the main body side microcomputer 41 does not arrange pixels in the composite image data for areas in which the acquisition of motion detection information in the motion area has failed. This prevents artifacts from occurring when the motion vector is not calculated accurately, and suppresses deterioration of image quality.

[0103] Next, it is determined whether or not the processing has been performed with all pixels in the short-exposure image FS(n) as target pixels (step S44). If there are any pixels that have not yet become target pixels, the pixel position of the target pixel is moved (step S45), and the process returns to step S41 to perform the above-mentioned processing.

[0104] In this way, if it is determined in step S44 that the processing has been performed on all pixels in the short-exposure image FS(n) as target pixels, the process returns to the processing shown in FIG.

[0105] After performing the process of step S12, n is incremented (step S13) and the process returns to step S10. The processes of steps S11 to S13 are repeated until it is determined in step S10 that n≧N. As a result, each pixel of the short-exposure images FS(1) to FS(N-1) is arranged in the composite image P.

[0106] In step S10, when it is determined that n≧N, each pixel of the short-exposure images FS(0) to FS(N-1), including FS(0) processed in step S8, is arranged in the composite image P, except in the case where the processing of S43 is skipped.

[0107] Next, the processor including the main body side microcomputer 41 generates a composite image P based on each arranged pixel (step S14). In the composite image P, there are pixel positions where one or more pixels from multiple short-exposure images are arranged, and there may also be pixel positions where no pixel is arranged. Therefore, for pixel positions where multiple pixels are arranged, a normalized pixel value is calculated by dividing the integrated value of the pixel values ​​by the number of pixels. For pixel positions where no pixel is arranged, an interpolation is performed using the normalized surrounding pixel values ​​to calculate the pixel value.

[0108] Once the composite image has been generated, the composite image is displayed on the LCD 36 as necessary under the control of the main body side microcomputer 41, and the composite image is recorded on the recording medium 38 (step S15), and the process returns from the process of FIG. 2 to the main process not shown.

[0109] According to the first embodiment, a frequency analysis is performed on a plurality of image data with different exposure times to detect a motion area, and block matching is performed on a plurality of image data with different exposure times to obtain motion detection information. For an area in the motion area where the motion detection information has not been obtained, pixels are not arranged in the composite image data. Therefore, even if there is an image part where block matching is difficult, it is possible to prevent the occurrence of artifacts and suppress deterioration of image quality. As a result, it is possible to appropriately generate composite image data with a higher resolution than the image data generated by the image sensor 23.

[0110] Furthermore, by controlling the optical diaphragm 12 so that the amount of exposure of the second image data becomes equal to the amount of exposure of the first image data, the accuracy of the frequency analysis can be improved.

[0111] Furthermore, distortion of a moving subject can be reduced when the exposure time is controlled by the mechanical shutter 21. On the other hand, when the exposure time is controlled by the electronic shutter of the image sensor 23, the time lag between exposures can be shortened, and the amount of movement of a moving subject can be reduced. [Second embodiment]

[0112] 11 to 13 show a second embodiment of the present invention, and Fig. 11 is a diagram showing an example of an image sensor capable of varying the exposure time according to the line. In this second embodiment, the same parts as those in the above-mentioned first embodiment are denoted by the same reference numerals, and the description will be omitted as appropriate, and mainly only the differences will be described.

[0113] The configuration of the imaging device 1 in this embodiment is basically the same as the configuration of the first embodiment shown in Fig. 1. However, the imaging element 23 is configured to include an electronic shutter capable of controlling different exposure times for a first line group and a second line group among a plurality of lines in which a plurality of pixels are arranged.

[0114] 11, the image sensor 23 has R, Gr, B, and Gb pixels arranged in a primary color Bayer array. When focusing on lines (basic array lines) consisting of a basic array of (2,2) pixels (two lines consisting of a line of R and Gr pixels and a line of B and Gb pixels adjacent to said line), for example, the odd-numbered basic array lines are called line A and the even-numbered basic array lines are called line B. In this case, the image sensor 23 is capable of controlling different exposure times for line A constituting the second line group and line B constituting the first line group.

[0115] Fig. 12 is a timing chart showing the order of capturing the short-exposure images FAS(n), FBS(n) and long-exposure image FAL(N) corresponding to lines A and B. Fig. 13 is a flowchart showing the process of generating a composite image in the imaging device 1.

[0116] The process in FIG. 13 differs from the process in FIG. 2 in the first embodiment in the following points.

[0117] Instead of the processing of step S4, a short-exposure image FS(n) is captured on lines A and B (step S4A). Specifically, as shown in Fig. 12, a short-exposure image FAS(n) is acquired on line A, a short-exposure image FBS(n) is acquired on line B, and the short-exposure image FS(n) is generated by combining FAS(n) and FBS(n) into one image.

[0118] 12, instead of the processing of step S6, a long-exposure image FAL(N) is acquired from line A (second line group), and a short-exposure image FBS(N) is acquired from line B (first line group) (step S6A). Note that it is also possible to acquire a short-exposure image FAS(N) from line A, and a long-exposure image FBL(N) from line B. In this case, line A becomes the first line group, and line B becomes the second line group.

[0119] That is, the main body side microcomputer 41 constituting the processor exposes a first group of lines of the imaging element 23 for a first exposure time to obtain first image data relating to the first group of lines, and at the same time exposes a second group of lines of the imaging element 23 for a second exposure time longer than the first exposure time to obtain second image data relating to the second group of lines.

[0120] By performing this process, the long-exposure image FAL(N) and the short-exposure image FBS(N) are acquired simultaneously, and the correlation between the images can be increased compared to when they are acquired at different shooting times. In this case, in order to further increase the correlation, it is preferable to match the exposure center time of the long-exposure image FAL(N) with the exposure center time of the short-exposure image FBS(n).

[0121] Instead of the process of step S7, a first motion detection is performed using the long-exposure image FAL(N) and the short-exposure image FBS(N) (step S7A). The first motion detection process is similar to the process shown in FIG. 3, except that the images used are the long-exposure image FAL(N) and the short-exposure image FBS(N).

[0122] That is, the main body side microcomputer 41 constituting the processor detects a movement area using the first image data related to the first line group and the second image data related to the second line group.

[0123] The long-exposure image FAL(N) and the short-exposure image FBS(N) each have half the normal image resolution, but the resolution is sufficient for frequency analysis in the first motion detection. However, these images are not used for high-resolution image synthesis, which prioritizes resolution. The images used for image synthesis are the full-resolution short-exposure images FS(0) to FS(N-1), acquired by setting line A and line B to the same short-exposure time S. Furthermore, the full-resolution short-exposure images FS(0) to FS(N-1) are also used for block matching in the second motion detection.

[0124] Therefore, the processes from step S8 onwards are the same as those in FIG.

[0125] According to the second embodiment, effects substantially similar to those of the first embodiment described above are achieved, and since frequency analysis is performed using the long-exposure image FAL(N) and short-exposure image FBS(N) that are acquired simultaneously and have a high correlation, the movement area in the image data can be detected with higher accuracy.

[0126] Although the present invention has been described above mainly in terms of an imaging device, the present invention is not limited to this, and may also be an image processing device that generates a composite image based on a plurality of first image data having a first exposure time with different exposure times, and second image data having a second exposure time longer than the first exposure time.

[0127] Specifically, in the imaging device 1, for example, the short-exposure images FS(0) to FS(N-1) and the long-exposure image FL are acquired and stored in the recording medium 38. Next, the recording medium 38 is connected to a computer having a CPU and memory, and the short-exposure images FS(0) to FS(N-1) and the long-exposure image FL are stored in the computer's memory. Also, a computer program for synthesizing high-resolution images is stored in the computer's memory. The computer program may be executed by the computer to perform the first motion detection, the second motion detection, and the image synthesis. In this case, the computer having the CPU and memory functions as an image processing device.

[0128] Furthermore, the present invention is not limited to an imaging device or an image processing device, but may also be an imaging method that performs processing similar to that of an imaging device, an image processing method that performs processing similar to that of an image processing device, a computer program for causing a computer to perform processing similar to that of the imaging method, a computer program for causing a computer to perform processing similar to that of the image processing method, a non-transitory storage medium readable by a computer that stores any of these computer programs, etc.

[0129] Here, some examples of storage media that store a computer program product include portable storage media such as a flexible disk or CD-ROM, or storage media such as a hard disk drive (HDD) or a solid state drive (SSD). The storage medium may store not only the entire computer program, but also a part of it. In addition, the entire computer program or a part of it may be distributed or provided via a communication network. A user can install the computer program from the storage medium into a computer, or download the computer program via a communication network and install it on a computer, whereby the computer reads the computer program and executes all or a part of the operation, thereby executing the operation of the imaging device or image processing device described above.

[0130] Furthermore, the present invention is not limited to the above-described embodiment as it is, and in the implementation stage, the components can be modified and embodied without departing from the gist of the invention. In addition, various aspects of the invention can be formed by appropriate combinations of multiple components disclosed in the above embodiments. For example, some components may be deleted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined. In this way, various modifications and applications are possible within the scope of the gist of the invention. [Explanation of symbols]

[0131] 1...Imaging device 2. Interchangeable lenses 3. Camera body 11...Lens 12...Optical aperture 13...Lens driver 14...Aperture driver 15...Lens side microcomputer 16...Flash memory 21...Mechanical shutter 22...Shutter driver 23...Image sensor 24...Image sensor driver 25...Image stabilization mechanism 26…Image stabilization driver 27...Interface 28…Analog processing circuit 29...A / D converter 30…Bus 31…DRAM 32...AF processing circuit 33…AE processing circuit 34...Image processing circuit 35...LCD driver 36...LCD 37…Memory I / F 38...Recording media 39...Operation device 40…Flash memory 41...Main body microcomputer

Claims

1. A lens that forms an optical image of a subject; an image sensor that exposes the optical image to generate image data; a first motion detection unit that performs frequency analysis on a plurality of pieces of image data having different exposure times to detect a motion area in the image data; a second motion detection unit that performs block matching on a plurality of pieces of image data having different exposure times to obtain motion detection information of a subject image in the image data; an image synthesis unit that synthesizes a plurality of pieces of image data to generate a single piece of synthetic image data; a control unit that controls the imaging element, the first motion detection unit, the second motion detection unit, and the image synthesis unit; Equipped with The control unit is causing the image sensor to perform exposure for a first exposure time a plurality of times to obtain a plurality of first image data frames having different exposure times; causing the image sensor to perform exposure for a second exposure time longer than the first exposure time to acquire second image data; causing the first motion detection unit to detect the motion area using the first image data and the second image data; causing the second motion detection unit to acquire the motion detection information using the plurality of first image data; causing the image synthesis unit to change a method for generating the synthetic image data from the plurality of first image data based on the motion detection information depending on whether the region is a motion region or not; 1. An imaging device comprising:

2. The imaging apparatus according to claim 1 , wherein the image synthesis unit generates the synthetic image data having a higher resolution than the image data generated by the imaging element.

3. an optical aperture for adjusting the amount of light of the optical image formed by the lens; The imaging apparatus according to claim 1 , wherein the control unit controls the optical diaphragm so that an exposure amount of the second image data is equal to an exposure amount of the first image data.

4. The camera further includes a mechanical shutter disposed between the lens and the image sensor.

2. The imaging apparatus according to claim 1, wherein the control unit controls an exposure time of the imaging element by using the mechanical shutter.

5. The imaging element has an electronic shutter function, 2. The image pickup apparatus according to claim 1, wherein the control unit controls an exposure time of the image pickup element by using the electronic shutter.

6. The imaging element includes a plurality of lines in which a plurality of pixels are arranged, the electronic shutter is capable of controlling different exposure times for a first line group and a second line group among the plurality of lines; The control unit is exposing the first line group for the first exposure time to obtain the first image data related to the first line group, and at the same time exposing the second line group for the second exposure time to obtain the second image data related to the second line group; causing the first motion detection unit to detect the motion area using the first image data related to the first line group and the second image data related to the second line group; 6. The imaging device according to claim 5.

7. The control unit is the image synthesis unit performs the following for the plurality of first image data: for an area in the motion area where acquisition of the motion detection information has failed, no pixels are arranged in the synthetic image data; and for an area in the motion area where acquisition of the motion detection information has succeeded and an area other than the motion area, pixels are arranged in the synthetic image data based on the motion detection information; and normalization is performed according to the number of pixels arranged at the same position to generate the synthetic image data.

2. The imaging device according to claim 1 .

8. the first motion detection unit performs the frequency analysis by analyzing a difference in edge components extracted by performing a filter process on the plurality of image data, or by analyzing a difference in frequency components extracted by performing a Fourier transform on the plurality of image data; 2. The imaging device according to claim 1 .

9. A memory that stores a plurality of first image data having a first exposure time different from the exposure time, and a second image data having a second exposure time longer than the first exposure time; a first motion detection unit that performs frequency analysis on the first image data and the second image data to detect a motion area in the image data; a second motion detection unit that performs block matching on the plurality of first image data to obtain motion detection information of a subject image in the image data; an image synthesis unit that changes a method for generating one piece of synthetic image data based on the motion detection information depending on whether or not the area is a motion region, and synthesizes the plurality of pieces of first image data to generate the synthetic image data; An image processing device comprising:

10. performing a frequency analysis on first image data having a first exposure time and second image data having a second exposure time longer than the first exposure time to detect a moving area in the image data; performing block matching on a plurality of first image data pieces each having a first exposure time different from each other, to obtain motion detection information of a subject image in the image data; a method for generating one piece of composite image data based on the motion detection information is changed depending on whether the area is a motion region, and the plurality of pieces of first image data are composited to generate the composite image data; 13. An image processing method comprising:

11. On the computer, performing a frequency analysis on first image data having a first exposure time and second image data having a second exposure time longer than the first exposure time to detect a moving area in the image data; performing block matching on a plurality of first image data pieces each having a first exposure time different from each other, thereby obtaining motion detection information of a subject image in the image data; a method for generating one piece of composite image data based on the motion detection information is changed depending on whether the area is a motion region, and the plurality of pieces of first image data are composited to generate the composite image data; A non-transitory computer-readable storage medium storing a computer program for performing the method.

Images