Image processing apparatus, imaging apparatus, and control method for the same

Abstract

To provide an imaging apparatus capable of instructing composite termination for each region while monitoring temporal changes in a composite image during composition of images equivalent to long-exposure photography.SOLUTION: The imaging apparatus has acquisition means for acquiring image data of multiple frames, composition means for composing the acquired image data, composite image display means for sequentially displaying the composite images obtained through the composition means, region segmentation means for segmenting the image data into regions, and composition process interruption means for interrupting the composing process for each segmented segmentation region. The imaging apparatus stops the composing process in accordance with the composition process interruption means.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an imaging device, and a control method for an image processing device, and more particularly to an image synthesis technique. [Background technology]

[0002] A method known as bulb photography is a type of photography that uses long exposures. In bulb photography, the photographer gives instructions from the start to the end of the exposure. During exposure, the image signal is not read from the sensor, so the brightness of the image being captured cannot be confirmed. In other words, it is difficult for the photographer to check midway whether the image was captured at the desired brightness.

[0003] In response to this, there is a technology in which image signals are read out periodically during exposure, sequentially additively combined, and the combined image results are displayed on a display device, as in Patent Document 1. By utilizing this technology, it is possible to instruct the end of exposure, i.e., the end of combination, while checking changes in brightness after shooting has started. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-117395 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the technology described in Patent Document 1, the entire image is additively synthesized using the read image data, so the overall brightness of the synthesized image changes uniformly. Therefore, when an object with different brightness levels is present in the image, it is difficult to obtain an image with appropriate brightness throughout the synthesized image.

[0006] The present invention has been made in consideration of the problems with the conventional technology, and in one aspect thereof, provides an imaging device that is capable of giving instructions to stop composition for each area while checking changes in the brightness of the composite image over time when compositing images. [Means for solving the problem]

[0007] The imaging device of the present invention has an acquisition means for acquiring image data of multiple frames, a synthesis means for synthesizing the image data of the multiple frames, a synthetic image display means for sequentially displaying the synthetic image obtained by the synthesis means, an area division means for dividing the image data into areas, and a synthesis processing interruption means for interrupting the synthesis processing for each divided area divided by the area division means, and stops the synthesis processing in response to the synthesis processing interruption means. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an image processing device that can issue an instruction to stop compositing for each area while checking the change in brightness of the composite image over time when compositing images. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of the functional configuration of an imaging apparatus according to an embodiment; [Figure 2] Flowchart for operations in composite photography mode in an embodiment [Figure 3] FIG. 1 is an explanatory diagram of a division region setting method according to an embodiment; [Figure 4] FIG. 10 is an explanatory diagram of a composite image and a display during shooting in an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] The present invention will be described in detail below based on exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Furthermore, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] The following description will be given of an embodiment of the present invention in an imaging device such as a digital camera. However, the present invention can also be implemented in computer devices (personal computers, tablet computers, media players, PDAs, etc.), smartphones, game consoles, robots, drones, and drive recorders, regardless of whether they have an imaging function, as long as they are capable of combining multiple images. These are merely examples, and the present invention can also be implemented in other electronic devices.

[0012] FIG. 1 is a block diagram showing an example of the functional configuration of an imaging device 10 as an example equipped with an image processing device according to an embodiment of the present invention.

[0013] The control unit 108 has one or more processors capable of executing programs, a RAM, and a ROM. The control unit 108 can load programs stored in the ROM into the RAM and execute them using the processor. By executing the programs, the control unit 108 controls the operation of the components of the imaging device 10, including the functional blocks shown in FIG. 1, and / or external devices connected to the imaging device 10 so as to be able to communicate with the components. The functions of the imaging device 10 can be implemented by the control unit 108 executing the programs. Note that the functions that can be realized by the control unit 108 executing the programs may be implemented using hardware (e.g., ASIC, FPGA, etc.). The ROM is, for example, electrically rewritable and stores programs executable by the processor of the control unit 108, as well as setting values ​​for the imaging device 10, GUI data, etc. The control unit 108 can be in the form of, for example, a system-on-chip (SoC) or a system-on-package (SiP).

[0014] The photographing lens 100 has a plurality of lenses and an aperture, and generates an optical image of a subject. The image sensor 102 is, for example, a CMOS image sensor, and has a plurality of photoelectric conversion units (pixels) arranged two-dimensionally.

[0015] The pixels of the image sensor 102 generate pixel signals having values ​​corresponding to the amount of charge generated during a charge accumulation period. Note that, in photography in which the mechanical shutter 101 is opened and closed, the charge accumulation period corresponds to the exposure period. In photography in which the mechanical shutter 101 is kept open, the charge accumulation period corresponds to the period from when the accumulated charge is reset to when the exposure period has elapsed. Generally, the former corresponds to still image photography, and the latter corresponds to video photography, but the latter can also apply to still image photography. Note that if the image sensor 102 has a global shutter mechanism, the mechanical shutter 101 does not need to be used.

[0016] When one charge accumulation period ends, a group of pixel signals (analog image signals) for one frame is read out from the image sensor 102. If the image sensor 102 can output a group of pixel signals in digital format, the A / D converter 103 may not be required.

[0017] At this stage, each pixel signal has only one color component corresponding to the color of the unit filter of the corresponding pixel. In this specification, such a digital image signal composed of pixel signals having only one color component corresponding to the color of the unit filter is called RAW data.

[0018] The RAW data output from the A / D conversion unit 103 or the image sensor 102 is temporarily stored in the memory 109. The memory 109 is used to temporarily store the RAW data, image data processed by the image processing unit 104, and the like.

[0019] The image processing unit 104 applies development processing to the RAW data. Development processing is a general term for multiple image processing processes, such as color interpolation processing and gradation correction processing (gamma processing). Color interpolation processing is a process that interpolates the values ​​of color components that cannot be obtained at the time of shooting, and is also called demosaicing processing. By applying color interpolation processing, each pixel comes to have multiple color components (for example, RGB or YCbCr) necessary for a color image, and the data is no longer RAW data.

[0020] The image processing unit 104 can apply detection processes such as detection of feature regions (for example, face regions or human body regions) and their movements, person recognition processes, synthesis processes, scaling processes, encoding processes, and decoding processes to the developed image data. The image processing unit 104 can also apply various image processing processes such as data processing such as header information generation processes, generation of signals and evaluation values ​​used for autofocus detection (AF), and evaluation value calculation processes such as calculation of evaluation values ​​used for automatic exposure control (AE). Note that these are examples of image processing that the image processing unit 104 can apply, and do not limit the image processing that the image processing unit 104 applies.

[0021] The image processing unit 104 can also combine multiple pieces of RAW data that have been captured and store the combined RAW data in the memory 109 .

[0022] In the image processing unit 104, for example, an addition mode, an average addition mode, and a comparatively bright mode can be selected as an executable synthesis processing mode.

[0023] The image synthesis method in each synthesis mode will be described. Assume that N frames (N is an integer equal to or greater than 2) of images are to be synthesized. The pixels constituting the image of each frame have coordinates (x, y) in an xy Cartesian coordinate system, and the luminance value of the pixel at coordinates (x, y) is I_i(x, y) (i = 1 to N), and the luminance value of the pixel at coordinates (x, y) in the synthesized image is I(x, y). The synthesis unit 108 calculates the luminance value I(x, y) of each pixel of the synthesized image as follows, depending on the synthesis mode. Addition mode I(x,y)=I_1(x,y)+I_2(x,y)+···+I_N(x,y)

[0024] In the addition mode, the composition unit 108 generates a composite image by adding the luminance values ​​of pixels at the same coordinates in each frame. The addition mode is used, for example, when generating an image with proper exposure by combining images of N frames captured with an exposure amount that is 1 / N of the proper exposure amount. - Additive averaging mode I(x,y)=(I_1(x,y)+I_2(x,y)+···+I_N(x,y)) / N

[0025] In the averaging mode, the synthesis unit 108 generates a synthesized image in which the luminance value of each pixel is the average value of N frames by dividing the calculated luminance value by the number of frames N, just as in the additive mode. The averaging mode is used, for example, to reduce noise in images captured at high sensitivity. Brighten Mode I(x,y)=max(I_1(x,y),I_2(x,y),...,I_N(x,y))

[0026] Here, max() is a function that extracts the maximum value of the elements in (). A composite image is obtained that is composed of the maximum brightness value of the N pixels that have the same coordinates in each frame. The comparatively bright mode is effective when compositing images of fireworks or starry skies, for example.

[0027] The display unit 105 is, for example, a touch display, and is used to display live view images, playback images, GUI, setting values ​​and information of the image capture device 10, and the like.

[0028] The operation unit 106 is a general term for input devices (such as switches, keys, buttons, dials, and touch panels) that allow the user to give various instructions to the imaging device 10. For example, the display unit 105 can be used as a touch panel and as one of the operation units 106.

[0029] The recording unit 107 records RAW data, composite RAW data, developed image data, and audio data associated with these data on a recording medium such as a memory card according to the shooting mode and recording settings.

[0030] Next, the operation of the imaging device 10 in composite shooting mode will be described using the flowchart in Fig. 2. The composite shooting mode is a shooting mode in which a composite image obtained by combining multiple frame images captured over time is displayed one after another, and the shooting (exposure) is ended based on the photographer's instructions, and the image is recorded. In this embodiment, the addition mode is used, but the composite mode may also be selectable. Furthermore, the composite shooting mode described in this embodiment can impart an effect equivalent to a long-time exposure to the composite image by combining multiple frame images captured over time.

[0031] In S201, the control unit 108 accepts settings of shooting conditions from the user via the operation unit 106. The shooting conditions set at this time are exposure conditions, such as sensitivity, aperture value, and shutter speed for sequential shooting. Depending on the shutter speed for sequential shooting and the number of images to be combined, an effect equivalent to a long exposure is imparted to the combined image.

[0032] In S202, the control unit 108 sets a divided region based on an input to the operation unit 106. As shown in FIG. 3(a), the divided region is set by the user setting the coordinates of a dividing line 301 within the region of an image (such as a live view image) displayed on the display unit 105, thereby dividing the region of the image. The coordinates of the dividing line 301 can be set using a touch panel or a button on the operation unit 106. In FIG. 3(b), the region is divided into a first region 302 and a second region 303, centered on the dividing line 301. In this embodiment, the region is set using a dividing line, but the region may also be divided into continuous regions according to brightness ranges in the image, and each of these regions may be used as a divided region. Furthermore, although the region is divided by a single dividing line 301 in FIG. 3, the region may also be divided by multiple dividing lines.

[0033] In S203, the control unit 108 detects that a shooting instruction has been input from the user via the operation unit 106. The shooting instruction may be, for example, a full press of the shutter button included in the operation unit 106. When the input of the shooting instruction is detected, the control unit 108 executes S204. In S204, the control unit 108 executes shooting processing for one frame. Then, the control unit 108 stores the RAW data obtained by shooting in the memory 109. When shooting the first frame (the first image after detecting the shooting instruction), the control unit 108 skips the synthesis processing in S205 and executes S206. Note that when shooting continues from the second frame onwards in the flowchart described below, the control unit 108 controls to execute S205.

[0034] In S205, the image processing unit 104 creates a new composite image (new composite RAW data) by combining the image (RAW data) obtained by the most recent shooting with the composite image (composite RAW data) up to the previous frame. The image processing unit 104 executes the composition process according to the set mode.

[0035] In S206, the control unit 108 displays the composite image created in S105 on the display unit 105. In this embodiment, since the composite image is RAW data, the image processing unit 104 performs development processing, and the developed composite image is displayed on the display unit 105.

[0036] In S207, the control unit 108 detects whether an instruction to stop compositing has been issued for the divided area, and the process proceeds to S208. The instruction to stop compositing is issued, for example, by the user selecting an area for which the user wants to stop exposure during continuous image capture processing. The area can be selected using a touch panel (which may also be the display unit 105) included in the operation unit 106 or a button. For example, when selecting using a touch panel, the user touches the divided area of ​​the composite image for which the user wants to stop exposure on the touch panel while checking the composite image displayed on the display unit 105, at the timing when the user wants to stop exposure. When selecting using a button, for example, the currently selected divided area is highlighted so that it can be confirmed on the display unit 105, and the control unit 108 detects that the button has been pressed, and controls to stop compositing of that area. The selected area may be switched, for example, by pressing a button assigned for area selection.

[0037] The above describes a case where the control unit 108 detects an instruction to stop compositing based on a user operation. Note that the image processing unit 104 may integrate the luminance of the divided regions and terminate compositing of the regions when the luminance reaches or exceeds a predetermined luminance value. The predetermined luminance value may be a value preset on the camera side, or may be a value that the user can select and set via the operation unit 106.

[0038] In S208, if the control unit 108 determines that no instruction to stop composition has been issued for all divided areas, it executes S209; if an instruction has been issued, it determines that photography has ended and executes S211.

[0039] In S209, the control unit 108 detects whether there is an instruction to stop shooting by a user operation, and proceeds to S210. The instruction to stop shooting is given, for example, by fully pressing the shutter button included in the operation unit 106. If an instruction to stop shooting is not detected in S210, the processes of S204 to S208 are repeatedly executed. At this time, the composite image that the control unit 108 causes the display unit 105 to display in S206 is a composite image (composite RAW data) of the image (RAW data) obtained in the most recent shooting (second time point) and the previous frame (first time point). In other words, the composite image that the control unit 108 causes the display unit 105 to display is updated to the latest composite image.

[0040] If an instruction to stop shooting is detected in S210, it is determined that shooting has ended, and S211 is executed. Note that the control unit 108 controls the imaging device 10 so that shooting continues while a shooting instruction is continuously input (while an instruction to stop shooting is not given).

[0041] In S211, the control unit 108 stops the imaging process, and the image processing unit 104 develops the composite image that has been processed up to that point, and generates an image data file storing the image data from the image data after the development process. The generated image data file is sent to the recording unit 107, and the process proceeds to S212.

[0042] In S212, the recording unit 107 records the image data file generated in S111 on a recording medium such as a memory card. Note that in addition to or instead of the developed image data, composite RAW data may also be recorded.

[0043] When recording by the recording unit 107 is completed, the control unit 108 ends the operation shown in Fig. 2 and returns to, for example, a shooting standby state. This concludes the description of the flowchart in Fig. 2.

[0044] Next, a composite image during a series of shooting and the display of the composite image during shooting will be described with reference to Fig. 4. The horizontal axis in Fig. 4 indicates the passage of time. Here, examples of composite images at times t1, t2, and t3 after shooting begins and composite image displays that can be displayed on the display unit 105 during shooting are shown. For simplicity, the description will be given assuming that the order from the start to the end of shooting is t1, t2, and t3.

[0045] A composite image 401 at time t1 after the start of shooting is shown, and a display of the image being composited on the display unit 105 is shown as 411. In the image being composited 411, a white dotted line acts as a dividing line, dividing the image into two regions. In this case, no instruction to stop composition has been given for any of the regions, so the image composition unit 104 performs composition processing on all regions.

[0046] At time t2, a composite image 402 is displayed when the user issues an instruction to stop combining the lower divided area of ​​the two divided areas, and a display of the image being combined on the display unit 105 is shown as 412. The display unit 105 grays out the lower area for which the instruction to stop combining has been issued in the image being combined 412. The gray-out display is achieved by having the image processing unit 104 fill in the area for which combining has been stopped with a single color in the display image obtained by developing the composite image, and displaying the resulting image on the display unit 105. At this time, the composite image 402 is brighter than the composite image 401 in all areas. Note that the method of indicating that an instruction to stop combining has been issued is not limited to this. For example, the user may be notified that combining has been completed by displaying an icon or the like in part of the area for which combining has been stopped, indicating that combining has been completed. The user may also be notified that combining has been completed by highlighting the outline of the area for which combining has been stopped.

[0047] At time t3, the user instructs to stop compositing for the upper divided area, and the composite image when compositing for all divided areas is stopped is shown in 403, and the display of the image being composited on the display unit 105 is shown in 413. As all divided areas are selected, the image being composited display 411 is displayed as grayed out, and shooting ends. At this time, because compositing for the lower divided areas has stopped, only the upper area of ​​composite image 403 is brighter than composite image 402, and the exposure time and brightness can be changed for each divided area.

[0048] As described above, according to this embodiment, when combining images, it is possible to issue an instruction to stop combining for each region while checking the change over time in brightness of the combined image.

[0049] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0050] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible. Therefore, the following claims are appended to clarify the scope of the invention.

[0051] The disclosure of this embodiment includes the following configuration, method, program, and storage medium.

[0052] (Configuration 1) An image processing device comprising: a division means for dividing a plurality of images taken over time into the same region; a synthesis means for creating a composite image from the plurality of images; an instruction means for instructing the synthesis means to end the synthesis; and a display unit for displaying the composite image, wherein the synthesis means ends the synthesis process for each region divided by the division means based on an instruction from the instruction means.

[0053] (Configuration 2) 2. The image processing device according to configuration 1, wherein the dividing means divides each of the plurality of images into two or more regions.

[0054] (Configuration 3) 3. The image processing device according to configuration 1 or 2, wherein the instruction means is capable of selecting at least one of the divided areas.

[0055] (Configuration 4) The image processing device according to any one of configurations 1 to 3, characterized in that the composite image is a first composite image created from a plurality of images taken up to a first time point, and a second composite image is newly created using an image taken after the first time point.

[0056] (Configuration 5) 5. The image processing device according to configuration 4, further comprising: a control means for controlling the display of the display unit, wherein the control means displays the composite image created by the composition means on the display unit.

[0057] (Configuration 6) 6. The image processing device according to configuration 5, wherein the control means controls the display on the display unit so as to change the display every time the synthesis means creates a new synthetic image.

[0058] (Configuration 7) 8. The image processing device according to any one of configurations 1 to 7, wherein the control means changes the display format of the area of ​​the composite image on the display unit after the composition by the composition means.

[0059] (Configuration 8) 8. The image processing device according to claim 7, wherein the control means highlights an area of ​​the composite image in which the composition means has completed composition.

[0060] (Configuration 9) The image processing device according to any one of configurations 1 to 9, characterized in that the composite image is given the effect of taking a longer time than the multiple images taken over time that were used to create the composite image.

[0061] (Configuration 10) 11. The image processing device according to any one of configurations 1 to 10, wherein the synthesis means creates the synthetic image by additive synthesis, additive averaging, or comparative brightness.

[0062] (Configuration 11) 11. An imaging device comprising: an imaging means; and the image processing device according to any one of configurations 1 to 10.

[0063] (Method 1) A control method for an image processing device, comprising: a division step for dividing a plurality of images taken over time into the same region; a compositing step for creating a composite image from the plurality of images; an instruction step for instructing the compositing means to end the compositing; and a display step for displaying the composite image, wherein the compositing step ends the compositing process for each region divided by the division step based on an instruction in the instruction step.

[0064] (Program 1) 11. A program for causing a computer to function as each of the means included in the image processing device according to any one of configurations 1 to 10.

[0065] (Storage medium 1) A computer-readable storage medium storing a program for causing a computer to function as each of the means possessed by the image processing device according to any one of configurations 1 to 10. [Explanation of symbols]

[0066] 10 Camera 100 Photographic Lenses 101 Mechanical Shutter 102 Image sensor 103 A / D conversion section 104 Image processing section 105 Display section 106 Operation section 107 Recording Section 108 Control Unit 109 Memory

Claims

1. a division means for dividing a plurality of images captured over time into the same regions; a synthesis means for creating a synthetic image from the plurality of images; an instruction means for instructing the synthesis means to end synthesis; a display unit that displays the composite image, The image processing device according to claim 1, wherein the combining means completes the combining process for each area divided by the dividing means based on an instruction from the instruction means.

2. 2. The image processing apparatus according to claim 1, wherein said dividing means divides each of said plurality of images into two or more regions.

3. 2. The image processing apparatus according to claim 1, wherein the instruction means is capable of selecting at least one of the divided areas.

4. 2. The image processing device according to claim 1, wherein the composite image is a first composite image created from a plurality of images taken up to a first time point, and a second composite image is newly created using an image taken after the first time point.

5. and a control unit for controlling the display of the display unit.

5. The image processing apparatus according to claim 4, wherein said control means displays the composite image created by said composition means on said display unit.

6. 6. The image processing apparatus according to claim 5, wherein said control means controls so that the display on said display unit is changed every time said synthesis means creates a new synthetic image.

7. 2. The image processing apparatus according to claim 1, wherein the control means changes the display form of the area of ​​the composite image on the display unit after the composition by the composition means.

8. 8. The image processing apparatus according to claim 7, wherein said control means highlights an area of ​​the composite image in which said combining means has completed the combining.

9. 2. The image processing device according to claim 1, wherein the composite image is given an effect of being taken at a longer time than the plurality of images taken over time that were used to create the composite image.

10. 2. The image processing apparatus according to claim 1, wherein said synthesis means creates said synthetic image by any one of additive synthesis, additive averaging, and comparative brightness.

11. An imaging means; An image processing device according to any one of claims 1 to 10; An imaging device having the above configuration.

12. a segmentation step of performing the same region segmentation on each of a plurality of images captured over time; a compositing step of creating a composite image from the plurality of images; an instruction step of instructing the synthesis means to end synthesis; a display step of displaying the composite image, The control method for an image processing apparatus, wherein the combining step completes a combining process for each area divided by the dividing step based on an instruction in the instruction step.

13. A program for causing a computer to function as each of the means included in the image processing device according to any one of claims 1 to 10.

14. 11. A computer-readable storage medium storing a program for causing a computer to function as each of the means included in the image processing device according to claim 1.

Images