Image processing device, imaging device, control method, and program

Abstract

The present invention provides an image processing device that can eliminate hue shift, brightness reduction, etc., even when pixels in a specific region of an image captured by segmented exposure are saturated. [Solution] An image processing apparatus for performing image processing on a given set of multiple exposure images, comprising: an extraction means for extracting a predetermined region in each of the multiple exposure images; and a synthesis means for generating a composite image using the extracted region, wherein the extraction means extracts saturated and unsaturated regions from each of the multiple images, and the synthesis means generates a composite image of saturated regions using saturated regions, generates a composite image of unsaturated regions using unsaturated regions, and synthesizes the composite image of saturated regions and the composite image of unsaturated regions.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an imaging apparatus, a control method, and a program.

Background Art

[0002] Patent Document 1 discloses a technique for suppressing the occurrence of overexposure, such as that similar to a physical ND filter, by performing synthesis on a plurality of images captured by split exposure without attaching a physical ND filter. Further, when the pixel values of a certain region in the image captured by split exposure are saturated, a phenomenon may occur in which the hue of the region corresponding to the certain region in the synthesized image after the addition average processing is shifted and the luminance decreases. Patent Document 2 discloses a technique for suppressing such hue shift and luminance decrease.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, when the pixels of a certain region in the image captured by split exposure are saturated, although such phenomena as hue shift and luminance decrease occurring in the image after the addition average processing can be suppressed, the current situation is that a radical solution has not been achieved.

[0005] An object of the present invention is to provide an image processing apparatus, an imaging apparatus, a control method, and a program capable of eliminating hue shift, luminance decrease, etc., even when the pixels of a specific region in the imaging image by split exposure are saturated.

Means for Solving the Problems

[0006] To achieve the above objective, one aspect of the present invention is an image processing apparatus for performing image processing on a plurality of images acquired by segmented exposure, comprising: an extraction means for extracting a predetermined region in each of the plurality of images; and a synthesis means for generating a composite image using the extracted region, wherein the extraction means extracts saturated regions and unsaturated regions in each of the plurality of images, and the synthesis means generates a composite image of saturated regions using the saturated regions, generates a composite image of unsaturated regions using the unsaturated regions, and synthesizes the composite image of saturated regions and the composite image of unsaturated regions. [Effects of the Invention]

[0007] According to the present invention, even when pixels in a specific region of an image captured by segmented exposure are saturated, it is possible to eliminate hue shift, brightness reduction, and other issues. [Brief explanation of the drawing]

[0008] [Figure 1] This is a configuration diagram of an imaging device according to an embodiment of the present invention. [Figure 2] This is a flowchart showing the process of the first embodiment of the present invention. [Figure 3] This is a flowchart showing the process of the second embodiment of the present invention. [Figure 4] This is an explanatory diagram of segmented exposure and composite exposure. [Figure 5] This is an explanatory diagram of the composite image in the present invention. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described in detail below with reference to the drawings. However, the configurations described in the following embodiments are merely illustrative, and the scope of the present invention is not limited to the configurations described in the embodiments.

[0010] This invention aims to achieve an effect equivalent to the "slow shutter effect" of an ND filter by combining adjustments such as ISO to match the brightness of a non-split exposure and additive averaging during "split exposure."

[0011] (Saturated region: Composite image of saturated region: Unsaturated region: Composite image of unsaturated region) Furthermore, a "saturated region" is a region where the brightness exceeds a predetermined value, causing problems (such as overexposure), while a "non-saturated region" is a region where the brightness is below a predetermined value and no problems occur. Both "saturated regions" and "non-saturated regions" are extracted from captured images. Additionally, a "saturated region composite image" is created by combining the "saturated regions" from multiple images, and similarly, a "non-saturated region composite image" is created by combining the "non-saturated regions" from multiple images.

[0012] Furthermore, "segmented exposure" refers to continuously capturing images over a continuous exposure period. This invention achieves an effect equivalent to the "slow shutter effect" of an ND filter by combining adjustments such as ISO to match the brightness of a non-segmented exposure with additive averaging during segmented exposure.

[0013] <Configuration: Figure 1> Figure 1 is a block diagram showing the basic configuration of an imaging device 100 equipped with an image processing apparatus according to an embodiment of the present invention. The imaging device 100 may be an imaging device such as a digital camera or a digital video camera. Alternatively, it may be an electronic device equipped with a camera function, such as a mobile phone with a camera function or a computer with a camera function.

[0014] The imaging device 100 comprises an optical system 101, an image sensor 102, a CPU 103, a display unit 108, an operation unit 109, an image processing device 105, a primary storage device 104, and a secondary storage device 107. The imaging device 100 is configured to allow the attachment and detachment of a recording medium 106.

[0015] (Optical system 101: Image sensor 102) The optical system 101 is composed of a lens, a shutter, an aperture, etc. The optical system 101 forms a subject image on the imaging device 102. The imaging device 102 photoelectrically converts the formed subject image and outputs an imaging signal (imaging image). The imaging device 102 is realized by, for example, a CCD image sensor, a CMOS image sensor, etc.

[0016] (CPU 103: Primary storage device 104) The CPU 103 controls each part constituting the imaging device 100 by executing a program stored in advance. The primary storage device 104 is a volatile storage device such as a RAM. A work area for the CPU 103 is formed in the primary storage device 104, and temporary data when the CPU 103 executes a program is stored. Further, the primary storage device 104 can also store image data on which the image processing device 105 performs various image processes and image data stored in the attached recording medium 106.

[0017] (Secondary storage device 107) The secondary storage device 107 is a non-volatile storage device such as an EEPROM or a flash memory. The secondary storage device 107 stores a program (firmware) for controlling the imaging device 100, various setting information set by the user operating the operation unit 109, etc. The CPU 103 reads the stored content stored in the secondary storage device 107 and writes data to the secondary storage device 107.

[0018] (Display unit 108: Operation unit 109) The display unit 108 displays various information. The display unit 108 can be realized by a liquid crystal device, an EL device, etc. Various information displayed by the display unit 108 includes a viewfinder image at the time of imaging, an imaging image, a GUI image enabling interactive operations, etc. The operation unit 109 is a group of input devices that receives the user's operations and sends a signal corresponding to the received operation to the CPU 103. The operation unit 109 is realized by operation members such as buttons, levers, touch panels, etc. The operation unit 109 may be configured to be able to receive various operations using sound, line of sight, etc.

[0019] (Recording medium 106) The recording medium 106 records the captured image data and the like stored in the primary storage device 104. The recording medium 106 is a recording device detachable from the imaging device 100, such as a semiconductor memory card, a USB memory, etc. The image data and the like recorded on the recording medium 106 can also be mounted on an electronic device such as a PC (personal computer), which is another device. Therefore, the image data and the like captured by the imaging device 100 are read and stored in another electronic device on which the recording medium 106 is mounted. Thus, the imaging device 100 has a detaching / attaching mechanism and a read / write function for the recording medium 106.

[0020] (Image processing device 105) The image processing device 105 can perform a plurality of types of image processing on the imaging signal (captured image) output from the image sensor 102. The user can operate the operation unit 109 to set the "imaging mode", and the image processing device 105 executes various image processing according to the set "imaging mode".

[0021] The image processing device 105 performs various image processing such as tone adjustment according to the "imaging mode", as well as the image processing called "development processing", on the captured image. Further, by the CPU 103 executing an image processing program, a part or all of the functions of the image processing device 105 can be realized. A part or all of the functions of the image processing device 105 may be realized by hardware such as an ASIC circuit.

[0022] (Fig. 4: Explanation of split exposure and synthesis) Fig. 4 is an explanatory diagram of a conventional example of split exposure and synthesis. It is an explanatory diagram of an example of suppressing whiteout and the like in the same manner as a physical ND filter by split exposure. In Fig. 4, imaging of a vehicle (high-brightness moving subject) is assumed as an example.

[0023] As shown in Figure 4(a), when imaging a "high-luminance moving subject" such as a vehicle with its headlights on using segmented exposure, each captured image will look like Figures 4(b) to 4(e). Figure 4(g) shows the image when using "long exposure imaging" with an ND filter.

[0024] Now, when the images in Figures 4(b) to 4(e) are subjected to "additive averaging," as shown in Figure 4(f), a hue shift and brightness reduction occur in the trajectory of a high-brightness moving subject. This is because, while "long exposure imaging" would normally involve additive averaging, imaging using "segmented exposure" results in "additive averaging," which is the process of adding and averaging each image.

[0025] Therefore, by using additive averaging, saturated and unsaturated regions are averaged out, resulting in a decrease in brightness compared to the actual value. Note that while Figures 4(b) to 4(e) show examples of segmented exposure using four captured images, the number of captured images is not limited to four, and the above-mentioned problem persists.

[0026] Therefore, in the embodiment of the present invention, a "composite image of the saturated region" and a "composite image of the unsaturated region" are created. Then, by combining the "composite image of the saturated region" and the "composite image of the unsaturated region," the focus is on eliminating the breaks in the trajectory of the "high-luminance moving subject" (see Figure 5).

[0027] <First Embodiment> Figure 2 is a flowchart showing the image synthesis process according to the first embodiment of the present invention. The first embodiment will be described with reference to Figure 2. In the following, imaging by segmented exposure is assumed.

[0028] (Step S201: Step S202: Step S203) First, in step S201, the CPU 103 controls the imaging device 100 to capture the "first" image and send it to the image processing device 105. Next, in step S202, the image processing device 105 extracts the "saturated region" from the first captured image and stores it in the primary storage device 104. Next, in step S203, the image processing device 105 extracts the "non-saturated region" from the first captured image and stores it in the primary storage device 104. Then, the image processing device 105 performs the same process from step S201 to step S203 for the second to "N-1" captured images.

[0029] (Step S204) Next, in step S204, the CPU 103 captures the Nth image and sends it to the image processing device 105. The image processing device 105 extracts the saturated and unsaturated regions from the Nth captured image and stores them in the primary storage device 104. Through this image processing, the "saturated region" and "unsaturated region" are extracted from each of the multiple captured images.

[0030] (Step S205) Next, in step S205, the image processing device 105 performs image synthesis of the "saturated regions" from the first to the "N-1" captured images and the "saturated regions" of the Nth captured image. More specifically, the image processing device 105 reads the "saturated regions" extracted from the first to the Nth images from the primary storage device 104 and synthesizes the images of each saturated region. This synthesis generates a "saturated region composite image". Furthermore, the synthesis method of the saturated regions is not limited as long as it does not cause hue shift or brightness reduction in the saturated regions (see Figure 5(a)). For example, in addition to additive averaging synthesis of each saturated region, there is also a method of comparative brightness synthesis. "Comparative brightness synthesis" is a method that compares two images pixel by pixel and synthesizes the pixels with higher brightness for each pixel.

[0031] (Step S206) Next, in step S206, the image processing device 105 performs image synthesis of the "unsaturated regions" of the first to "N-1" captured images and the "unsaturated region" of the Nth captured image (see Figure 5(b)). More specifically, the image processing device 105 reads the "unsaturated regions" of the first to Nth images from the primary storage device 104 and synthesizes the unsaturated regions of each captured image. This synthesis generates an "unsaturated region composite image". The synthesis method can be the same as in step S205, including additive averaging synthesis and comparative brightness synthesis.

[0032] (Step S207) In step S207, the CPU 103 determines whether or not imaging is complete. If the CPU 103 determines that imaging is complete (Yes), it proceeds to step S208; otherwise, it returns to step S204. In other words, the CPU 103 repeatedly processes the image synthesis of saturated and unsaturated regions of the first to Nth captured images until the image processing device 105 finishes, and proceeds to step S208 when the processing is complete.

[0033] (Step S208) Next, in step S208, the image processing device 105 combines the saturated region composite image obtained in step S205 with the unsaturated region composite image obtained in step S206 (see Figure 5(c)). Then, in step S209, the image processing device 105 develops the composite image obtained in step S208.

[0034] By generating a composite image of the saturated region and a composite image of the unsaturated region as shown in the first embodiment, and then combining the two, it is possible to suppress overexposure and other issues, similar to a physical ND filter, and prevent hue shift and brightness reduction caused by split exposure.

[0035] <Second Embodiment> The second embodiment of the present invention is characterized by the creation of a "map" in addition to the synthesis process in the first embodiment, and by referring to this map to correct the saturated region composite image and the unsaturated region composite image. Figure 3 is a flowchart of the process of the second embodiment. The second embodiment will be described below with reference to Figure 3.

[0036] (Step S301: Step S302) First, in step S301, the CPU 103 controls the imaging device 100 to capture the first image and send it to the image processing device 105. Next, in step S302, the image processing device 105 extracts the saturated region from the first captured image and stores it in the primary storage device 104.

[0037] (Step S303: Map) Next, in step S303, the image processing device 105 creates a map and stores it in the primary storage device 104. This map stores the "region type," the "number of images (number used during synthesis)" involved in the composite image of the region of that region type, and the "position" of the region of that region type on the composite image. Here, "region type" refers to information indicating whether it is a "saturated region" or a "non-saturated region."

[0038] (Step S304) The image processing device 105 can refer to the map to determine, for example, the number of saturated region images used to combine images of a saturated region at a certain location. Conversely, the image processing device 105 can refer to the map to determine the number of non-saturated region images used to combine images of a non-saturated region at a certain location. Next, in step S304, the image processing device 105 extracts the non-saturated region from the first captured image and stores it in the primary storage device 104. Then, the image processing device 105 performs the same processing from steps S301 to S304 for the second to "N-1" captured images.

[0039] (Step S305) Next, in step S305, the CPU 103 captures the Nth image and sends it to the image processing device 105. The image processing device 105 extracts the saturated region from the Nth captured image and stores it in the primary storage device 104.

[0040] (Step S306; Step S307) Next, in step S306, the image processing device 105 performs image synthesis of the saturated regions in the first to "N-1" captured images and the saturated region of the Nth captured image. More specifically, the image processing device 105 reads the extracted saturated regions from the first to Nth images from the primary storage device 104 and synthesizes the images of each saturated region. This generates a "saturated region composite image". The synthesis method in step S306 will be explained in the saturated region composite image correction in step S310. Next, in step S307, the image processing device 105 updates the contents of the map. For example, if a saturated region exists in the Nth captured image, the image processing device 105 adds it to the map.

[0041] (Step S308) Next, in step S308, the image processing device 105 performs image synthesis of the "unsaturated regions" of the first to "N-1" captured images and the "unsaturated region" of the Nth captured image. This generates an "unsaturated region composite image". More specifically, the image processing device 105 reads the extracted unsaturated regions from the first to Nth images from the primary storage device 104 and synthesizes the images of each unsaturated region. The synthesis method in step S308 will be explained when the correction of the unsaturated region composite image is explained in step S311.

[0042] (Step S309) In step S309, the CPU 103 determines whether or not imaging is complete. If the CPU 103 determines that imaging is complete (Yes), it proceeds to step S310; otherwise, it returns to step S305. In other words, the CPU 103 repeatedly processes the image synthesis of saturated and unsaturated regions of the first to Nth captured images until the image processing device 105 finishes, and proceeds to step S310 when the processing is complete.

[0043] (Step S310) In step S310, the image processing device 105 corrects the "saturated region composite image" obtained in step S306 to generate a "corrected saturated region composite image". The method of this correction is not limited as long as it does not cause hue shift or brightness reduction in the saturated region (see Figure 5(e)). For example, in the processing of step S306, the image processing device 105 performs a "simple addition" of each saturated region. Subsequently, in step S307, the image processing device 105 refers to the contents of the updated map and corrects by taking the average value of each saturated region.

[0044] Furthermore, the method of averaging each saturated region also includes, for example, when averaging is performed on four images with split exposures and there are only three images in the "unsaturated region," the method of multiplying the unsaturated region by the gain, multiplying by "4 / 3," and dividing by "4" for the whole. In this way, the image processing device 105 refers to the contents of the stored map and corrects the composite image of the unsaturated region in which a predetermined number of images have not been composited.

[0045] (Step S311) In step S311, the image processing device 105 corrects the "non-saturated region composite image" obtained in step S308 to generate a "corrected non-saturated region composite image". Normally, when non-saturated regions are simply averaged, the brightness at the location of the non-saturated region decreases, as shown in Figure 5(b). This is because, by combining saturated and non-saturated regions, the number of composite images of the non-saturated region becomes less than the number of images of the saturated region.

[0046] As an example, the image processing device 105 performs additive averaging of the unsaturated regions extracted in steps S304 and S308. Then, in step S311, the image processing device 105 refers to the "saturated region count map" and increases the number of images contributing to the unsaturated region composite image by the number of unsaturated regions that are lacking. Another example is that the image processing device 105 first performs only simple additive averaging of the unsaturated regions in step S308. Subsequently, in step S311, the image processing device 105 refers to the map and generates an average image of the unsaturated regions based on the number of images in the unsaturated regions. This prevents hue shift and brightness reduction caused by the number of images combined in the unsaturated region synthesis (see Figure 5(e)).

[0047] (Step S312: Step S313) Next, in step S312, the image processing device 105 combines the "corrected saturated region composite image" obtained in step S310 with the "corrected unsaturated region composite image" obtained in step S311 (see Figure 5(c)). Then, in step S313, the image processing device 105 develops the composite image obtained in step S312.

[0048] As shown in the second embodiment, by combining a saturated region composite image and a non-saturated region composite image, and correcting these composite images, it is possible to suppress overexposure and other issues similar to those caused by a physical ND filter, and to prevent hue shift and brightness reduction due to split exposure.

[0049] Furthermore, the image processing device 105 may be configured to perform either the correction in step S310 or the correction in step S311. Also, as described above, the image processing device 105 may be configured to correct the composite image of unsaturated regions in which a predetermined number of composite images have not been performed in the unsaturated region composite image by referring to the stored contents of the map.

[0050] <Addendum> This embodiment includes the following configurations, methods, and programs. (Configuration 1) An image processing apparatus that performs image processing on multiple images acquired by segmented exposure, Extraction means for extracting a predetermined region from each of the aforementioned plurality of images, The system includes a synthesis means for generating a composite image using the extracted region, The extraction means is Saturated and unsaturated regions are extracted from each of the aforementioned multiple images. The aforementioned synthesis means is Using the saturated region, a composite image of the saturated region is generated, and using the unsaturated region, a composite image of the unsaturated region is generated. An image processing apparatus characterized by combining a composite image of the saturated region and a composite image of the unsaturated region. (Configuration 2) The synthesis means is The image processing apparatus according to configuration 1, characterized in that it generates a composite image of the saturated regions by performing additive averaging or comparative brightness blending on the saturated regions extracted in each of the plurality of images. (Configuration 3) The synthesis means is The image processing apparatus according to configuration 1 or 2, characterized in that it generates a composite image of the unsaturated regions by summing and averaging the unsaturated regions extracted in each of the plurality of images. (Configuration 4) The image processing apparatus according to Configuration 1, further comprising correction means for performing correction on the composite image of the saturated region and / or the composite image of the unsaturated region. (Configuration 5) The configuration further comprises a map generation means that generates a map that stores information indicating whether a region is saturated or unsaturated, and the number of images required to generate the saturated region in association with that information, The correction means is The image processing apparatus according to configuration 4, characterized in that correction is performed on the composite image of the saturated region and the composite image of the unsaturated region by referring to the map. (Configuration 6) The correction means is The image processing apparatus according to claim 5, characterized in that, by referring to the map, it corrects the composite image of the unsaturated region in which a predetermined number of images have not been composited. (Configuration 7) imaging means, An imaging apparatus characterized by comprising the image processing apparatus described in configuration 1 or 2. (Method) A method for controlling an image processing device, An extraction step in which a predetermined region is extracted from each of the multiple images acquired by segmented exposure, The process includes a synthesis step of generating a composite image using the extracted region, The extraction step is, Saturated and unsaturated regions are extracted from each of the aforementioned multiple images. The aforementioned synthesis step is Using the saturated region, a composite image of the saturated region is generated, and using the unsaturated region, a composite image of the unsaturated region is generated. A control method for an image processing apparatus, characterized by combining a composite image of the saturated region with a composite image of the unsaturated region. (Program) A program that causes a computer to function as each of the means of the image processing apparatus described in any one of claims 1 to 7.

[0051] Although preferred embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications and changes are possible within the scope of its gist. The present invention can also be realized by supplying a program that implements one or more of the functions of this embodiment to a system or device via a network or storage medium, and having one or more general-purpose processors (ASICs) of the computer of that system or device read and execute the program. Furthermore, the present invention can also be realized by a dedicated processor (e.g., ASIC, FPGA, etc.) that implements one or more functions. Moreover, the present invention can also be realized by a combination of a general-purpose processor and a dedicated processor. Here, "processor" refers to a processor in a broad sense and includes both general-purpose processors and dedicated processors. Furthermore, the process of realizing the present invention may be executed by only one processor, or it may be executed by the cooperation of multiple processors located in physically separate locations. [Explanation of Symbols]

[0052] 100 Imaging device 101 Optical system 102 Image sensor 103 CPU 104 Primary storage 105 Image Processing Device 106 Recording media 107 Secondary storage 108 Display section 109 Operation section

Claims

1. An image processing apparatus that performs image processing on multiple images acquired by segmented exposure, Extraction means for extracting a predetermined region from each of the aforementioned plurality of images, The system includes a synthesis means for generating a composite image using the extracted region, The extraction means is Saturated and unsaturated regions are extracted from each of the aforementioned multiple images. The aforementioned synthesis means is Using the saturated region, a composite image of the saturated region is generated, and using the unsaturated region, a composite image of the unsaturated region is generated. An image processing apparatus characterized by combining a composite image of the saturated region and a composite image of the unsaturated region.

2. The aforementioned synthesis means is The image processing apparatus according to claim 1, characterized in that it generates a composite image of the saturated regions by additive averaging or comparative brightness blending of the saturated regions extracted in each of the plurality of images.

3. The aforementioned synthesis means is The image processing apparatus according to claim 1 or 2, characterized in that it generates a composite image of the unsaturated regions by summing and averaging the unsaturated regions extracted in each of the plurality of images.

4. The image processing apparatus according to claim 1, further comprising correction means for performing correction on the composite image of the saturated region and / or the composite image of the unsaturated region.

5. The system further comprises a map generation means that generates a map that stores information indicating whether a region is saturated or unsaturated, and the number of images required to generate the saturated region, in association with this map. The correction means is The image processing apparatus according to claim 4, characterized in that correction is performed on the composite image of the saturated region and the composite image of the unsaturated region by referring to the map.

6. The correction means is The image processing apparatus according to claim 5, characterized in that, by referring to the map, it corrects the composite image of the unsaturated region in which a predetermined number of images have not been composited.

7. Imaging means, An imaging apparatus comprising the image processing apparatus described in claim 1 or 2.

8. A method for controlling an image processing device, An extraction step in which a predetermined region is extracted from each of the multiple images acquired by segmented exposure, The process includes a synthesis step of generating a composite image using the extracted region, The extraction step is, Saturated and unsaturated regions are extracted from each of the aforementioned multiple images. The aforementioned synthesis step is Using the saturated region, a composite image of the saturated region is generated, and using the unsaturated region, a composite image of the unsaturated region is generated. A control method for an image processing apparatus, characterized by combining a composite image of the saturated region with a composite image of the unsaturated region.

9. A program that causes a computer to function as one of the means of an image processing apparatus according to any one of claims 1 to 7.

Images