Image processing apparatus and its control method, imaging apparatus and program

Abstract

This reduces tone jumps in composite images obtained by combining multiple images. [Solution] The imaging device 10 includes an image processing unit 104 that generates a composite image of multiple still images obtained by continuous shooting by combining RAW data, and a control unit 108 that determines the amount of noise to be added to the composite image based on the shooting conditions of the multiple still images or information extracted from the composite image. When generating the composite image, the image processing unit 104 adds the amount of noise determined by the control unit 108 to the composite image.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, a control method thereof, an imaging apparatus, and a program, and particularly relates to an image synthesis technique.

Background Art

[0002] When photographing a high-brightness subject with an imaging apparatus such as a digital camera, a method of suppressing the occurrence of whiteout and achieving proper exposure by reducing the amount of light incident on the imaging element using an ND filter is widely used.

[0003] However, there is a problem that the attachment / detachment operation of the ND filter is troublesome. Further, when the ND filter is built into the imaging apparatus, insertion and retraction into the incident optical path of the ND filter can be easily performed, but problems such as an increase in the size, weight, and cost of the imaging apparatus occur.

[0004] Therefore, a technique for suppressing whiteout in the same manner as when photographing using a physical ND filter by synthesizing a plurality of images captured by split exposure without attaching a physical ND filter is disclosed (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] When performing addition-average synthesis of a plurality of images captured by split exposure as in the technique disclosed in Patent Document 1 above, the noise of the synthesized image may be reduced, resulting in a more prominent tone jump.

[0007] This invention has been made in view of these circumstances, and aims to provide an image processing apparatus that can reduce tone jumps in a composite image obtained by combining multiple images. [Means for solving the problem]

[0008] The present invention relates to an image processing apparatus comprising: an acquisition means for acquiring RAW data of a plurality of still images; a synthesis means for generating a composite image of the plurality of still images by synthesizing the RAW data; an addition means for adding noise to the composite image; and a determination means for determining the amount of noise to add to the composite image based on the shooting conditions of the plurality of still images or information extracted from the composite image. [Effects of the Invention]

[0009] According to the present invention, tone jumps in a composite image obtained by combining multiple images can be reduced. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram showing the schematic configuration of the imaging device according to this embodiment. [Figure 2] This is a flowchart explaining the operation of an imaging device in ND composite mode. [Figure 3] This is an example of a definition table for the amount of noise applied to a composite image. [Modes for carrying out the invention]

[0011] Embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Here, we will focus on an imaging device equipped with the functions of an image processing apparatus according to the present invention.

[0012] Figure 1 is a block diagram showing the schematic configuration of an imaging device 10 according to an embodiment. The imaging device 10 includes a lens 100, a mechanical shutter 101, an image sensor 102, an A / D conversion unit 103, an image processing unit 104, a display unit 105, an operation unit 106, a recording unit 107, a control unit 108, and a memory 109.

[0013] The control unit 108 is a microcomputer composed of a processor (CPU) for executing programs, RAM, ROM, etc. The control unit 108 may be a system-on-a-chip (SoC) or a system-on-a-package (SiP), etc.

[0014] The control unit 108 comprehensively controls the operation of each part of the imaging device 10 by having the CPU load programs stored in ROM into RAM and execute them, as well as controlling the operation of external devices (not shown) connected to the imaging device 10. In other words, the control unit 108 can perform each function of the imaging device 10 by executing a predetermined program. Functions that can be realized by the control unit 108 executing a program may also be implemented by hardware such as an ASIC or FPGA. The ROM provided in the control unit 108 is electrically rewritable and stores settings values ​​and GUI data for the imaging device 10, in addition to the programs executed by the CPU.

[0015] Although only one lens 100 is shown in Figure 1, it is actually composed of multiple lenses that form an optical image of the subject on the imaging surface of the image sensor 102. In this embodiment, the lens 100 includes an aperture (not shown). The image sensor 102 is, for example, a CMOS sensor, and it generates an analog electrical signal as a pixel signal having a value corresponding to the amount of charge generated during the charge accumulation period from the optical image formed on the imaging surface. The image sensor 102 has a light-shielding region.

[0016] The mechanical shutter 101 controls the exposure time (charge accumulation time) for the image sensor 102 by opening and closing during still image shooting, and is kept open during video recording. Note that if the image sensor 102 has a global shutter mechanism, the mechanical shutter 101 may not be necessary.

[0017] When a charge accumulation period is completed in the image sensor 102, one frame's worth of pixel signals (analog pixel signals) are read out from the image sensor 102, and the read-out analog pixel signals are input to the A / D conversion unit 103. The A / D conversion unit 103 performs A / D conversion on the analog pixel signals to generate digital pixel signals and transmits them to the memory 109 and the image processing unit 104. Note that the A / D conversion unit 103 is unnecessary if the image sensor 102 is configured to output digital pixel signals.

[0018] The digital pixel signal output from the A / D conversion unit 103 has only one color component corresponding to the color of the filter (unit filter) provided on the corresponding pixel (generally one of red (R), green (G), or blue (B)). In this embodiment, a digital image signal consisting of such a digital pixel signal having only one color component corresponding to the color of the unit filter is referred to as "RAW data".

[0019] The RAW data output from the A / D conversion unit 103 is temporarily stored in the memory 109. In addition to the RAW data, the memory 109 temporarily stores image data and other data generated by the image processing unit 104 after it has undergone a predetermined development process on the RAW data.

[0020] The image processing unit 104 performs development processing on the RAW data acquired from the A / D conversion unit 103 to generate image data. The development processing is a general term for a plurality of image processes such as color interpolation processing and gradation correction processing (gamma processing). Note that the color interpolation processing is a process of interpolating values of color components that cannot be obtained during shooting, and is also called demosaicing processing. By applying the color interpolation processing, the digital signal for each pixel constituting the image data comes to have a plurality of color components (for example, RGB or YCbCr) necessary for a color image.

[0021] The image processing unit 104 also performs detection processing of feature regions such as a face region and a human body region, detection processing of the movement thereof, person recognition processing, detection processing of various objects, image synthesis processing, scaling processing, encoding / decoding processing, etc. on the image data. The image processing unit 104 further performs various image processes such as data processing such as header information generation processing, generation of signals and evaluation values used for autofocus detection (AF), calculation of evaluation values used for automatic exposure control (AE), etc. Note that the image processes executable by the image processing unit 104 are not limited to these.

[0022] The display unit 105 is, for example, a touch panel type display, and displays a live view image, a playback image, a GUI, setting values and information of the imaging device 10, etc. The operation unit 106 is composed of an input device (switch, key, button, dial, touch panel, etc.) for the user to give various instructions to the imaging device 10 (control unit 108). Also, since the display unit 105 is composed of a touch panel, it functions as one of the operation units 106. The recording unit 107 is a recording medium such as a memory card that records RAW data, image data, synthesized RAW data, information associated with these data, audio data, etc. according to the shooting mode and set recording conditions. Note that the synthesized RAW data is generated by synthesizing a plurality of RAW data in the image processing unit 104.

[0023] In the imaging device 10 configured as described above, it is possible to set and execute the ND synthesis mode (dimming synthesis mode). In the ND synthesis mode, the density of the ND is set, and split exposure is performed according to the density of the ND, and addition average synthesis is performed. By using the ND synthesis mode, it is possible to perform shooting that suppresses the occurrence of overexposure even in slow shutter shooting without attaching an ND filter, similar to the case of attaching an ND filter.

[0024] Before explaining the operation in the ND synthesis mode in the imaging device 10, the addition process, addition average process, and generation and addition process of noise components, which are particularly important in the image synthesis process executed in the ND synthesis mode, will be explained.

[0025] The image synthesis process is a process of synthesizing a plurality of still images to generate a synthesized image. Each of the plurality of still images is composed of a large number of pixels arranged two-dimensionally, and using the xy orthogonal coordinate system, the coordinates of the pixels are represented as '[x, y]'. Note that the coordinates of the pixels in the synthesized image are also represented as '[x, y]'. Let the number of still images to be synthesized be 'N (N: an integer of 2 or more)', and the luminance value of the pixel at the coordinates [x, y] in the still image be represented as 'I_i[x, y] (i = 1 to N)', and the luminance value of the pixel at the coordinates [x, y] in the synthesized image be represented as 'I[x, y]'.

[0026] The image processing unit 104 calculates the luminance value I[x, y] of each pixel of the synthesized image according to the synthesis processing method used.

[0027] In the addition process, the image processing unit 104 generates a synthesized image according to the following formula 1. In the addition process, the luminance value of the pixel at the same coordinates of each still image is added to obtain the luminance value of each pixel of the synthesized image. The addition process is used, for example, when synthesizing N images taken with an exposure amount of 1 / N of the appropriate exposure amount to generate an image with appropriate exposure.

[0028] In response to this, the image processing unit 104 generates a composite image using the following equation 2 in the averaging process. In the averaging process, the brightness value of each pixel in the composite image is obtained by adding the brightness values ​​of pixels at the same coordinates in each still image and then performing a gain reduction by dividing the brightness value obtained by the number of still images. The averaging process can, for example, combine multiple images with the same exposure and simulate overexposure without causing blown-out highlights. Increase the shutter speed This method is used to obtain an image. In additive averaging, the brightness can be adjusted for each region within the image by changing the gain reduction amount for each feature region and object region detected by the image processing unit 104.

[0029]

number

[0030] The image processing unit 104 can perform noise component generation and addition processing when performing image synthesis. The noise component generated here is, for example, Gaussian noise with a mean value of 0 (zero) and a standard deviation of σ, but is not limited to this, and may also be uniformly distributed noise. The amount of Gaussian noise can be changed by changing the standard deviation σ to an arbitrary value. By adding the noise component generated for each pixel to the input image, an image with added noise is obtained.

[0031] Next, we will explain the operation of the imaging device 10 in ND composite mode (light reduction composite mode). Figure 2 is a flowchart illustrating the operation of the imaging device 10 in ND composite mode (light reduction composite mode). Each process (step) indicated by the number S in the flowchart of Figure 2 is realized by the control unit 108 executing a predetermined program to comprehensively control the operation of each part of the imaging device 10.

[0032] In S201, the control unit 108 receives exposure condition settings from the user through the operation unit 106 and sets the ISO sensitivity, aperture value, and shutter speed.

[0033] In S202, the control unit 108 receives the ND concentration setting from the user via the operation unit 106 and sets the ND concentration. The configurable ND concentrations are ND2, ND4, ND8, ND16, ND32, etc., but even higher concentrations may be configurable.

[0034] In S203, the control unit 108 determines whether or not it has received a shooting instruction from the user through the operation unit 106. A shooting instruction is, for example, a full press of the shutter button included in the operation unit 106. If the control unit 108 determines that it has not received a shooting instruction (NO in S203), it repeats the determination process in S203. If it determines that it has received a shooting instruction (YES in S203), it executes the process in S204.

[0035] In S204, the control unit 108 performs continuous shooting. The exposure conditions and number of shots for continuous shooting are determined by the exposure conditions and ND density set in S201 and S202. Specifically, the shutter speed is determined by using the ND density value as the number of shots and dividing the set shutter speed equally by the number of shots. For example, if the user sets the ND density to ND2 and the shutter speed to 4 seconds, continuous shooting will be performed with 2 shots and a shutter speed of 2 seconds. The control unit 108 saves the RAW data obtained from the continuous shooting process in S204 to the memory 109.

[0036] In S205, the control unit 108 instructs the image processing unit 104 to perform the process of generating a composite image (composite RAW data) by combining all the continuously captured images (RAW data). At this time, the image processing unit 104 performs the synthesis process using an averaging process.

[0037] In S206, the control unit 108 determines the amount of noise to be added to the composite image. The amount of noise to be added is determined by changing the standard deviation σ of the Gaussian noise according to the ISO sensitivity set in S201 and the number of images used for image synthesis in S204 (i.e., the set value of the ND density).

[0038] Figure 3 shows an example of a definition table for the amount of noise set according to ISO sensitivity and the number of composite images (the number of consecutive images used to generate the composite image). The lower the ISO sensitivity and the higher the number of composite images, the larger the standard deviation σ is set to, resulting in a greater amount of added noise. Conversely, the higher the ISO sensitivity and the lower the number of composite images, the smaller the standard deviation σ is set to, resulting in a smaller amount of added noise. When the standard deviation σ = 0 (zero), no noise is added.

[0039] In this embodiment, the amount of noise added to the composite image is determined according to the combination of ISO sensitivity and the number of composite images, but it may also be determined based on either ISO sensitivity or the number of composite images. Furthermore, if there is little texture in the image, the areas where tone jumps are noticeable will increase, so the amount of texture in the image may be detected, and if there is a lot of texture, the amount of noise added may be set to zero (i.e., standard deviation σ=0). In this case, for example, edge extraction is performed on the composite image to generate an edge image, region division is performed on the edge image, the integral value of the edge amount for each region is calculated, and regions where the integral value of the edge amount is above a predetermined threshold are judged to have a lot of texture. Here, a general edge extraction filter such as a Sobel filter can be used for edge extraction. The ratio of the region where the integral value of the edge amount is above the threshold to the total number of divided regions is calculated, and if the obtained ratio is greater than a pre-set value, the amount of noise added is set to zero (i.e., no noise is added).

[0040] In S207, the control unit 108 adds the noise amount determined in S206 to the composite image generated in S205 by the image processing unit 104.

[0041] In S208, the control unit 108, using the image processing unit 104, develops the composite image to which noise was added in S207, and generates an image data file containing the developed image data.

[0042] In S209, the control unit 108 sends the image data file generated by the image processing unit 104 to the recording unit 107, records (stores) it in the recording unit 107, and terminates this process. As a result, the imaging device 10 transitions to the shooting standby state. Note that in S209, composite RAW data may be recorded in addition to or instead of the developed image data.

[0043] According to this embodiment, when generating a composite image by combining multiple images using additive averaging, the amount of noise added to the composite image is determined based on shooting conditions such as ISO sensitivity and / or the number of images to be combined, as well as the combining conditions. This makes it possible to effectively suppress the occurrence of tone jumps in the composite image.

[0044] Next, another example of a method for determining the amount of noise will be described. In the above embodiment, the amount of noise was determined based on the ISO sensitivity and / or the number of composite images, but instead of this method, the standard deviation λ of the signal value in the light-shielding region of the composite RAW data may be detected, and the amount of noise may be determined based on the standard deviation λ.

[0045] In this case, a threshold τ for determining whether or not noise needs to be added in comparison with the standard deviation λ is set in advance and stored in the control unit 108 (or its ROM, etc.). Then, the difference amount δ between the standard deviation λ and the threshold τ in the light-shielding region of the composite image is detected, and if the standard deviation λ is smaller than the threshold τ, the standard deviation σ of the added noise is changed according to the difference amount δ. At this time, the larger the difference amount δ, the larger the standard deviation σ of the added noise is made to increase the amount of added noise. On the other hand, if the standard deviation λ is greater than or equal to the threshold τ, the standard deviation σ of the added noise is set to zero so that no noise is added. This method also makes it possible to effectively suppress the occurrence of tone jumps in the composite image.

[0046] Next, another example of a method for generating a composite RAW image will be described. In the above embodiment, a composite RAW image was generated using additive averaging, but instead, additive averaging and additive averaging may be switched and applied depending on the region in the image, thereby achieving the effect of suppressing overexposure depending on the region in the image.

[0047] In this case, the user selects the area where they want to suppress overexposure using a boundary line, separating the areas where overexposure is suppressed from those where it is not. When using additive averaging, tone jumps become more noticeable due to the reduction in noise level, so additive averaging is performed in the area where overexposure is suppressed, along with a predetermined amount of noise. On the other hand, additive averaging is performed in areas other than those where overexposure is suppressed, but no noise is added. By doing so, tone jumps can be reduced in the areas where overexposure is suppressed without adding noise to the areas other than those where overexposure is suppressed.

[0048] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Furthermore, each of the embodiments described above is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0049] For example, in the above embodiment, the present invention was described as being embodied in a digital camera. However, the present invention does not necessarily have to be embodied in a device equipped with an imaging function, and can be applied to various electronic devices equipped with image processing means capable of performing synthesis processing of multiple images composed of digital data. For example, the present invention can be applied to various computers such as personal computers and tablet computers, and electronic devices such as smartphones.

[0050] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0051] This embodiment includes the following configurations, methods, and programs. (Configuration 1) An image processing apparatus comprising: acquisition means for acquiring RAW data of multiple still images; synthesis means for generating a composite image of the multiple still images by synthesizing the RAW data; addition means for adding noise to the composite image; and determination means for determining the amount of noise to add to the composite image based on the shooting conditions of the multiple still images or information extracted from the composite image. (Configuration 2) The image processing apparatus according to Configuration 1, wherein the shooting condition is an ISO sensitivity value, and the determination means increases the amount of noise as the ISO sensitivity value decreases. (Configuration 3) The image processing apparatus according to Configuration 1 or 2, wherein the plurality of still images are captured by continuous shooting, and the determination means increases the amount of noise as the number of plurality of still images increases. (Configuration 4) An image processing apparatus according to Configuration 1, comprising: a generation means for generating an edge image by edge extraction on the composite image; and a division means for dividing the edge image into regions, wherein the determination means determines the noise amount based on the edge amount for each region divided by the division means. (Configuration 5) An image processing apparatus according to Configuration 1, comprising: a calculation means for calculating the standard deviation of the signal value in the light-shielding region of the composite image; and a storage means for storing a threshold value for determining whether or not noise should be added to the composite image based on the value of the standard deviation, wherein the determination means increases the amount of noise as the difference between the standard deviation and the threshold increases when the standard deviation is smaller than the threshold, and sets the amount of noise to zero when the standard deviation is equal to or greater than the threshold. (Configuration 6) The image processing apparatus according to any one of Configurations 1 to 5, characterized in that the synthesis means synthesizes the plurality of still images by averaging. (Configuration 7) An image processing apparatus according to any one of Configurations 1 to 5, comprising a selection means for accepting a user's selection of a region to suppress overexposure, wherein the synthesis means applies an averaging process to the region and an addition process to the other regions, and the addition means adds a noise amount to the region to which the averaging process has been applied and does not add noise to the region to which the addition process has been applied. (Configuration 8) The image processing apparatus according to any one of Configurations 1 to 7, characterized in that the noise added to the composite image is Gaussian noise with a predetermined standard deviation. (Configuration 9) An imaging device characterized by comprising: an imaging means; a synthesis means for generating a composite image of a plurality of still images captured by the imaging means by synthesizing the RAW data of the plurality of still images; an addition means for adding noise to the composite image; and a determination means for determining the amount of noise to add to the composite image based on the shooting conditions of the plurality of still images or information extracted from the composite image. (Method 1) A method for controlling an image processing device, comprising the steps of: acquiring RAW data of a plurality of still images; generating a composite image of the plurality of still images by combining the RAW data of the plurality of still images; determining the amount of noise to be added to the composite image based on the shooting conditions of the plurality of still images or information extracted from the composite image; and adding the noise of the specified noise amount to the composite image. (Program 1) A program that causes a computer to function as one of the means of the image processing apparatus described in any one of configurations 1 to 8. [Explanation of Symbols]

[0052] 10 Imaging device 102 Image sensor 104 Image Processing Unit 108 Control Unit 109 memory

Claims

1. A means for acquiring RAW data of multiple still images, A synthesis means for generating a composite image of the plurality of still images by combining the RAW data, An additional means for adding noise to the aforementioned composite image, An image processing apparatus characterized by having a determination means for determining the amount of noise to be added to the composite image based on the shooting conditions of the plurality of still images or information extracted from the composite image.

2. The aforementioned shooting conditions are ISO sensitivity values, The image processing apparatus according to claim 1, characterized in that the determination means increases the amount of noise as the ISO sensitivity value decreases.

3. The aforementioned multiple still images were captured by continuous shooting. The image processing apparatus according to claim 1 or 2, characterized in that the determination means increases the amount of noise as the number of the plurality of still images increases.

4. A generation means for generating an edge image by extracting edges from the aforementioned composite image, The system includes a division means for performing region division of the edge image, The image processing apparatus according to claim 1, characterized in that the determination means determines the noise amount based on the edge amount for each region divided by the division means.

5. A calculation means for calculating the standard deviation of the signal value in the light-shielding region of the composite image, It has a storage means for storing a threshold value for determining whether or not noise should be added to the composite image based on the standard deviation value, The image processing apparatus according to claim 1, characterized in that the determination means increases the amount of noise as the difference between the standard deviation and the threshold increases when the standard deviation is smaller than the threshold, and sets the amount of noise to zero when the standard deviation is equal to or greater than the threshold.

6. The image processing apparatus according to claim 1, characterized in that the synthesis means synthesizes the plurality of still images by additive averaging.

7. It has a selection mechanism that accepts the user's selection of areas to suppress overexposure, The synthesis means applies an averaging process to the region and an addition process to the other regions. The image processing apparatus according to claim 1, characterized in that the additional means adds a noise amount to the region to which the additive averaging process is applied, and does not add noise to the region to which the additive process is applied.

8. The image processing apparatus according to claim 1, characterized in that the noise added to the composite image is Gaussian noise with a predetermined standard deviation.

9. Imaging means, A synthesis means that generates a composite image of multiple still images captured by the imaging means by combining the RAW data of the multiple still images, An additional means for adding noise to the aforementioned composite image, An imaging apparatus characterized by having a determination means for determining the amount of noise to be added to the composite image based on the shooting conditions of the plurality of still images or information extracted from the composite image.

10. A method for controlling an image processing device, Steps include acquiring RAW data of multiple still images, The steps include generating a composite image of the plurality of still images by combining the RAW data of the plurality of still images, The steps include determining the amount of noise to be added to the composite image based on the shooting conditions of the multiple still images or information extracted from the composite image, A control method for an image processing apparatus, characterized by comprising the step of adding the noise of the aforementioned noise amount to the composite image.

11. A program that causes a computer to function as each of the means of the image processing apparatus described in claim 1.

Images