Image processing apparatus, control method thereof, and storage medium

Abstract

The present invention provides an image processing apparatus, a control method thereof, and a storage medium. The image processing apparatus acquires an image captured during imaging, and sets a white balance recording mode for recording the image as a still image and a white balance display mode for displaying an image before being captured as a still image on a display. The apparatus controls white balance correction performed on the image captured during imaging. When a first display mode included in the display mode is set, the apparatus sets an amount of a light source color remaining after white balance correction of the image before being captured as a still image in the first display mode to be greater than an amount of the light source color remaining after white balance correction of the still image in the recording mode.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, a control method thereof, and a storage medium. Background Art

[0002] Conventionally, a mirrorless camera including an electronic viewfinder (EVF) instead of an optical viewfinder has been known. With this camera, the same image processing parameters as those applied to the recorded still image are applied to the image displayed on the EVF, so that the photographer can confirm how the completed still image will look before taking the image.

[0003] On the other hand, when the same image processing parameters as those applied to the recorded still image are applied to the image displayed on the EVF, the tone may be different from the tone that the photographer actually sees with the naked eye in the shooting environment. Therefore, a photographer who is accustomed to shooting images with a camera having an optical viewfinder may feel uncomfortable when viewing the image displayed on the EVF. Specifically, in the case of shooting video using the automatic white balance function, by correcting the slight red color of the bulb light source and the slight blue color of the day shade, etc., so that the atmosphere of the light source color does not remain compared to the atmosphere seen with the naked eye, and then recording the image as a still image, the recorded image may look more favorable. Therefore, when the white balance correction value applied to the recorded still image is applied to the image displayed on the EVF, a significant difference may occur between how the image displayed on the EVF looks and how the subject looks with the naked eye.

[0004] Japanese Unexamined Patent Application Publication No. 2002-218495 is known as a document that proposes a technique for recording a still image using the automatic white balance function. Specifically, Japanese Unexamined Patent Application Publication No. 2002-218495 proposes a technique in which, when a photographer records a still image using the automatic white balance function, the photographer manually sets the degree of correction in the automatic white balance applied to the still image and sets the degree of the atmosphere of the light source color to a desired degree.

[0005] In the case where the technique disclosed in Japanese Unexamined Patent Application Publication No. 2002-218495 is used together with the display on the EVF to make the tone of the image displayed on the EVF closer to the tone seen with the naked eye (the degree of the atmosphere of the light source color increases), not only in the displayed image but also in the recorded still image, the light source color may remain considerably. That is, a technique is desired that reduces the tone of the light source in the recorded still image and also controls the tone of the light source in the image displayed on the EVF. Summary of the Invention

[0006] The present invention is made in view of the above problems and implements the following technology. With this technology, even when the hue of the light source in the recorded still image is reduced, the hue of the light source in the image displayed on the EVF can be made close to the hue as seen with the naked eye.

[0007] To solve the above problems, one aspect of the present invention provides an image processing apparatus, including: one or more processors; and a memory that stores instructions which, when executed by the one or more processors, cause the image processing apparatus to function as: an acquisition unit configured to acquire an image captured by an imaging unit; a setting unit configured to be able to set a white balance recording mode and a white balance display mode, the white balance recording mode being applied to record the image as a still image, the white balance display mode being applied to display an image before being captured as the still image on a display unit; and a control unit configured to control white balance correction performed on the image captured by the imaging unit, wherein, when the first display mode included in the display mode is set, the control unit sets the amount of the light source color remaining after white balance correction of the image before being captured as the still image in the first display mode to be greater than the amount of the light source color remaining after white balance correction of the still image in the recording mode.

[0008] Another aspect of the present invention provides a control method for an image processing apparatus, including: acquiring an image captured by an imaging unit; setting a white balance recording mode and a white balance display mode, the white balance recording mode being applied to record the image as a still image, the white balance display mode being applied to display an image before being captured as the still image on a display unit; and controlling white balance correction performed on the image captured by the imaging unit, wherein, in the control, when the first display mode included in the display mode is set, the amount of the light source color remaining after white balance correction of the image before being captured as the still image in the first display mode is set to be greater than the amount of the light source color remaining after white balance correction of the still image in the recording mode.

[0009] Another aspect of the present invention provides a computer-readable storage medium including instructions for performing a control method of an image processing device, the control method including: obtaining an image captured by an imaging unit; setting a white balance recording mode and a white balance display mode, the white balance recording mode being applied to record the image as a still image, the white balance display mode being applied to display an image before being captured as the still image on a display unit; and controlling white balance correction performed on the image captured by the imaging unit, wherein, in the control, when a first display mode included in the display mode is set, an amount of a light source color remaining after performing white balance correction on the image before being captured as the still image in the first display mode is set to be greater than an amount of the light source color remaining after performing white balance correction on the still image in the recording mode.

[0010] According to the present invention, even when the hue of a light source in a recorded still image is reduced, the hue of the light source in the image displayed on the EVF can be made close to the hue as seen with the naked eye.

[0011] Other features of the present invention will become apparent from the following description of exemplary embodiments (with reference to the accompanying drawings). BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram showing an example of a functional structure of a digital camera as an example of an image processing device according to a first embodiment.

[0013] Figure 2 is a block diagram showing an example of a functional structure of a white balance control unit according to a first embodiment.

[0014] Figure 3 is a flowchart showing a series of operations of imaging processing according to a first embodiment.

[0015] Figure 4 is a flowchart showing a series of operations of white balance correction value calculation processing according to a first embodiment.

[0016] Figure 5 is a diagram showing a white detection range according to a first embodiment.

[0017] Figures 6A to 6D is a diagram showing a limiter control block according to a first embodiment.

[0018] Figures 7A to 7D is a diagram showing control of a limiter control block according to a first embodiment.

[0019] Figure 8 is a diagram showing a relationship between an amount of a remaining light source color and a combination of white balance modes according to a first embodiment.

[0020] Figure 9 It is a block diagram showing an example of the functional structure of a white balance control unit according to a second embodiment.

[0021] Figure 10 It is a flowchart showing a series of operations of a white balance correction value calculation process according to a second embodiment. Detailed Description of the Invention

[0022] Embodiments will be described in detail below with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. In the embodiments, multiple features are described, but the invention is not limited to requiring all of these features, and multiple of these features can be appropriately combined. Further, in the drawings, the same or similar structures are given the same reference numerals, and their repeated description is omitted.

[0023] An example of a digital camera that can capture still images will be described below as an example of an image processing device. However, the present embodiment is not limited to a digital camera that can capture still images, and is also applicable to a digital camera that can capture moving images and any electronic device such as a mobile phone and a computer that can capture images.

[0024] First Embodiment

[0025] Structure of Digital Camera

[0026] Figure 1 It is a block diagram showing an example of the functional structure of a digital camera as an example of an image processing device according to the present embodiment. Figure 1 One or more of the functional blocks shown therein can be implemented using hardware such as an ASIC or a programmable logic array (PLA), or by a programmable processor such as a CPU or a GPU that executes software. Alternatively, Figure 1 One or more of the functional blocks shown therein can be implemented by a combination of software and hardware. Even in the case where processing operations are described as being performed by different functional blocks, the processing operations described below can be mainly performed by the same hardware.

[0027] The optical system 101 includes, for example, a lens, a shutter, and an aperture, and exposes the imaging element 102 with an optical image of a subject. The optical system 101 is configured to be able to communicate with a central processing unit (hereinafter simply referred to as "CPU") 103, and send information related to the focal length, shutter speed, aperture value, etc. to the CPU 103.

[0028] The imaging element 102 can be, for example, a CCD image sensor or a CMOS image sensor, etc., and serves as an imaging unit. The imaging element 102 includes, for example, an RGB Bayer array, etc., and converts the optical image formed by the optical system 101 into luminance information pixel by pixel. The luminance information is digitized by an AD converter (not shown) and stored as raw data in the main storage device 104 before image display processing. The electrical gain (hereinafter simply referred to as "ISO sensitivity") of the imaging element 102 is set by the CPU 103.

[0029] The photometric device 105 divides the raw data into a plurality of photometric regions and detects the subject luminance of each region from the amount of light incident through the optical system 101. The photometric device 105 sends data related to the detected subject luminance to the CPU 103.

[0030] The CPU 103 serves as the control unit of the digital camera 100. The CPU 103 controls the constituent units of the digital camera 100 based on the signals input from other structural elements and the programs pre-stored in the auxiliary storage device 108, etc., to implement various functions of the digital camera 100.

[0031] The main storage device 104 is, for example, a volatile storage medium such as a RAM, and is used by the CPU 103 as a working memory. In addition, the information stored in the main storage device 104 can be used by the image processing device 106, or can be recorded in the storage medium 107.

[0032] The auxiliary storage device 108 is, for example, a non-volatile storage device such as an EEPROM, and stores programs (firmware or applications) for controlling the digital camera 100 and various types of setting information. The programs stored in the auxiliary storage device 108 are read and executed by the CPU 103. In addition, the setting information stored in the auxiliary storage device 108 is also read and used by the CPU 103. Various setting values for image processing described later used in this embodiment are also stored in the auxiliary storage device 108.

[0033] The storage medium 107 is, for example, a non-volatile storage medium such as a semiconductor memory, and records image data, etc., obtained by imaging and stored in the main storage device 104. The storage medium 107 can be configured to be attachable to and detachable from the digital camera 100, such as a semiconductor memory card. In this case, the data recorded in the storage medium 107 can be read by other devices such as a personal computer. As described above, the digital camera 100 can have at least one of an attachment / detachment mechanism for the storage medium 107 and a read / write function.

[0034] The display unit 109 serves as a display unit and displays a viewfinder image for a captured image, the captured image, a GUI image for interactive operations, and the like. The display unit 109 may include a plurality of display panels. For example, the viewfinder image and the captured image may be displayed on different display panels.

[0035] The operation unit 110 is a set of input devices that receive user operations and send input information to the CPU 103, and may include, for example, buttons, a lever, a touch panel, and input devices using voice and line of sight. The operation unit 110 also includes a release button for starting imaging. The digital camera 100 of the present embodiment has a plurality of image processing modes applied to the captured image by the image processing device 106, and any one of these modes can be set as the imaging mode from the operation unit 110.

[0036] The image processing device 106 performs image processing, also referred to as "imaging processing", on the raw data before the imaging process, such as white balance processing, color interpolation processing for converting an RGB Bayer array signal into an RGB three-plane signal, gamma correction processing, chromaticity correction, and hue correction. The white balance calculation performed in the present embodiment is also performed by the image processing device 106. At least a part of the functions of the image processing device 106 can be implemented by the CPU 103 that executes a program.

[0037] Structure of the image processing device

[0038] Next, an example of the structure of the white balance control unit 200 that performs white balance processing will be described with reference to Figure 2 the following. Figure 2 The white balance control unit 200 shown is included in, for example, the image processing device 106. The digital camera 100 of the present embodiment realizes an automatic white balance function of automatically calculating the white balance according to the light source in the shooting scene by including the white balance control unit 200.

[0039] The white balance control unit 200 may be a circuit or a software module that performs automatic white balance calculation. The white balance control unit 200 includes a block division unit 201, a white judgment unit 202, a white balance correction value limit unit 203, and a white balance correction value calculation unit 204. Details of the processing performed in each block will be described later.

[0040] The digital camera 100 of this embodiment has the following two display modes as operation modes for displaying an image before being captured as a still image as a viewfinder image on the display unit 109. One display mode is a mode for performing exposure control and controlling image processing operations such as white balance, gamma correction, and color correction processing (referred to as "still image recording equivalent display mode"). In this mode, the viewfinder image has the same degree of completion as the captured still image. Another display mode is a mode for performing exposure control and controlling image processing operations such as white balance, gamma correction, and color correction processing (referred to as "naked-eye viewing equivalent display mode"). In this mode, the viewfinder image has a degree of completion different from that of the captured still image and is close to what it looks like to the naked eye. Different automatic white balance calculations are performed in these two display modes. Details of these two display modes will be described later.

[0041] In addition to the above two display modes, the digital camera 100 also has two still image recording modes as automatic white balance modes applied to images recorded as still images. One still image recording mode is a mode for performing control to reduce the remaining degree of the atmosphere of the light source color (also referred to as "the remaining amount of the light source color") in the automatic white balance applied to the still image (referred to as "white priority mode"). Another still image recording mode is a mode in which the remaining amount of the light source color applied to the still image is set to be larger than that in the white priority mode (referred to as "atmosphere priority mode").

[0042] When the still image recording equivalent display mode is set, the image processing device 106 performs the same automatic white balance calculation on the image displayed on the display unit 109 as the recording mode (white priority mode or atmosphere priority mode) that has been set. As a result, in the still image recording equivalent display mode, the digital camera 100 can output an image (viewfinder image) having the same hue as the hue of the still image output to the display unit 109 in the white priority mode or the atmosphere priority mode.

[0043] On the other hand, when the naked-eye viewing equivalent display mode is set, the image processing device 106 performs an automatic white balance calculation independent of the recording mode (white priority mode or atmosphere priority mode) that has been set. For example, the image processing device 106 performs an automatic white balance calculation for making the remaining amount of the light source color more than that in the atmosphere priority mode on the image displayed on the display unit 109 regardless of the recording mode. As a result, the digital camera 100 can output an image whose hue is close to the hue of the ambient light source color observed by the user (or in other words, the photographer) with the naked eye to the display unit 109.

[0044] A series of operations for image recording by shooting

[0045] Next, with reference to Figure 3 a series of operations for image recording by imaging will be described. The CPU 103 deploys the program stored in the auxiliary storage device 108 to the main storage device 104, executes the program, and operates the imaging element 102, the image processing device 106, etc. (including internal structural elements) to implement the processing operations.

[0046] In S301, the CPU 103 receives a user input related to imaging from the operation unit 110. The user input includes, for example, inputs related to various settings of the digital camera 100, such as the focal length of the optical system 101, the shutter speed, the aperture value, the ISO sensitivity, the exposure correction amount setting, manual exposure, and automatic exposure. In addition, the display mode (still image recording equivalent display mode or naked eye observation equivalent display mode) and the recording mode (white priority mode or atmosphere priority mode) are also set.

[0047] In S302, the CPU 103 performs photometry control for imaging. For example, when the imaging settings are set to the automatic exposure mode in S301, the CPU 103 divides the image output from the imaging element 102 into n×m regions, where n represents the number of regions in the horizontal direction, and m represents the number of regions in the vertical direction. n and m can be any integers. In this embodiment, for example, both n and m are 16. The CPU 103 divides the image into a total of 256 regions and obtains RGB signals from each divided region. Then, the CPU 103 calculates the Y value based on Equation 1 according to the RGB signals from the regions, and further calculates the average value of the Y values of the regions to calculate the brightness of the subject.

[0048] Y = 3×R + 6×G + B... Equation 1

[0049] By calculating the brightness of the subject, the CPU 103 can control the ISO sensitivity, the shutter speed, and the aperture value based on the calculated brightness of the subject so that the image displayed on the display unit 109 has the brightness specified by the user.

[0050] In S303, the CPU 103 adjusts the settings for imaging. For example, the CPU 103 adjusts settings such as the focal length of the optical system 101, the shutter speed, the aperture value, and the ISO sensitivity of the imaging element 102 based on the setting information received in S301 and the brightness of the subject calculated in S302. When the manual exposure mode is set in S301, the exposure is controlled based on the ISO sensitivity, shutter speed, and aperture value specified by the user. However, when the display mode is set to the naked-eye observation equivalent display mode, the CPU 103 operates such that even if the manual exposure mode has been set, the image displayed on the display unit 109 is in the automatic exposure mode. At this time, for the exposure during still image recording in S310 described later, the CPU 103 performs imaging based on the settings specified by the user as the manual exposure settings.

[0051] In S304, the CPU 103 controls the imaging element 102 used for imaging. For example, the CPU 103 also adjusts the settings (such as ISO sensitivity) of the imaging element 102 based on the imaging settings information adjusted in S303 and performs imaging processing. In S305, the CPU 103 performs focusing control. For example, the CPU 103 performs automatic focusing control based on the captured image to focus at the focal position with the highest contrast. Known methods can be used to perform automatic focusing control. The above order of performing S302 to S305 is merely an example, and thus this embodiment is not limited to this order. The order of performing S302 to S305 can be changed according to the processing.

[0052] In S306, the CPU 103 uses the image processing device 106 to calculate the white balance correction value of the captured image. Details of the white balance correction value calculation will be given later after describing the flow of a series of imaging operations performed until S314.

[0053] In S307, the CPU 103 performs imaging processing. For example, the CPU 103 controls the image processing device 106 to perform imaging processing operations such as white balance correction processing using the white balance correction value calculated in S306, color interpolation processing, gamma correction processing, chromaticity correction processing, and hue correction processing. Then, in S308, the CPU 103 displays the image imaged in S307 on the display unit 109.

[0054] In S309, the CPU 103 determines whether the operation unit 110 has received a user instruction to start still image shooting. For example, in a case where it is determined based on the user input received by the operation unit 110 that an instruction to start still image shooting has not been received, the CPU 103 returns the process to S302 and repeats the operations from S302 to S308. On the other hand, in a case where it is determined based on the user input received by the operation unit 110 that an instruction to start still image shooting has been received, the CPU 103 advances the process to S310 and shifts to still image shooting control.

[0055] In S310, the CPU 103 shoots a still image based on settings such as ISO sensitivity, shutter speed, and aperture value used for still image shooting. In a case where the manual exposure mode is set, the CPU 103 may set the ISO sensitivity, shutter speed, and aperture value to the same set values calculated in S302. Additionally, in a case where the automatic exposure mode is set, the CPU 103 may set the ISO sensitivity, etc. to the same set values calculated in S302, or may separately calculate the settings used for still image shooting during the photometry control in S302 and use the calculated set values. In S311, the CPU 103 stores the image shot in S310 in the main storage device 104.

[0056] In S312, the CPU 103 calculates a white balance correction value for the image stored in the main storage device 104 in S311. In the white balance correction value calculation performed here, a white balance correction value corresponding to either the white priority mode or the atmosphere priority mode of the already set recording mode is calculated. Details thereof will be described later.

[0057] In S313, the CPU 103 performs imaging processing. For example, the CPU 103 controls the image processing device 106 to perform imaging processing operations such as white balance correction processing using the white balance correction value calculated in S312, color interpolation processing, gamma correction processing, chromaticity correction processing, and hue correction processing. Then, in S314, the CPU 103 records the image imaged in S313 in the storage medium 107. Thereafter, the CPU 103 ends this series of operations.

[0058] White Balance Correction Value Calculation Processing

[0059] Next, a detailed description will be given with reference to Figure 4 the above white balance correction value calculation processing (in S306 and S312). This processing is implemented by the CPU 103 executing a program stored in the auxiliary storage device 108 or by the CPU 103 controlling the image processing device 106.

[0060] In S401, the CPU 103 uses the block division unit 201 of the image processing device 106 to divide the captured image into block regions. For example, the CPU 103 uses the block division unit 201 to divide the captured image into n×m regions, where n represents the number of regions in the horizontal direction, and m represents the number of regions in the vertical direction. n and m can be any integers. In this embodiment, for example, both n and m are 16. That is, the block division unit 201 divides the image into a total of 256 regions. In the following description, the divided regions will be referred to as "block division regions".

[0061] In S402, the CPU 103 obtains Rave, Gave, and Bave representing the average values of the R, G, and B signals in each block division region based on Equation 2 below.

[0062] Rave = Rall / Rcount

[0063] Gave = Gall / Gcount…Equation 2

[0064] Bave = Ball / Bcount

[0065] Here, Rall, Gall, and Ball represent the integrated values of the R, G, and B signals in each block division region. Rcount, Gcount, and Bcount respectively represent the number of R signals, the number of G signals, and the number of B signals in each block division region.

[0066] In S403, the CPU 103 calculates the R and B signals (R / G value and B / G value) normalized by the G signal in each block division region based on Equation 3.

[0067] R / G = Rave / Gave

[0068] B / G = Bave / Gave…Equation 3

[0069] In S404, the CPU 103 performs a white determination process using the white determination unit 202 of the image processing device 106. Specifically, the white determination unit 202 extracts the blocks plotted within the white detection range 907 for the R / G and B / G values in each block region. Here, the white detection range 907 will be described in detail with reference to Figure 5 shown. Here, reference will be made to Figure 5 for a detailed description of the white detection range 907.

[0070] Figure 5 The reference numeral 501 shown in represents the plotted position of a light source on the blackbody radiation locus (also referred to as the "blackbody emission locus") in a graph where the x-axis represents R / G and the y-axis represents B / G. In addition, the reference numerals 502 to 507 schematically representFigure 5 Examples of the plotted positions of various types of light sources in the coordinate system shown. Reference numeral 502 denotes an example of the plotted position of sunlight (about 5200 Kelvin (hereinafter simply denoted by K)), reference numeral 503 denotes an example of the plotted position of shaded sunlight (about 7000 K), reference numeral 504 denotes an example of the plotted position of a bulb-colored light source (about 2800 K), reference numeral 505 denotes an example of the plotted position of a white fluorescent lamp, reference numeral 506 denotes an example of the plotted position of a daylight white fluorescent lamp, and reference numeral 507 denotes an example of the plotted position of a daylight-colored fluorescent lamp. The white detection range 907 is set to include light sources on the blackbody radiation locus and fluorescent light sources denoted by reference numerals 502 to 507. As a result of setting the white detection range 907 in the above-described manner, by using a known automatic white balance method, pixels in the image area that may have a light source color can be extracted, and the white balance can be corrected by changing the pixel values of the extracted pixels to achromatic colors.

[0071] Next, the white determination unit 202 calculates the average values RaveAll, GaveAll, and BaveAll of the average values Rave, Gave, and Bave of the R, G, and B signals in each block plotted in the white detection range 907 based on Equation 4.

[0072] RaveALL = (Integral value of Rave of the extraction blocks) / (Number of extraction blocks)

[0073] GaveALL = (Integral value of Gave of the extraction blocks) / (Number of extraction blocks)

[0074] BaveALL = (Integral value of Bave of the extraction blocks) / (Number of extraction blocks) … Equation 4

[0075] If there are no plotted blocks within the white detection range 907, the white determination unit 202 uses the average values Rave, Gave, and Bave of the R, G, and B signals in all the blocks to calculate the average values RaveAll, GaveAll, and BaveALL. Then, the white determination unit 202 calculates R / Gall and B / Gall, which are the R and B signals normalized by the G signal, based on Equation 5.

[0076] R / Gall = RaveALL / GaveALL

[0077] B / Gall = BaveALL / GaveALL … Equation 5

[0078] Through the processing up to here, a pair of the R signal and the B signal normalized by the G signal (or in other words, a point in the plane based on B / G and R / G) is calculated. The white determination unit 202 ends the white determination process in S402.

[0079] In this embodiment, as described above, an example of a method for extracting pixels that may have a light source color from an image has been described. However, the method for determining the values corresponding to R / Gall and B / Gall is not limited to this, and any other method may be used. For example, a method of directly obtaining the values corresponding to R / Gall and B / Gall from an image by using deep learning, which is an example of machine learning, may be used. For example, an image and the values of R / Gall and B / Gall corresponding to the image can be used as teacher data to train a deep neural network. Then, by inputting the image into the trained deep neural network, estimated values of R / Gall and B / Gall can be obtained.

[0080] In S405, the CPU 103 performs a limiter process using the white balance correction value limiter unit 203 of the image processing device 106. The limiter process in S405 is a process for limiting the correction amount of the white balance correction value so as to retain the light source color as white balance for the R / Gall and B / Gall pairs calculated in S404. The limiter process will be described in detail below with reference to Figures 6A to 6D Describe the limiter process in detail.

[0081] Figure 6A An example of the limiter control box 608 used in the limiter process is shown. Similar to Figure 5 the reference numeral 501 shown, Figure 6A the reference numeral 601 shown in Figure 5 represents the plotting position of a light source on the blackbody radiation locus in a graph where the x-axis represents R / G and the y-axis represents B / G. In addition, the reference numerals 602 to 607 represent the plotting positions of the light sources and fluorescent light sources (in order, sunlight, shaded sunlight, incandescent light source, white fluorescent lamp, daylight white fluorescent lamp, and daylight fluorescent lamp) on the blackbody radiation locus represented by the reference numerals 502 to 507 in

[0082] When the R / Gall and B / Gall calculated in S404 are plotted outside the limiter control box 608, the R / Gall and B / Gall are moved to the position on the closest limiter control box. Then, R / Gall_Limit and B / Gall_Limit are calculated, which will be the R / Gall and B / Gall after the limiter process. In Figure 6AIn the example shown, reference numerals 603 to 607 are plotted outside the limiter control box 608. Therefore, when R / Gall and B / Gall calculated in S404 are plotted at the same position as any of the plotting positions 603 to 607, R / Gall_Limit and B / Gall_Limit are calculated. On the other hand, reference numeral 602 is plotted inside the limiter control box 608. Therefore, when R / Gall and B / Gall are plotted at the same position as reference numeral 602, the above-mentioned movement process is not required.

[0083] When the limiter process ends, the CPU 103 uses the white balance correction value calculation unit 204 to calculate an R gain, a G gain, and a B gain representing the final white balance correction values (gains applied to the R, G, and B signals) based on Equation 6. As a result, through white balance correction, for the light source 602, the light source color does not remain, and for each of the light sources 603 to 607, the light source color remains.

[0084] Rgain = 1 / (R / Gall_Limit)

[0085] Ggain = 1 … Equation 6

[0086] Bgain = 1 / (B / Gall_Limit)

[0087] Here, the process for retaining the light source color for the automatic white balance mode in the display mode (still image recording equivalent display mode or naked-eye observation equivalent display mode) and the recording mode (white priority mode or atmosphere priority mode) according to the present embodiment will be described. The limiter control box 608 is set such that the size of the area varies according to the luminance. That is, the limiter control box 608 is set to have a smaller area when the luminance is high, and is set to have a larger area when the luminance is low. The luminance used here is the luminance of the subject calculated in the photometric control of S302. The reason why the limiter control box is set to have a smaller area when the luminance is high is as follows. When the luminance is high, the light source is likely to be outdoor sunlight. Therefore, it is not necessary to perform white balance correction on light sources other than sunlight. In addition, it is possible to reduce the incorrect correction and transition correction of white balance caused by the color of the subject being erroneously detected as the light source color in the above white determination process.

[0088] Given Figure 6A The limiter control box 608 shown as an example of the limiter control box when the luminance is high in the white priority mode, while Figure 6BAn example of the limiter control box 609 when the luminance is low in the white priority mode is shown. When the luminance is low, the light sources 602 to 607 are set to be plotted within the limiter control box 609. As a result, for each of the light sources 602 to 607, the light source color does not remain after white balance correction. The limiter control box when the luminance is high can be common in the automatic white balance mode in the display modes (the still image recording equivalent display mode and the naked-eye viewing equivalent display mode) and the recording modes (the white priority mode and the atmosphere priority mode). On the other hand, when the luminance is low, the limiter control box is changed in each mode. That is, in the present embodiment, when the luminance is greater than or equal to the luminance threshold, the color temperature of the limiter control box (or in other words, the amount of the light source color remaining after white balance correction) is set to be the same between the display modes. On the other hand, when the luminance is lower than the luminance threshold, the color temperature of the limiter control box (the amount of the light source color remaining after white balance correction) changes between the modes. At this time, the luminance threshold is set to, for example, the luminance when the color temperature is 5000K in the following Figures 7A to 7D shown below.

[0089] Figure 7A An example of controlling the low color temperature side of the limiter control box according to the luminance in the white priority mode of the recording mode is shown. In Figures 7A to 7D below, the x-axis represents the luminance, and the y-axis represents the plotted position of the color temperature on the low color temperature side of the limiter control box. Figure 7A An example is shown in which when the luminance of the subject is high, the color temperature of the limiter control box is set to 5000K, and as the luminance decreases, the color temperature gradually decreases to 2800K. Figure 7A The 5000K shown Figure 6A below is the color temperature corresponding to the low color temperature side of the limiter control box 608 when the luminance is high. In addition, Figure 7A the 2800K shown Figure 6B below is the color temperature corresponding to the low color temperature side of the limiter control box 609 when the luminance is low. The high color temperature side of the limiter control box and the fluorescent side (the origin side with respect to the blackbody radiation locus) can also be controlled such that, like the control of the low color temperature side of the limiter control box, the color temperature gradually changes according to the luminance.

[0090] Figure 6C An example of the limiter control box 610 when the luminance is low in the atmosphere priority mode of the recording mode is shown. When the luminance is low in the atmosphere priority mode, the limiter control box 610 is set such that the light sources 603 to 607 are located at positions slightly outside the limiter control box 610. As a result, for each of the light sources 603 to 607, a small amount of the light source color remains after white balance correction.

[0091] Figure 7BThis is a diagram showing the control of the low color temperature side of the limiter control box according to the brightness in the atmosphere priority mode. Figure 7B It shows an example where, when the brightness is high, the color temperature is set to 5000K, and as the brightness decreases, the color temperature gradually decreases to 3200K. Figure 7B The 5000K shown is Figure 6A the color temperature corresponding to the low color temperature side of the limiter control box 608 when the brightness is high. Additionally, Figure 7B the 3200K shown is Figure 6C the color temperature corresponding to the low color temperature side of the limiter control box 610 when the brightness is low.

[0092] As described above, in the case of selecting the still image recording equivalent display mode of the display mode, the CPU 103 sets the limiter control box corresponding to either the white priority mode or the atmosphere priority mode of the already set recording mode, and calculates the white balance correction value. That is, in the case where the still image recording equivalent display mode is set as the display mode, the amount of the light source color remaining after white balance correction of the image is set to be equal to the amount of the light source color remaining after white balance correction of the still image in the recording mode.

[0093] On the other hand, in the case of selecting the naked eye viewing equivalent display mode of the display mode, the CPU 103 sets a limiter control box different from the limiter control box set in the recording mode (white priority mode and atmosphere priority mode).

[0094] Figure 6D It shows the limiter control box 611 when the brightness is low in the naked eye viewing equivalent display mode. When the brightness is low in the naked eye viewing equivalent display mode, the limiter control box 611 is set to have a smaller area than the limiter control box 610 set when the brightness is low in the atmosphere priority mode. As a result, the light sources 603 to 607 are significantly spaced apart from the limiter control box, and for each of the light sources 603 to 607, a large amount of the light source color remains after white balance correction. As described above, Figure 6A the shown limiter control box 608 is the limiter control box when the brightness is high, so in the naked eye viewing equivalent display mode, the limiter control box when the brightness is high is common in the two recording modes.

[0095] Figure 7C It shows the control of the low color temperature side of the limiter control box according to the brightness in the naked eye viewing equivalent display mode. Figure 7C It shows an example where, when the brightness is high, the color temperature is set to 5000K, and as the brightness decreases, the color temperature gradually decreases to, for example, 4500K. Figure 7C The 5000K shown is Figure 6AThe color temperature corresponding to the low color temperature side of the limiter control box 608 when the brightness is high, as shown. Additionally, Figure 7C The 4500K shown is Figure 6D The color temperature corresponding to the low color temperature side of the limiter control box 611 when the brightness is low, as shown. As described above, in the case where the equivalent display mode for naked-eye viewing is set as the display mode, the amount of the light source color remaining after white balance correction of the image is set to be greater than the amount of the light source color remaining after white balance correction of the still image in the recording mode.

[0096] By controlling the limiter control box as described above, more light source color remains in the equivalent display mode for naked-eye viewing. Therefore, compared with the display in the equivalent display mode for still image recording, a display closer to how it looks to the naked eye can be achieved.

[0097] The additional processing performed in the equivalent display mode for naked-eye viewing will be described below. By setting the ISO sensitivity high and the shutter speed slow when shooting an image, the brightness of the image displayed on the viewfinder can be set to be higher than the brightness when viewed with the naked eye. Therefore, the user can identify dark subjects that cannot be recognized with the naked eye by viewing the viewfinder. However, in conditions where the display on the viewfinder is brighter than the display when viewed with the naked eye (such as a night scene), when performing limiter control in the equivalent display mode for naked-eye viewing such that the light source color remains, there is a problem that the light source color appears to remain more than when viewed with the naked eye.

[0098] In view of this, in the present embodiment, the above problem is solved by performing a process for enlarging the limiter control range (limiter control range enlargement process) under the condition that the brightness of the subject on the viewfinder appears higher than the brightness when viewed with the naked eye. Figure 7D An example of the color temperature change control corresponding to the low color temperature side of the limiter control box in the equivalent display mode for naked-eye viewing in the case where the limiter control range enlargement process is added is shown. The brightness of the subject corresponding to the point indicated by the reference numeral 701 shown in Figure 7D is the brightness at which the image displayed on the viewfinder starts to be brighter than when viewed with the naked eye. The shutter speed when shooting the image displayed on the viewfinder cannot be set slower than the frame rate of the image displayed on the viewfinder. For this reason, when the frame rate changes, the brightness of the subject for which the image displayed on the viewfinder can be set to be brighter than when viewed with the naked eye (or in other words, the point 701) also changes. Similarly, when the maximum value of the ISO sensitivity of the image displayed on the viewfinder changes, the brightness of the subject for which the image displayed on the viewfinder can be set to be brighter than when viewed with the naked eye also changes.

[0099] In this case, in the present embodiment, a predetermined brightness threshold (the point 701 of the brightness of the subject) for changing the color temperature of the limiter control frame changes according to the settings of the digital camera such as the ISO sensitivity of the image displayed on the viewfinder. Then, the CPU 103 compares the brightness of the subject included in the image with the threshold value. In the above manner, by changing the point of the brightness of the subject according to the frame rate of the viewfinder and the maximum ISO sensitivity of the image displayed on the viewfinder, etc., the limiter control frame can be appropriately changed regardless of conditions such as the frame rate and the maximum ISO sensitivity during imaging.

[0100] In Figure 7D , the dashed line 702 shows an example where the frame rate of the viewfinder is lower than the point 701, or the maximum ISO sensitivity of the image displayed on the viewfinder is higher than the point 701. Figure 7D Only two patterns including the point 701 and the dashed line 702 are shown, but a structure that divides the pattern more finely according to conditions such as the frame rate or the maximum ISO sensitivity can be used. In addition, in the following description, an example of the color temperature change corresponding to the low color temperature side of the limiter control frame is described. However, on the other hand, when the brightness of the subject on the viewfinder looks higher than the brightness when viewed with the naked eye, control can be performed to enlarge the limiter control frame.

[0101] In short, the relationship of the remaining amount of the light source color explained so far can be shown in Figure 8 . Figure 8 Shows the combination of the white balance mode for still image recording and the white balance mode for viewfinder display in the case of performing the limiter process in the automatic white balance mode.

[0102] The combinations shown in Figure 8 will be described in order starting from the top row. First, when the still image recording mode is set to the "atmosphere priority mode" and the viewfinder display mode is set to the "naked eye observation equivalent display mode", the light source color remains more in the viewfinder display mode than in the still image recording mode. That is, in the case of setting the naked eye observation equivalent display mode (the first display mode), the amount of the light source color remaining after the white balance correction in the naked eye observation equivalent display mode is greater than the amount of the light source color remaining after the white balance correction in the recording mode.

[0103] As a second combination, when the still image recording mode is set to the "atmosphere priority mode" and the viewfinder display mode is set to the "still image recording equivalent display mode", the amount of the light source color remaining after white balance correction is the same in the viewfinder display mode and the still image recording mode. That is, when the still image recording equivalent display mode (the second display mode) is set, the amount of the light source color remaining after white balance correction in the still image recording equivalent display mode is equal to the amount of the light source color remaining after white balance correction in the recording mode.

[0104] As a third combination, when the still image recording mode is set to the "white priority mode" and the viewfinder display mode is set to the "naked eye observation equivalent display mode", more light source color remains in the viewfinder display mode than in the still image recording mode. In addition, as a fourth combination, when the still image recording mode is set to the "white priority mode" and the viewfinder display mode is set to the "still image recording equivalent display mode", the remaining amount of the light source color is the same in the viewfinder display mode and the still image recording mode.

[0105] Among the above four combinations, the remaining amount of the light source color in the "atmosphere priority mode" of the recording mode is greater than the remaining amount of the light source color in the "white priority mode" of the recording mode. For this reason, the amount of the light source color remaining after white balance correction when setting the combination of the naked eye observation equivalent display mode and the atmosphere priority mode is greater than the amount of the light source color remaining after white balance correction when setting the combination of the naked eye observation equivalent display mode and the white priority mode. In addition, the amount of the light source color remaining after white balance correction when setting the combination of the still image recording equivalent display mode and the atmosphere priority mode is greater than the amount of the light source color remaining after white balance correction when setting the combination of the still image recording equivalent display mode and the white priority mode.

[0106] Through the above processing, the white balance correction value calculation process in S306 ends. The white balance correction value calculation process for still images performed in S312 is the same as the control process performed in the white priority mode or the atmosphere priority mode of the recording mode, so repeated description will be omitted.

[0107] As described above, this embodiment is configured to be able to set a white balance recording mode that is applied to record an image as a still image and a white balance display mode that is applied to display an image before being shot as a still image on a display unit. In the case of setting the "naked-eye observation equivalent display mode" of the display mode, the amount of the light source color remaining after white balance correction in the image before being shot as a still image is set to be greater than the amount of the light source color remaining after white balance correction in the still image in the recording mode. In this way, the image displayed on the EVF can be made closer to the color observed with the naked eye, while keeping the color of the recorded still image as an appropriate color so that the hue of the light source does not remain too much. In other words, even when the hue of the light source in the recorded still image is reduced, the hue of the light source in the image displayed on the EVF can be made close to the hue as seen with the naked eye.

[0108] Second Embodiment

[0109] Next, the second embodiment will be described. The first embodiment is configured such that: a white balance correction value is calculated based on pixel information related to some divided blocks of the captured image included in the white judgment region, and the white balance correction value is corrected according to the display mode. On the other hand, the second embodiment is configured such that a table for arranging white balance correction values according to the light source is used to correct the white balance correction value. The structural elements of the image processing apparatus according to this embodiment and the white balance correction value calculation process are different from those in the first embodiment, but other structural elements and processing operations are the same as or substantially the same as those in the first embodiment. For this reason, the same or substantially the same structural elements and processing operations are given the same reference numerals, and their descriptions will be omitted, and only the differences will be mainly described.

[0110] Structure of Image Processing Apparatus

[0111] Figure 9 An example of the structure of the white balance control unit 900 according to the second embodiment is shown. The white balance control unit 900 is included in the image processing apparatus 106 and performs white balance processing. The white balance control unit 900 can be a circuit or software module that performs automatic white balance calculation. The white balance control unit 900 includes a block division unit 901, a light source determination unit 902, and a white balance correction value calculation unit 903. The processing performed in each block will be described later.

[0112] A Series of Operations for Image Recording Using Shooting

[0113] Next, with reference to Figure 3 and 10To illustrate a series of operations using image recording by a camera. The program stored in the auxiliary storage device 108 is deployed to the main storage device 104 by the CPU 103 and the program is executed, and operations of the imaging element 102, the image processing device 106, etc. (including internal structural elements) are performed to achieve the processing operations.

[0114] The CPU 103 performs the Figure 3 processing operations of S301 to S314 shown in the same manner as in the first embodiment. In this embodiment, the white balance correction value calculation processing performed in S306 and S312 is different from that in the first embodiment. Therefore, reference will be made to Figure 10 to illustrate this processing. In the second embodiment, the auxiliary storage device 108 stores the white balance correction values according to various light sources in the form of a table. In this table, Rgain, Ggain, and Bgain representing the white balance correction values (gains applied to the R, G, and B signals) corresponding to the white priority mode, the atmosphere priority mode, and the equivalent display mode for naked-eye observation are stored according to the light source.

[0115] In addition, for the equivalent display mode for naked-eye observation of the display mode, two types of white balance correction values are stored: the white balance correction value used when the brightness of the image displayed on the viewfinder is higher than the brightness of the image observed with the naked eye, and the white balance correction value used when the brightness of the image displayed on the viewfinder is the same as the brightness of the image observed with the naked eye. At this time, the white balance correction value of the light source is set so that the residual amount of the light source color increases in the following order: the equivalent display mode for naked-eye observation, the atmosphere priority mode, and the white priority mode. Also, in the equivalent display mode for naked-eye observation, the white balance correction value used when the brightness of the image displayed on the viewfinder is the same as the brightness of the image observed with the naked eye is set so that more of the light source color remains compared to the white balance correction value used when the brightness of the image displayed on the viewfinder is higher than the brightness of the image observed with the naked eye.

[0116] In S1001, as in the first embodiment, the CPU 103 divides the captured image into n×m regions using the block division unit 901, where n represents the number of regions in the horizontal direction, and m represents the number of regions in the vertical direction. n and m can be any integers. In this embodiment, n and m are both 16, and the image is divided into a total of 256 regions.

[0117] In S1002, the CPU 103 performs a light source determination process using the light source determination unit 902. The light source determination process is performed using the color distribution of the block segmentation data created in S1001 and the information related to the brightness of the subject calculated in S302. The light source determination process is performed by selecting the light source with the closest characteristics from the color distributions of the block segmentation data corresponding to various light sources stored in the auxiliary storage device 108 and the information related to the brightness of the subject, based on the color distribution of the block segmentation data and the information related to the brightness of the subject. The light source determination process can be configured to perform rule-based processing or to determine the optimal light source for the color distribution of the block segmentation data and the brightness of the subject by using deep learning processing with a trained neural network.

[0118] In S1003, the CPU 103 uses the white balance correction value calculation unit 903 to calculate the white balance correction value of the light source detected in S1002, based on the table stored in the auxiliary storage device 108 and arranging the white balance correction values according to the light source.

[0119] In the case where the user selects the still image recording equivalent display mode of the display mode, the white balance correction value calculation unit 903 first determines whether the recording mode is set to the white priority mode or the atmosphere priority mode. Then, it looks up the white balance correction value corresponding to the determined recording mode from the table stored in the auxiliary storage device 108 and arranging the white balance correction values according to the light source.

[0120] On the other hand, in the case where the user selects the naked-eye observation equivalent display mode of the display mode, the white balance correction value calculation unit 903 first determines whether the brightness of the image displayed on the viewfinder is the same as the brightness of the image when observed with the naked eye, or whether the brightness of the image displayed on the viewfinder is higher than the brightness of the image when observed with the naked eye. Then, it looks up the white balance correction value corresponding to the determination result from the table stored in the auxiliary storage device 108 and arranging the white balance correction values according to the light source.

[0121] Also in the second embodiment, Figure 8 The remaining amount of the light source color in each combination of the white balance mode for still image recording and the white balance mode for viewfinder display shown is the same as in the first embodiment. When the CPU 103 finishes the process in S1003, the CPU 103 ends the white balance correction value calculation process in S306 and returns to the original process. The white balance correction value calculation process for still images performed in S312 is the same as the process performed when recording an image.

[0122] As described above, in the present embodiment, the white balance correction values for various light sources are stored in the auxiliary storage device 108 in the form of a table, and the white balance correction value corresponding to the light source determined for each block in the image is applied. In this way, as in the first embodiment, the image displayed on the EVF can be made closer to the color observed with the naked eye, while maintaining the color of the recorded still image as an appropriate color so that the color tone of the light source does not remain too much. That is, even when the color tone of the light source in the recorded still image is reduced, the color tone of the light source in the image displayed on the EVF can be made close to the color tone seen with the naked eye.

[0123] Other embodiments

[0124] Embodiments of the present invention can also be implemented by the following method, that is, by providing software (program) that executes the functions of the above-described embodiments to a system or device through a network or various storage media, and a method in which a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0125] Although the present invention has been described with reference to exemplary embodiments, it should be understood that the present invention is not limited to the disclosed exemplary embodiments. The scope of the following claims should be given the broadest interpretation to encompass all such modifications as well as equivalent structures and functions.

Claims

1. An image processing apparatus, comprising: one or more processors; and a memory that stores instructions which, when executed by the one or more processors, cause the image processing apparatus to function as: an acquisition unit configured to acquire an image captured by an imaging unit; a setting unit configured to be able to set a white balance recording mode and a white balance display mode, the white balance recording mode being applied to record the image as a still image, and the white balance display mode being applied to display an image before it is captured as the still image on a display unit; and a control unit configured to control white balance correction of an image captured by the imaging unit, wherein, when a first display mode included in the white balance display mode is set, the control unit sets an amount of a light source color remaining after white balance correction of the image before it is captured as the still image in the first display mode to be greater than an amount of the light source color remaining after white balance correction of the still image in the white balance recording mode.

2. The image processing apparatus according to claim 1, wherein, the white balance display mode further includes a second display mode, and when the second display mode of the white balance display mode is set, the control unit sets an amount of a light source color remaining after white balance correction of the image before it is captured as the still image in the second display mode to be equal to an amount of the light source color remaining after white balance correction of the still image in the white balance recording mode.

3. The image processing apparatus according to claim 1, wherein, in white balance correction applied when recording the still image, the white balance recording mode includes: a first recording mode in which an amount of a light source color in an image captured by the imaging unit is reduced; and a second recording mode in which an amount of the remaining light source color is set to be greater than the amount of the remaining light source color in the first recording mode.

4. The image processing apparatus according to claim 1, wherein, in white balance correction applied when recording the still image, the white balance recording mode includes: a first recording mode in which an amount of a light source color in an image captured by the imaging unit is reduced; and a second recording mode in which an amount of the remaining light source color is set to be greater than the amount of the remaining light source color in the first recording mode, and the control unit sets an amount of a light source color remaining after white balance correction of the image before it is captured as the still image when the first display mode and the second recording mode are set to be greater than an amount of the light source color remaining after white balance correction of the still image when the first display mode and the first recording mode are set.

5. The image processing apparatus according to claim 2, wherein, in white balance correction applied when recording the still image, the white balance recording mode includes: a first recording mode in which an amount of a light source color in an image captured by the imaging unit is reduced; and A second recording mode, in which the amount of the residual light source color is set to be greater than the amount of the residual light source color in the first recording mode, and the control unit sets the amount of the residual light source color after performing white balance correction on the image before being captured as the still image when the second display mode and the second recording mode are set to be greater than the amount of the residual light source color after performing white balance correction on the still image when the second display mode and the first recording mode are set.

6. The image processing apparatus according to claim 1, wherein, when the first display mode is set and the image before being captured as the still image has a luminance lower than a threshold value, the control unit sets the amount of the residual light source color after performing white balance correction on the image before being captured as the still image in the first display mode to be greater than the amount of the residual light source color after performing white balance correction on the still image in the white balance recording mode.

7. The image processing apparatus according to claim 1, wherein, in the first display mode, when the image before being captured as the still image has a luminance lower than a predetermined luminance threshold value, the control unit sets the amount of the residual light source color after performing white balance correction on the image before being captured as the still image to be less than the amount of the residual light source color after performing white balance correction on the still image when the image has a luminance higher than the predetermined luminance threshold value.

8. The image processing apparatus according to claim 7, wherein, the control unit determines the luminance of the image before being captured as the still image by comparing the luminance of the subject included in the image before being captured as the still image with the predetermined luminance threshold value.

9. The image processing apparatus according to claim 7, wherein, the control unit changes the predetermined luminance threshold value according to the setting of the display unit or the imaging unit.

10. The image processing apparatus according to claim 9, wherein, the control unit changes the predetermined luminance threshold value according to at least one of the frame rate displayed on the display unit and the ISO sensitivity set for the imaging unit.

11. A control method for an image processing apparatus, comprising: obtaining an image captured by an imaging unit; setting a white balance recording mode and a white balance display mode, the white balance recording mode being applied to record the image as a still image, and the white balance display mode being applied to display the image before being captured as the still image on a display unit; and controlling white balance correction performed on the image captured by the imaging unit, wherein, in the control, when the first display mode included in the white balance display mode is set, the amount of the residual light source color after performing white balance correction on the image before being captured as the still image in the first display mode is set to be greater than the amount of the residual light source color after performing white balance correction on the still image in the white balance recording mode.

12. A computer-readable storage medium, which includes instructions for performing a control method of an image processing device, and the control method includes: obtaining an image captured by an imaging unit; setting a white balance recording mode and a white balance display mode, where the white balance recording mode is applied to record the image as a still image, and the white balance display mode is applied to display an image before being captured as the still image on a display unit; and controlling white balance correction for the image captured by the imaging unit, wherein, in the control, when a first display mode included in the white balance display mode is set, an amount of a light source color remaining after white balance correction of the image before being captured as the still image in the first display mode is set to be greater than an amount of the light source color remaining after white balance correction of the still image in the white balance recording mode.

Images