Image processing apparatus and image processing method, and imaging apparatus

Abstract

To provide an image processing device capable of achieving stable and accurate automatic white balance adjustment for various types of images. [Solution] The image processing device estimates information about the ambient light source of the shooting scene of the image data based on the image data that has undergone white balance adjustment processing using a white balance correction value that is independent of the image data. Then, the image processing device calculates a white balance correction value to be used for automatic white balance adjustment processing of the image data based on the estimated information about the ambient light source and the image data.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and an imaging apparatus, and particularly relates to white balance adjustment technology.

Background Art

[0002] Image processing apparatuses such as imaging apparatuses use an automatic white balance adjustment function. The automatic white balance adjustment function is a function that estimates an achromatic region from an image, and adjusts the color tone of the entire image so that the RGB values of the estimated achromatic region become equal.

[0003] For the purpose of improving the accuracy of the automatic white balance adjustment function, the use of a machine learning model has been proposed. In Patent Document 1, a machine learning model is used to estimate the color temperature of a light source from a region of a specific color (for example, skin color) included in a region of a specific subject such as a person detected from an image.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the method described in Patent Document 1, when the detection accuracy of the region of the specific subject or the region of the specific color is low, or when the region cannot be detected, the accuracy of white balance adjustment decreases.

[0006] In one aspect, the present invention provides an image processing apparatus capable of realizing automatic white balance adjustment with stable accuracy for various images.

Means for Solving the Problems

[0007] In one aspect, the present invention provides an image processing apparatus comprising: a processing means for performing a white balance adjustment process on demosaiced image data using a white balance correction value independent of the image data; an estimation means for estimating information about the ambient light source of the shooting scene of the image data based on the image data that has undergone white balance adjustment processing by the processing means; and a calculation means for calculating a white balance correction value to be used for automatic white balance adjustment processing on demosaiced image data based on the information about the ambient light source estimated by the estimation means and the demosaiced image data. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an image processing device that can achieve stable and accurate automatic white balance adjustment for various images. [Brief explanation of the drawing]

[0009] [Figure 1] Block diagram showing an example of the functional configuration of an electronic camera, which is an example of an image processing apparatus according to an embodiment of the present invention. [Figure 2] Block diagram showing an example of the functional configuration of the image processing unit according to this embodiment. [Figure 3] Block diagram showing an example of the functional configuration of the DL unit according to this embodiment. [Figure 4] This is a diagram illustrating the neural network according to this embodiment. [Figure 5] Flowchart showing detailed processing of the WB control unit according to the first embodiment [Figure 6] Figure showing the white detection range according to this embodiment. [Figure 7] Flowchart showing detailed processing of the WB control unit according to the second embodiment [Modes for carrying out the invention]

[0010] The present invention will be described in detail below with reference to the attached drawings, based on exemplary embodiments thereof. Note that the following embodiments do not limit the invention to the claims. Furthermore, while multiple features are described in the embodiments, not all of them are essential to the invention, and the multiple features may be combined arbitrarily. In addition, in the attached drawings, the same or similar configurations are given the same reference numeral, and redundant descriptions are omitted.

[0011] In the following, an embodiment of the present invention using a digital camera as an example of an image processing device will be described. However, imaging functionality is not essential to the present invention, and it can be implemented in any electronic device having one or more arithmetic circuits or processors. Such electronic devices include video cameras, computer equipment (personal computers, tablet computers, media players, PDAs, etc.), smartphones, smartwatches, game consoles, robots, drones, and dashcams. These are examples, and the present invention can be implemented in other electronic devices as well.

[0012] ●(First Embodiment) Figure 1 is a block diagram showing an example of the functional configuration of an electronic camera 100 as an image processing device according to the embodiment. The lens group 101 is an optical system that forms an optical image of the subject on the imaging surface of the imaging unit 103. The lens group 101 includes fixed lenses and movable lenses. The movable lenses may include a focusing lens, a zoom lens, an image blur correction lens, and the like. The operation of the movable lenses is controlled by the system control unit 50.

[0013] The shutter 102 is a mechanical shutter and also functions as an aperture. A separate aperture may also be provided in addition to the shutter 102. The operation of the shutter 102 is controlled by the system control unit 50.

[0014] The imaging unit 103 includes an image sensor and peripheral circuits necessary for the operation of the image sensor (such as a clock signal supply circuit, scanning circuit, and readout circuit). The image sensor may be, for example, a known CCD or CMOS color image sensor having a primary color Bayer array color filter. The image sensor has a pixel array in which multiple pixels are arranged in two dimensions and peripheral circuits for reading signals from each pixel. Each pixel accumulates charge according to the amount of incident light by photoelectric conversion. By reading signals with a voltage corresponding to the amount of charge accumulated during the exposure period from each pixel, a group of pixel signals (analog image signals) representing the subject image formed on the imaging surface is obtained.

[0015] The A / D conversion unit 104 converts the analog image signal read from the imaging unit 103 into a digital image signal (image data). The image data is supplied to the image processing unit 105 or temporarily stored in the memory 106 via the memory control unit 107.

[0016] The image processing unit 105 applies predetermined image processing to image data output by the A / D conversion unit 104 and image data read from the recording medium 112, generating signals and image data according to the application, and acquiring and / or generating various types of information. The image processing unit 105 may be a dedicated hardware circuit, such as an ASIC (Application Specific Integrated Circuit) designed to implement a specific function. Alternatively, the image processing unit 105 may be configured such that a processor, such as a DSP (Digital Signal Processor) or GPU (Graphics Processing Unit), executes software to implement a specific function. The image processing unit 105 outputs the acquired or generated information and data to the system control unit 50, memory control unit 107, etc., according to the application.

[0017] The image processing applied by the image processing unit 105 to the image data may include, for example, preprocessing, color interpolation, correction, detection, data processing, evaluation value calculation, and special effects processing. The pre-processing may include signal amplification, reference level adjustment, defective pixel correction, and the like. The color interpolation processing is performed when an image sensor is provided with a color filter, and is processing for interpolating values of color components not included in individual pixel data constituting image data. The color interpolation processing is also referred to as demosaicing processing. The correction processing may include processes such as white balance adjustment, gradation correction, correction of image degradation caused by optical aberration of the lens group 101, correction of the influence of peripheral dimming of the lens group 101, and color correction. The detection processing may include detection of characteristic regions (for example, a face region and a human body region) and the motion thereof, person recognition processing, and the like. The data processing may include processes such as region clipping (trimming), composition, scaling, encoding and decoding, and header information generation (data file generation). Generation of display-use image data and recording-use image data is also included in the data processing. Note that the encoding and decoding may be performed by a codec unit 110. The evaluation value calculation processing may include processes such as generation of signals and evaluation values used for autofocus detection (AF), and generation of evaluation values used for automatic exposure control (AE). The special effect processing may include processes such as addition of a blur effect, color tone change, and relighting. Note that these are examples of processes applicable to an image processing unit 105, and do not limit the processes to be applied by the image processing unit 105.

[0018] Note that an electronic camera 100 is capable of capturing still images and moving images. Although handling of audio is not specifically described in the present specification, the electronic camera 100 also records audio in parallel with images when capturing a moving image, and generates moving image data with audio. Processing related to audio necessary for generating moving image data is executed by the image processing unit 105 or a system control unit 50.

[0019] Memory 106 is used as a buffer memory to temporarily store image data output by the A / D conversion unit 104, image data read from the recording medium 112, and recording image data. Memory 106 is also used as a work memory to temporarily store image data and intermediate data to which the image processing unit 105 applies image processing. Part of memory 106 is also used as video memory for the display unit 109. By writing display image data to the video memory area, an image based on the display image data is displayed on the display unit 109.

[0020] The memory control unit 107 controls the reading and writing of data to the memory 106. The D / A conversion unit 108 converts the display image data stored in the video memory area of ​​the memory 106 into an analog signal suitable for display on the display unit 109 and outputs it to the display unit 109.

[0021] The display unit 109 has a display device such as an LCD and displays captured images, images read from the recording medium 112, live view images, etc. The display unit 109 also displays information about the electronic camera 100, various settings, information about the currently displayed image, and user interfaces such as menu screens.

[0022] The codec unit 110 performs encoding and decoding of image data stored in the memory 106. The codec unit 110 conforms to known encoding schemes applicable to still image data and video data, such as JPEG and MPEG. The encoding and decoding may also be performed by the image processing unit 105.

[0023] The recording medium interface 111 mechanically and electrically connects a removable recording medium 112, such as a semiconductor memory card or a card-type hard disk, to the electronic camera 100. The face detection unit 113 analyzes the image data and detects the face region contained within the image represented by the image data. Face region detection may also be performed by the image processing unit 105. Furthermore, a configuration for detecting specific regions other than faces may be added.

[0024] The system control unit 50 is, for example, a processor (CPU, MPU, microprocessor, etc.) capable of executing programs. The system control unit 50 controls the operation of each part of the electronic camera 100 and realizes the functions of the electronic camera 100 by reading the program stored in the non-volatile memory 121 into the system memory 122 and executing it.

[0025] The non-volatile memory 121 is, for example, electrically rewritable. The non-volatile memory 121 stores programs executed by the system control unit 50, various settings for the electronic camera 100, GUI data, and the like. The system memory 122 is a volatile memory such as DRAM. The system memory 122 temporarily stores the program executed by the system control unit 50, as well as constants, variables, parameters, etc., that the system control unit 50 uses during program execution.

[0026] The operation unit 120 is a general term for input devices (buttons, switches, dials, etc.) provided for the user to input various instructions to the electronic camera 100. The input devices that make up the operation unit 120 have names according to the function they are assigned. For example, the operation unit 120 includes a release switch, a video recording switch, a shooting mode selection dial for selecting a shooting mode, a menu button, directional keys, and a select key. The release switch is a switch for recording still images, and the system control unit 50 recognizes the half-pressed state of the release switch as a shooting preparation instruction and the fully pressed state as a shooting start instruction. When a shooting preparation instruction is detected, the system control unit 50 executes still image shooting preparation operations such as AF operation and AE operation. When a shooting start instruction is detected, the system control unit 50 controls the operation of the shutter 102 according to the shooting conditions determined by the AE operation and exposes the image unit 103. The system control unit 50 then controls a series of operations from generating an image file containing the still image data for recording from the analog image signal obtained by shooting, until the image file is recorded on the recording medium 112.

[0027] Furthermore, the system control unit 50 recognizes the video recording switch as an instruction to start video recording when pressed in the shooting standby state, and as an instruction to stop recording when pressed during video recording. Upon detecting the instruction to start recording, the system control unit 50 performs AF and AE operations while shooting video at a predetermined frame rate. The system control unit 50 then controls a series of operations, from generating an image file containing the video data for recording from the analog image signal at the predetermined frame rate obtained by shooting, to recording the image file onto the recording medium 112.

[0028] Figure 2 is a diagram representing a series of image processing steps performed by the image processing unit 105 in this embodiment, using functional blocks. Note that the functional blocks are for convenience only, and the functions of multiple functional blocks may be performed by a single integrated circuit. Furthermore, one or more functional blocks may be performed by a processor executing a program. Also, Figure 2 only shows functional blocks related to the processing necessary for explaining this embodiment among the processing steps that the image processing unit 105 can perform.

[0029] The demosaicing processing unit 201 performs demosaicing (also called debayering or simulcasting) on ​​the image data output from the A / D conversion unit 104. Image data obtained using an image sensor equipped with a single-color filter for each pixel has only one color component for each individual pixel. This type of image data format is referred to here as the RAW format. Demosaicing is a process that adds missing color components to pixels in image data obtained using an image sensor equipped with a color filter. By applying demosaicing, the image data can be treated as color image data.

[0030] The demosaicing processing unit 201 outputs the processed image data to the DL (Deep Learning) unit 202 and the WB control unit 203.

[0031] Deep learning (DL) is a machine learning technique for neural networks. The DL unit 202 performs training of the neural network and inference processing using the trained neural network. As will be described in detail later, the DL unit 202 generates input image data from the image data output by the demosaicing unit 201, infers the ambient light source of the shooting scene represented by the input image data, and outputs light source estimation information. The light source estimation information can take any form that can identify the type or color temperature of the ambient light source. For example, the light source estimation information may be the probability for each type of light source.

[0032] The WB control unit 203 calculates a white balance (WB) correction value based on the image data output from the demosaicing processing unit 201 and the light source estimation information output from the DL unit 202. The WB control unit 203 then adjusts the white balance by applying the calculated WB correction value to each color component of the image data. The method for calculating the WB correction value will be described later. The WB control unit 203 outputs the image data after white balance adjustment to the color conversion matrix (MTX) circuit 204 and the Y generation circuit 211.

[0033] The color conversion matrix (MTX) circuit 204 multiplies the white balance-adjusted image data output by the WB control unit 203 by a color gain, converting the RGB image data into two color difference signal data, RY and BY. The LPF (low-pass filter) circuit 205 limits the bandwidth of the color difference signal data RY and BY. The CSUP (chroma suppress) circuit 206 suppresses false color signals in the saturated portion of the color difference signal data that has been bandwidth-limited by the LPF circuit 205.

[0034] The Y generation circuit 211 generates luminance signal data Y from the image data after white balance adjustment. The edge enhancement circuit 212 applies edge enhancement processing to the luminance signal data Y.

[0035] The RGB conversion circuit 207 converts YUV format image data, which consists of color difference signal data RY and BY output from the CSUP circuit 206 and luminance signal data Y output from the edge enhancement circuit 212, into RGB format image data.

[0036] The gamma (γ) correction circuit 208 performs gamma correction processing on the image data output by the RGB conversion circuit 207. Gamma correction processing is a gradation correction process according to predetermined γ characteristics. The gamma-corrected image data is output to the color-luminance conversion circuit 209.

[0037] The color-luminance conversion circuit 209 converts RGB image data into YUV image data for encoding in JPEG format. The JPEG encoding circuit 210 encodes the image data output by the color-luminance conversion circuit 209 and generates a JPEG image file. The data of the generated image file is stored in the memory 106 and then recorded on the recording medium 112. Note that the JPEG image file may be generated by the codec unit 110 instead of the JPEG encoding circuit 210 (image processing unit 105).

[0038] Although not shown in Figure 2, other correction processes such as defective pixel correction and correction of image degradation caused by optical aberrations and vignetting of the lens group 101 are actually applied by the image processing unit 105.

[0039] Figure 3 is a diagram that shows a series of image processing steps performed by the DL unit 202 in this embodiment, represented using functional blocks. 301 is RGB image data output by the demosaicing processing unit 201. The resizing processing unit 302 resizes the image data 301 to the resolution of the image to be input to the machine learning model (neural network) used by the DL processing unit 304. Typically, resizing is a reduction in size.

[0040] The development processing unit 303 performs development processing for light source estimation on the resized image data. The development processing performed here includes at least the white balance adjustment processing among the image processing performed downstream from the DL unit 202 in Figure 2.

[0041] The white balance adjustment process performed during the development process for light source estimation uses a fixed WB correction value that is independent of the image data (i.e., it is not an auto white balance adjustment process). The fixed WB correction value may be, for example, a WB correction value that assumes a specific ambient light source (e.g., sunlight (daylight)). Therefore, for images taken under ambient light sources with a different color temperature from a specific ambient light source, the influence of the ambient light source remains in the color tone of the image data after development processing.

[0042] For example, if a specific ambient light source is sunlight (daylight), images taken under a low color temperature light source such as an incandescent bulb will remain reddish overall after processing. Conversely, images taken under a high color temperature light source, such as in a cloudy or sunny shaded environment, will remain bluish overall after processing. In this way, by performing processing using a fixed white balance correction value, the influence of the light source color remains throughout the processed image, allowing for accurate estimation of the ambient light source color temperature from the image, regardless of the presence or absence of specific subjects or color regions. When a fixed white balance correction value is used assuming a specific ambient light source, images taken under that specific ambient light source will have an appropriate color tone. Therefore, if the color tone of the image is appropriate, it can be inferred that it was taken under that specific ambient light source.

[0043] In the learning phase, the DL processing unit 304 trains a neural network using input data and ground truth data, for example, through supervised learning. In the inference phase, the DL processing unit 304 provides input data to the trained neural network and obtains light source estimation information from the neural network. Note that the training may be performed on a device other than the electronic camera 100. Here, the DL processing unit 304 obtains light source estimation information using data representing the trained neural network that is pre-stored in the non-volatile memory 121.

[0044] Here, the neural network is trained to output at least one of the following as light source estimation information 305: light source color temperature, Δuv (color deviation), and light source type. The neural network may also be configured and trained to output a confidence score indicating the likelihood of the estimation result. The light source estimation information 305 is output to the WB control unit 203 and used for white balance adjustment processing.

[0045] Figure 4 is a schematic diagram of a multilayer perceptron (MLP) as an example of a neural network that can be used by the DL processing unit 304. 401 represents multiple neurons constituting the input layer, 403 represents multiple neurons constituting the hidden layer, and 404 represents a single neuron constituting the output layer. Multiple hidden layers may exist. Each neuron 403 constituting the hidden layer has a link 402 that receives a value obtained by multiplying the output of all neurons constituting the previous layer by a weight. Learning the neural network is the process of adjusting the weights of each link 402 so that the difference between the output of the output layer 404 and the ground truth data is smaller than a predetermined value.

[0046] Figure 2 shows a scenario where light source estimation is performed during still image capture (recording), but light source estimation may also be performed on the frame images of the video captured for live view display in the shooting standby state. In this case, the white balance adjustment process for the still image data is performed using the results of the light source estimation performed immediately before still image capture. Note that even when light source estimation is performed in parallel with live view display, the fixed WB correction value used in the development process for light source estimation remains the same. Since auto white balance adjustment processing is performed on the video generated for live view display, the input data used by the DL processing unit 304 may have a different color tone than the video displayed in live view.

[0047] Next, the details of the WB correction value calculation process in the WB control unit 203 will be explained using the flowchart shown in Figure 5. As mentioned above, the WB control unit 203 is a schematic representation of the functions of the image processing unit 105, so the processes described below as the operation of the WB control unit 203 are actually executed by the image processing unit 105.

[0048] In S501, the WB control unit 203 sets the color range to be detected as a white region in the image data supplied from the demosaicing processing unit 201, based on the light source estimation information obtained by the DL unit 202.

[0049] Figure 6(a) shows the range of the white region corresponding to the type of light source in the CxCy color space. The relationship between RGB values ​​and CxCy color space coordinates is as follows. Cx = (RB) / Y × 10²⁴ Cy = (R + B - 2G) / Y × 10²⁴ However, Y = (R + 2G + B) / 4

[0050] The negative x-coordinate (Cx) represents the color evaluation value for high color temperature subjects (e.g., a white subject photographed in the shade on a sunny day), while the positive x-coordinate represents the color evaluation value for low color temperature subjects (e.g., a white subject photographed under an incandescent light source). The difference between the y-coordinate (Cy) and the blackbody radiation locus corresponds to the color deviation (Δuv). In the CxCy color space, a negative Δuv (downward from the blackbody radiation locus) indicates a larger green component, while a positive Δuv (upward from the blackbody radiation locus) indicates a larger reddish-purple component. Therefore, the color evaluation values ​​for typical fluorescent lights fall within the negative Δuv range.

[0051] If the type of light source is output as light source estimation information 305, the WB control unit 203 sets the white detection range according to the type of light source shown in Figure 6(a). If the light source color temperature and Δuv are output, the WB control unit 203 identifies the type of light source from the relationship between the two and sets the white detection range according to the identified type of light source.

[0052] Furthermore, if the reliability of the light source estimation information is provided, the WB control unit 203 can change the size of the white detection range to be set according to the reliability.

[0053] For example, suppose the type of light source is estimated to be a sunny day in the shade. If the confidence level of the estimation is above a threshold (high), the WB control unit 203 sets a white detection range 601 that is approximately the same as the white range of a sunny day in the shade in Figure 6(a), as shown in Figure 6(b). On the other hand, if the confidence level of the estimation is below a threshold (low), the WB control unit 203 sets a wide white detection range 602 that corresponds to multiple types of ambient light sources so that the white range under other types of light sources can also be detected, as shown in Figure 6(c). The white detection range 602 can be determined from color evaluation values ​​calculated in advance by photographing a white subject under different light sources.

[0054] In S502, the WB control unit 203 divides the image data for one frame, output by the demosaicing processing unit 201 and stored in memory 106, into m (>1) blocks.

[0055] In S503, the WB control unit 203 calculates a color average value (R[i], G[i], B[i]) for each block by adding the values ​​of the pixels contained in the block together for each color component. Then, it substitutes the color components of the color average value into R, G, and B in the above formula to calculate a color evaluation value (Cx[i], Cy[i]) for each block. Note that i is the block number and is an integer from 1 to m.

[0056] In S504, the WB control unit 203 determines whether the color evaluation value (Cx[i], Cy[i]) of the i-th block calculated in S503 falls within the white detection range set in S501. If it is determined to fall within the range, it executes S505; otherwise, it executes S506.

[0057] In S505, the WB control unit 203 calculates the average color values ​​(R[i], G[i], B[i]) for each color component for blocks where the color evaluation values ​​(Cx[i], Cy[i]) are determined to be within the white detection range.

[0058] Furthermore, the processes in S504 and S505 can be expressed by the following formulas.

number

[0059] In S506, the WB control unit 203 determines whether or not the processing from S503 onwards has been performed for all blocks. If it determines that the processing has been performed, it executes S507. If it does not determine that the processing has been performed, it executes the processing from S503 onwards for one of the unprocessed blocks.

[0060] In S507, the WB control unit 203 calculates the white balance correction value WBCo(WBCo_R,WBCo_G,WBCo_B) from the integral value of the average color (SumR,SumG,SumB) according to the following formula, and then terminates the WB correction value calculation process. The white balance correction value WBCo is the white balance correction value used in the auto white balance adjustment process.

[0061] WBCo_R = SumY × 1024 / SumR WBCo_G = SumY × 1024 / SumG WBCo_B = SumY × 1024 / SumB However, SumY = (SumR + 2 × SumG + SumB) / 4

[0062] (modified version) Here, when the reliability of the light source estimation information is high, a relatively narrow white detection range is set according to the type of light source. However, the WB correction value may also be modified so that the color evaluation value of a pixel to which the WB correction value calculated by setting a wide white detection range, such as the white detection range 602 in Figure 6(c), is applied falls within the white detection range based on the light source estimation information (for example, the white detection range 601).

[0063] According to this embodiment, by performing light source estimation using an image whose white balance has been adjusted using a fixed white balance correction value, it becomes possible to perform light source estimation with stable accuracy, without depending on the area of ​​a specific subject or the area of ​​a specific color.

[0064] ●(Second Embodiment) Next, a second embodiment will be described. This embodiment is the same as the first embodiment, except for the WB correction value calculation process. Therefore, the WB correction value calculation process in this embodiment will be explained using the flowchart shown in Figure 7.

[0065] The processes in S501 to S507 are equivalent to those in the first embodiment, so their explanation is omitted. In S708, the WB control unit 203 retrieves light source estimation information that was previously acquired by the DL unit 202 and stored, for example, in the non-volatile memory 121. If no light source estimation information is stored, the WB control unit 203 executes S711.

[0066] Furthermore, if the difference between the acquisition date and time of the stored light source estimation information and the current date and time exceeds a threshold, the WB control unit 203 may execute S711 without acquiring past light source estimation information. Alternatively, past light source estimation information may be acquired only when certain conditions are met, such as during continuous shooting of still images or during video recording.

[0067] In S709, the WB control unit 203 determines whether or not there has been a scene change. Specifically, the WB control unit 203 compares the past light source estimation information obtained in S708 with the latest light source estimation information obtained in S701, and determines that there has been a scene change if the type of light source has changed, or if the difference in color temperature or color deviation (Δuv) is greater than or equal to a threshold. If the WB control unit 203 determines that there has been a scene change, it executes S711; otherwise, it executes S710.

[0068] In S710, the WB control unit 203 decides to use the WB correction value based on the past light source estimation information acquired in S708 for white balance adjustment in the WB control unit 203. Then, the WB control unit 203 acquires the WB correction value based on the past light source estimation information, which is stored, for example, in the non-volatile memory 121.

[0069] In S711, the WB control unit 203 decides to use the WB correction value (latest WB correction value) calculated in S507 for white balance adjustment in the WB control unit 203. The WB control unit 203 then updates the past light source estimation information and the WB correction value based on the past light source estimation information stored in the non-volatile memory 121 with the latest light source estimation information and the latest WB correction value.

[0070] According to this embodiment, the same WB correction value is continuously used unless a scene change is detected, thereby suppressing fluctuations in white balance. Therefore, it is particularly useful when shooting multiple related frames over time, such as in continuous shooting of still images or shooting video.

[0071] Alternatively, instead of using past WB correction values ​​directly, you may use a WB correction value obtained by weighting (e.g., averaging) the latest WB correction value and past WB correction values.

[0072] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0073] This embodiment includes the following image processing apparatus, imaging apparatus, image processing method, and program. (Item 1) A processing means that performs a white balance adjustment process on demosaiced image data using a white balance correction value independent of the image data, An estimation means for estimating information regarding the ambient light source of the shooting scene of the image data based on the image data that has undergone white balance adjustment processing by the processing means, A calculation means calculates a white balance correction value to be used for auto white balance adjustment processing of the demosaiced image data, based on the information regarding the ambient light source estimated by the estimation means and the demosaiced image data. An image processing apparatus characterized by having (Item 2) The image processing apparatus according to item 1, characterized in that the estimation means estimates information about the ambient light source using a trained machine learning model, and the machine learning model is trained by supervised learning using image data that has undergone white balance adjustment processing using a white balance correction value independent of the image data. (Item 3) The image processing apparatus according to item 1 or 2, characterized in that the white balance correction value independent of the image data is a white balance correction value that assumes a specific ambient light source. (Item 4) The image processing apparatus according to any one of items 1 to 3, characterized in that the estimation means estimates one or more of the following as information relating to the ambient light source: the type of ambient light source, the color temperature of the ambient light source, and the color deviation (Δuv) of the ambient light source. (Item 5) The image processing apparatus according to item 4, characterized in that the estimation means further outputs the reliability of the estimated information regarding the ambient light source. (Item 6) The image processing apparatus according to item 5, characterized in that the calculation means sets different sizes for the white detection range according to the reliability of the information regarding the ambient light source. (Item 7) The image processing apparatus according to any one of items 1 to 4, characterized in that the calculation means modifies the white balance correction value calculated by setting a white detection range corresponding to a plurality of types of ambient light sources, based on the information about the ambient light source estimated by the estimation means. (Item 8) A determination means for determining whether or not there is a scene change between the shooting scene of the latest image data and a past shooting scene, The white balance correction value used in the auto white balance adjustment process of the latest image data is If it is determined that there has been a scene change, the calculation means shall use the white balance correction value calculated for the latest image data. If it is determined that there is no scene change, the calculation means determines to use the white balance correction value that it has calculated in the past. An image processing apparatus according to any one of items 1 to 7, further comprising (Item 9) Image sensor and Demosaicing means for performing demosaicing on image data captured using the image sensor, An image processing device described in any one of items 1 to 8, A white balance adjustment means that performs an auto white balance adjustment process on the demosaiced image data using a white balance correction value calculated by the image processing device, An imaging device characterized by having the following features. (Item 10) An image processing method performed by an image processing device, The demosaiced image data is subjected to a white balance adjustment process using a white balance correction value that is independent of the image data. Based on the image data that has undergone the white balance adjustment process, information regarding the ambient light source of the shooting scene of the image data is estimated. Based on the estimated information regarding the ambient light source and the demosaiced image data, a white balance correction value to be used for automatic white balance adjustment processing of the demosaiced image data is calculated. An image processing method characterized by having the following features. (Item 11) A program for causing a computer to function as one of the means of an image processing device described in any one of items 1 through 8.

[0074] The present invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0075] 50...System control unit, 100...Electronic camera, 105...Image processing unit, 202...DL unit, 302...Resizing unit, 303...Developing unit, 304...DL processing unit

Claims

1. A processing means that performs a white balance adjustment process on demosaiced image data using a white balance correction value independent of the image data, An estimation means for estimating information regarding the ambient light source of the shooting scene of the image data based on the image data that has undergone white balance adjustment processing by the processing means, A calculation means calculates a white balance correction value to be used for auto white balance adjustment processing of the demosaiced image data, based on the information regarding the ambient light source estimated by the estimation means and the demosaiced image data. An image processing apparatus characterized by having

2. The image processing apparatus according to claim 1, characterized in that the estimation means estimates information about the ambient light source using a trained machine learning model, and the machine learning model is trained by supervised learning using image data that has undergone white balance adjustment processing using a white balance correction value independent of the image data.

3. The image processing apparatus according to claim 1, characterized in that the white balance correction value independent of the image data is a white balance correction value that assumes a specific ambient light source.

4. The image processing apparatus according to claim 1, characterized in that the estimation means estimates one or more of the following as information relating to the ambient light source: the type of ambient light source, the color temperature of the ambient light source, and the color deviation (Δuv) of the ambient light source.

5. The image processing apparatus according to claim 4, characterized in that the estimation means further outputs the reliability of the estimated information regarding the ambient light source.

6. The image processing apparatus according to claim 5, characterized in that the calculation means sets different sizes for the white detection range according to the reliability of the information regarding the ambient light source.

7. The image processing apparatus according to claim 1, characterized in that the calculation means modifies the white balance correction value calculated by setting a white detection range corresponding to a plurality of types of ambient light sources, based on the information about the ambient light source estimated by the estimation means.

8. A determination means for determining whether or not there is a scene change between the shooting scene of the latest image data and a past shooting scene, The white balance correction value used in the auto white balance adjustment process of the latest image data is If it is determined that there has been a scene change, the calculation means shall use the white balance correction value calculated for the latest image data. If it is determined that there is no scene change, the calculation means determines to use the white balance correction value that it has calculated in the past. The image processing apparatus according to claim 1, further comprising the above.

9. Image sensor and Demosaicing means for performing demosaicing on image data captured using the image sensor, An image processing apparatus according to any one of claims 1 to 8, A white balance adjustment means that performs an auto white balance adjustment process on the demosaiced image data using a white balance correction value calculated by the image processing device, An imaging device characterized by having the following features.

10. An image processing method performed by an image processing device, The demosaiced image data is subjected to a white balance adjustment process using a white balance correction value that is independent of the image data. Based on the image data that has undergone the white balance adjustment process, information regarding the ambient light source of the shooting scene of the image data is estimated. Based on the estimated information regarding the ambient light source and the demosaiced image data, a white balance correction value to be used for automatic white balance adjustment processing of the demosaiced image data is calculated. An image processing method characterized by having the following features.

11. A program for causing a computer to function as each of the means of the image processing apparatus described in any one of claims 1 to 8.

Images