Image processing device, image processing method, imaging device, and program

Abstract

To provide an image processing device, an image processing method, an imaging device, and a program capable of efficiently controlling the white balance gain applied to images.SOLUTION: The RGB statistic value of a latest frame F0 is acquired. A WB gain calculation unit 2 calculates a WB gain WBG of the latest frame. The WB gain output unit 3 outputs the kept WB gain WG as a WB gain to be applied to the latest frame F0. A data conversion unit 4 converts the RGB statistic value of the latest frame F0 into an HSV value. A histogram creation unit 5 creates a histogram of the value of H from the HSV value. A difference degree calculation unit 7 calculates the degrees of difference DH1 to DHN between the histogram of the latest frame F0 and each of the past frames. A difference degree determination unit 8 controls the update of the WB gain WBG kept by the WB gain output unit 3 on the basis of the degree of difference.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, an image processing method, an imaging device, and a program. [Background technology]

[0002] Digital cameras and other imaging devices display an image showing the imaging range in real time on a display so that the user can grasp the subject. At this time, to ensure the image quality, the white balance is automatically adjusted according to the ambient light conditions.

[0003] As an example of auto white balance, Patent Document 1 proposes a method of dividing an image frame into a plurality of blocks, and then determining a white balance gain from RGB values ​​to adjust the white balance of the frame.

[0004] Patent Document 2 proposes a method for adjusting white balance based on the difference in RGB values ​​between a current frame and a temporally previous frame. Non-Patent Document 1 also proposes a method for calculating the difference between the histograms of RGB values ​​of two frames and adding the calculated differences to find the similarity between the frames. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Special Publication No. 2009-506700 [Patent Document 2] Japanese Patent Application Laid-Open No. 2018-517369 [Non-patent literature]

[0006] [Non-Patent Document 1] Masashi Moritaki, "Verification of a basic method for measuring image similarity," Shimane University, Internet, retrieved December 7, 2023.<URL: http: / / www.shojiro-tanaka.net / alumni / 2008 / moritaki.pdf> Summary of the Invention [Problem to be solved by the invention]

[0007] However, because the above-mentioned method uses RGB values, there is a problem in that comparing RGB histograms between frames makes it difficult to accurately evaluate color differences. For example, when creating an RGB histogram, consider the case where there is a pixel with [R,G,B]=[20,40,70] and another pixel with [R,G,B]=[20,40,0]. In this case, even though the colors of the two pixels are different, the frequency of the bin with an R value of 20 is added by two, and the frequency of the bin with a G value of 40 is added by two. In other words, the frequency of the same bin increases based on pixels of different colors.

[0008] Therefore, because pixel color information cannot be accurately reflected in the histogram, two dissimilar frames may be mistakenly determined to be similar, resulting in unnecessary adjustment of the frame's white balance, resulting in an image with poor visibility.

[0009] The present disclosure has been made in consideration of the above circumstances, and aims to provide an image processing device, an image processing method, an imaging device, and a program that can efficiently control the white balance gain applied to an image. [Means for solving the problem]

[0010] An image processing device according to the present disclosure includes a statistical value acquisition unit that receives successive input frames of a RAW image and acquires statistical values ​​of RGB values ​​of the latest frame, a white balance gain calculation unit that calculates a white balance gain for the latest frame, a white balance gain output unit that holds the white balance gain calculated by the white balance gain calculation unit and outputs the held white balance gain as a white balance gain to be applied to the latest frame, a conversion unit that converts the statistical value for the latest frame into an HSV value, a histogram creation unit that creates a histogram of H values ​​from the HSV values, a difference calculation unit that calculates a difference between the histogram of the latest frame and each of histograms of one or more previous frames captured before the latest frame, and a difference determination unit that controls updating of the white balance gain held by the white balance gain output unit based on the difference. This allows pixels of different colors to be reliably identified, differences between frames to be detected with high accuracy, and appropriate white balance gains to be applied to the frames.

[0011] The image processing device according to the present disclosure is the above-mentioned image processing device, wherein the histogram creation means further creates a histogram of S and H values ​​from the HSV values, thereby making it possible to more reliably identify pixels of different colors, detect differences between frames with high accuracy, and apply suitable white balance gains to the frames.

[0012] The image processing device according to the present disclosure is the image processing device described above, wherein the dissimilarity calculation means calculates, for all bins, absolute values ​​of frequencies between a bin in a histogram of a value of interest among the HSV values ​​of the latest frame and the same bin in a histogram of the value of interest of the previous frame, and calculates an added value of the calculated values ​​as a dissimilarity between a bin in the histogram of the value of interest of the latest frame and the histogram of the value of interest of the previous frame, thereby making it possible to detect differences between frames as differences in histograms of HSV values.

[0013] The image processing device according to the present disclosure is the image processing device described above, wherein the dissimilarity determination means compares the dissimilarity with a first threshold, and if all of the calculated dissimilarity are smaller than the first threshold, stops output of the white balance gains from the white balance gain calculation means to the white balance gain output means, thereby fixing the white balance gains to be applied to the latest frame when the difference between frames is within a predetermined range.

[0014] The image processing device according to the present disclosure is the image processing device described above, wherein the first threshold is determined in advance for each of the values ​​of interest, thereby making it possible to perform appropriate threshold determination for each of the dissimilarity degrees calculated for the HSV values.

[0015] The image processing device according to the present disclosure is the image processing device described above, wherein the statistical value acquisition means acquires statistical values ​​for each of RGB values, using a frame input after the white balance gain calculation means has stopped outputting the white balance gains to the white balance gain output means as the latest frame, the difference calculation means calculates a difference between a histogram of the latest frame input after the white balance gain calculation means has stopped outputting the white balance gains to the white balance gain output means and a histogram of a previous frame captured before the latest frame, and the difference determination means compares the difference with a second threshold and, if any of the calculated differences is greater than the second threshold, resumes output of the white balance gains from the white balance gain calculation means to the white balance gain output means. This makes it possible to resume calculation of the white balance gains to be applied to the latest frame if the difference between frames falls outside a predetermined range after the white balance gains have been fixed.

[0016] The image processing device according to the present disclosure is the image processing device described above, wherein the second threshold is predetermined for each of the HSV values ​​of interest, thereby enabling pixels of different colors to be more reliably identified, differences between frames to be detected with high accuracy, and appropriate white balance gains to be applied to the frames.

[0017] The image processing device according to the present disclosure is the image processing device described above, wherein the second threshold is greater than the first threshold for each of the values ​​of interest, thereby preventing the white balance gain from being frequently released from its fixed state.

[0018] The image processing device according to the present disclosure is the image processing device described above, wherein the statistical value acquisition means divides the latest frame into a plurality of blocks of a predetermined size and acquires statistical values ​​of RGB values ​​for each block, the conversion means converts the statistical values ​​for the plurality of blocks into HSV values, the histogram creation means creates a histogram of a value of interest among the HSV values ​​for each of the plurality of blocks, and the dissimilarity calculation means calculates, for the plurality of blocks, a dissimilarity between the block in the histogram of the latest frame and a block in a histogram of a frame older than the latest frame that is located at the same position as the block in the histogram of the latest frame. This makes it possible to accurately detect local differences within a frame.

[0019] The image processing device according to the present disclosure is the above-mentioned image processing device, further comprising a smoothing processing means for smoothing the histogram created by the histogram creation means, and the dissimilarity calculation means calculates the dissimilarity based on the histogram smoothed by the smoothing processing means, thereby suppressing changes in dissimilarity between frames and making it easier to fix the white balance gain.

[0020] The image processing device according to the present disclosure further includes a histogram storage means for storing the latest frame and the previous frames created by the histogram creation means, and the dissimilarity calculation means is configured to be able to read the latest frame and the previous frames from the histogram storage means, thereby making it possible to reliably hold the histograms used in calculating the dissimilarity.

[0021] In the image processing device according to the present disclosure, the histogram storage means is configured as a ring buffer, and the histogram of the oldest frame among the histograms of frames before the latest frame that have already been stored is overwritten by the histogram of the latest frame, thereby making it easy to manage the number of frames that are the subject of dissimilarity calculation.

[0022] The image processing device according to the present disclosure further includes state information storage means for storing state information indicating a state of the image processing device, and the dissimilarity determination means stores the state information in the state information storage means to control whether the state of the image processing device is a state in which the white balance gain calculation means outputs the white balance gain to the white balance gain output means, or a state in which output of the white balance gain from the white balance gain calculation means to the white balance gain output means is stopped, and the white balance gain calculation means switches whether to calculate the white balance gain by referring to the state information.

[0023] In the image processing device according to the present disclosure, the statistical values ​​are average values ​​or integrated values ​​of each of the RGB values.

[0024] An image processing method according to the present disclosure acquires statistical values ​​of each of the RGB values ​​of the latest frame of continuously input RAW images, calculates a white balance gain for the latest frame, retains the white balance gain, and outputs the retained white balance gain as a white balance gain to be applied to the latest frame, converts the statistical values ​​for the latest frame into HSV values, creates a histogram of H values ​​from the HSV values, calculates a difference between the histogram of the latest frame and each of the histograms of one or more previous frames captured before the latest frame, and controls updating of the retained white balance gain based on the difference. This makes it possible to reliably identify pixels of different colors, detect differences between frames with high accuracy, and apply an appropriate white balance gain to the frame.

[0025] An imaging apparatus according to the present disclosure includes an imaging unit for capturing a RAW image, a statistical value acquisition unit that receives successive frames of the RAW image and acquires statistical values ​​of RGB values ​​of a most recent frame, a white balance gain calculation unit that calculates a white balance gain for the most recent frame, a white balance gain output unit that holds the white balance gain calculated by the white balance gain calculation unit and outputs the held white balance gain as a white balance gain to be applied to the most recent frame, a conversion unit that converts the statistical value for the most recent frame into an HSV value, a histogram creation unit that creates a histogram of H values ​​from the HSV values, a dissimilarity calculation unit that calculates a dissimilarity between the histogram of the most recent frame and each of histograms of one or more previous frames captured before the most recent frame, and a dissimilarity determination unit that controls updating of the white balance gain held by the white balance gain output unit based on the dissimilarity. This allows for reliable identification of pixels of different colors, highly accurate detection of differences between frames, and application of appropriate white balance gains to frames.

[0026] A program according to the present disclosure causes a computer to execute the following processes: acquiring statistical values ​​of RGB values ​​of the latest frame of continuously input RAW images; calculating white balance gains for the latest frame; retaining the white balance gains and outputting the retained white balance gains as white balance gains to be applied to the latest frame; converting the statistical values ​​for the latest frame into HSV values; creating a histogram of H values ​​from the HSV values; calculating a difference between the histogram of the latest frame and each of histograms of one or more previous frames captured before the latest frame; and controlling update of the retained white balance gains based on the difference. This enables accurate detection of differences between frames by reliably identifying pixels of different colors, and applying appropriate white balance gains to the frames. [Effects of the Invention]

[0027] According to the present disclosure, it is possible to provide an image processing device, an image processing method, an imaging device, and a program that can efficiently control the white balance gain applied to an image. [Brief explanation of the drawings]

[0028] [Figure 1] FIG. 1 is a diagram schematically illustrating an example of the configuration of an electronic imaging device. [Figure 2] 1 is a block diagram schematically illustrating a configuration of an image processing device according to a first embodiment. [Figure 3] FIG. 2 is a block diagram showing a more detailed configuration of the image processing device according to the first embodiment. [Figure 4] FIG. 2 is a diagram illustrating the temporal relationship of frames. [Figure 5] 4 is a flowchart of a WB gain control operation of the image processing device according to the first embodiment. [Figure 6] FIG. 2 is a state transition diagram of the image processing device according to the first embodiment. [Figure 7]4 is a flowchart of a WB gain control operation of the image processing device according to the first embodiment. [Figure 8] FIG. 10 is a block diagram schematically illustrating a configuration of an image processing device according to a second embodiment. [Figure 9] 10 is a flowchart of a WB gain control operation of the image processing device according to the second embodiment. [Figure 10] FIG. 10 is a block diagram schematically illustrating a configuration of an image processing device according to a third embodiment. [Figure 11] FIG. 10 is a diagram illustrating an example of an RGB block. [Figure 12] 11 is a flowchart of a WB gain control operation of the image processing device according to the third embodiment. [Figure 13] FIG. 10 is a block diagram schematically illustrating a configuration of an image processing device according to a fourth embodiment. [Figure 14] FIG. 10 is a diagram schematically illustrating an example of a smoothing filter. [Figure 15] 13 is a flowchart of a WB gain control operation of the image processing device according to the fourth embodiment. [Figure 16] FIG. 1 is a diagram illustrating an example of a hardware configuration for realizing an image processing device. DETAILED DESCRIPTION OF THE INVENTION

[0029] Specific embodiments will be described in detail below with reference to the drawings. However, the present invention is not limited to the following embodiments. For clarity of explanation, the following description and drawings have been simplified as appropriate. The same elements are given the same reference numerals, and duplicate explanations will be omitted.

[0030] Embodiment 1 An embodiment of the present invention will now be described with reference to the accompanying drawings. The image processing device described below is a device that is installed in, for example, an electronic imaging device and controls white balance gains for performing auto white balance processing on image frames captured by the electronic imaging device.

[0031] For simplicity, white balance will be abbreviated as WB (White Balance) below, and auto white balance will be abbreviated as AWB (Auto White Balance).

[0032] 1 is a diagram schematically illustrating an example of the configuration of an electronic imaging device 1000. The electronic imaging device 1000 includes an imaging unit 1001, a pre-processing unit 1002, a white balance adjustment unit (also referred to as a WB adjustment unit) 1003, a post-processing unit 1004, a display unit 1005, a storage unit 1006, and an image processing device 100.

[0033] The imaging unit 1001 is configured as a sensor module including, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor that captures an image. The imaging unit 1001 may include, for example, a lens, an image sensor having an RGB Bayer color filter, an amplifier, and an A / D converter. The imaging unit 1001 outputs the captured image as RAW data DAT, which is a digital signal. The imaging unit 1001 may have any configuration as long as it can output RAW data composed of pixels having RGB values.

[0034] The pre-processing unit 1002 performs predetermined image processing on the RAW data DAT. The pre-processing unit 1002 may perform processing such as noise reduction, pixel linearization, and shading correction on the RAW data DAT. The pre-processing unit 1002 outputs an image frame F obtained after performing predetermined processing on the RAW data DAT, which is the latest imaging result, to the WB adjustment unit 1003 and the image processing device 100.

[0035] The image processing device 100 determines a white balance gain (also referred to as WB gain) WBG suitable for displaying a white object in the image frame F in white, based on the RGB values ​​of the image frame F. Then, the image processing device 100 outputs the determined WB gain WBG to the white balance adjustment unit 1003.

[0036] The WB adjustment unit 1003 adjusts the R, G, and B gains of the image frame F based on the WB gain WBG received from the image processing device 100. Then, the WB adjustment unit 1003 adjusts the R, G, and B gains of the image frame F after the WB adjustment. WB is output to the post-processing unit 1004.

[0037] The post-processing unit 1004 processes the frame F WB The post-processing unit 1004 performs predetermined processing such as color filter array interpolation, color space conversion, gamma correction, RGB / YUV conversion, and image sharpening on the image data IMG to generate image data IMG. The post-processing unit 1004 outputs the image data IMG to the display unit 1005. The post-processing unit 1004 may also output the image data IMG to the storage unit 1006.

[0038] The display unit 1005 is configured to be able to display image data IMG, thereby allowing the user of the electronic imaging device 1000 to check the imaging range captured by the imaging unit 1001 by looking at the image displayed in real time on the display unit 1005.

[0039] The storage unit 1006 is configured to be able to store image data IMG, thereby allowing the storage unit 1006 to appropriately store images and videos captured by the electronic imaging device 1000 in response to a command from the user.

[0040] Next, a description will be given of the image processing device 100 according to this embodiment. The electronic imaging device 1000 is configured as a WB gain control device that controls the WB gain WBG to be provided to the white balance adjustment unit 1003 by performing an operation according to any one of a frame saving state, a dissimilarity determination state, and a WB gain fixed state.

[0041] Fig. 2 is a block diagram schematically showing the configuration of the image processing device 100 according to the first embodiment. The image processing device 100 includes a statistical value acquisition unit 1, a WB gain calculation unit 2, a WB gain output unit 3, a data conversion unit 4, a histogram creation unit 5, a dissimilarity calculation unit 7, and a dissimilarity determination unit 8. Fig. 3 is a block diagram showing the configuration of the image processing device 100 according to the first embodiment in more detail. The image processing device 100 may include a ring buffer 6 and a state information storage unit 9 configured as storage means.

[0042] Here, the WB gain calculation unit 2 and the WB gain output unit 3 are described as being configured separately from the white balance adjustment unit 1003, but this is merely an example. For example, the WB gain calculation unit 2, the WB gain output unit 3, and the white balance adjustment unit 1003 may be included in a general AWB module.

[0043] The statistical value acquisition unit 1 acquires RGB statistical values ​​from the latest frame, which is the RAW data of the latest image captured by the electronic imaging device 1000. Here, the statistical value acquisition unit 1 acquires the average R value of the R value in the latest frame. A , the average value of G A and the average value of B A However, the RGB statistical value may be other statistical value such as an integrated value in addition to the average value.

[0044] The WB gain calculation unit 2 performs AWB processing based on the RGB statistical values ​​of the latest frame, and calculates a WB gain WBG to be applied to the latest frame.

[0045] The WB gain output unit 3 holds the WB gain WBG output from the WB gain calculation unit 2. Then, the WB gain output unit 3 outputs the held WB gain WBG to the WB adjustment unit 1003 as necessary.

[0046] The data conversion unit 4 converts the RGB statistical value R A , G A and B Aare converted into HSV values. Here, the range of the H value, which indicates the hue, is set to 0.0° to 360.0°. The range of the saturation S value is set to 0.0 to 1.0. The range of the lightness V value is set to 0.0 to 1.0.

[0047] The histogram creation unit 5 creates a histogram of H from the HSV values ​​converted by the data conversion unit 4. Here, as an example, the histogram creation unit 5 creates a histogram of H with 72 bins at 5° intervals.

[0048] The ring buffer 6 is configured as a storage unit that stores histograms created by the histogram creation unit 5. The histogram creation unit 5 can store histograms of multiple frames that are sequentially input in the ring buffer 6. When the ring buffer 6 is full with histograms of previously input frames, the histogram creation unit 5 overwrites the histogram of the oldest frame stored in the ring buffer 6 with the histogram created for the latest frame.

[0049] The dissimilarity calculation unit 7 calculates the dissimilarity of H between the latest frame and each of a predetermined number N of past frames based on the histogram of H stored in the ring buffer 6. N is an integer equal to or greater than 1.

[0050] FIG. 4 is a diagram showing the temporal relationship of frames. Here, the timing at which the latest frame is captured is T=0, and the latest frame is F0. Also, the timing at which a previous frame is captured is T=i, and the previous frame is F1. i Here, i is an integer equal to or greater than 1 that indicates the order of past frames, and the larger the value, the earlier the timing.

[0051] The dissimilarity calculation unit 7 calculates the histogram h[H0] of H of the latest frame F0 and the histogram h[H0] of H of the past frames F1 to F2. N The degree of difference between the histograms h[H1] to h[HN] of H and is calculated.

[0052] Here, the number of bins in the histogram of H is M H, Frame F i The frequency of the bin is bH i,j Note that j is 1 or more and M H The dissimilarity calculation unit 7 calculates the frequency bH of the j-th bin of the histogram h[H0] of H of the latest frame F0. 0,j and the kth past frame F k The frequency of the jth bin of the histogram h[Hk] of H is bH k,j Absolute value of the difference between |bH 0,j -bH k,j | is calculated for all bins. Then, the dissimilarity calculation unit 7 calculates the calculated M H The sum of the absolute values ​​of the latest frame F0 and the previous frame F k The degree of difference D between Hk This is calculated as follows:

number

[0053] The difference determination unit 8 calculates the difference D H1 ~D HN The fixed gain threshold H TH1 is a predetermined value. Hereinafter, comparing the dissimilarity of H with the fixed gain threshold to determine which of the two values ​​is larger will be referred to as a fixed gain determination. The fixed gain threshold will also be referred to as a first threshold.

[0054] The difference determination unit 8 determines the difference D H1 ~D HN All of these are at the fixed gain threshold H TH1 The dissimilarity determining unit 8 makes a state transition of the image processing device 100 according to the determination result.

[0055] The state information storage unit 9 is configured to be able to store state information INF indicating the state of the image processing device 100.

[0056] Next, a WB gain control operation of the image processing device 100 will be described. Fig. 5 is a flowchart of the WB gain control operation of the image processing device 100 according to the first embodiment. Fig. 6 is a state transition diagram of the image processing device 100 according to the first embodiment. In the initial state, the state of the image processing device 100 is set to a frame saving state.

[0057] Step ST1 The statistical value acquisition unit 1 calculates the RGB statistical values ​​of the latest frame F0.

[0058] Step ST2 The WB gain calculation unit 2 determines whether the current state is a fixed WB gain state by referring to the state information INF stored in the state information storage unit 9. If the current state is a fixed WB gain state, the WB gain calculation unit 2 proceeds to step ST10.

[0059] Step ST3 If the current state is not the WB gain fixed state, the WB gain calculation unit 2 executes AWB processing of the latest frame F0 to calculate the WB gain WBG, and outputs the calculated WB gain WBG to the WB gain output unit 3.

[0060] As a result, the WB gain output unit 3 holds the latest calculated WB gain WBG. If the WB gain output unit 3 already holds a WB gain WBG calculated in the past, the held WB gain WBG calculated in the past is discarded and the latest WB gain WBG is newly held.

[0061] Step ST4 The data conversion unit 4 refers to the state information INF stored in the state information storage unit 9, and if the current state is not a fixed WB gain state, converts the RGB statistical values ​​of the latest frame F0 into HSV values. Note that if it is ensured in step ST2 that the current state is not a fixed WB gain state, the state confirmation in the data conversion unit 4 may be omitted as appropriate.

[0062] Step ST5 The histogram creation unit 5 creates a histogram h[H0] of H based on the HSV values ​​converted by the data conversion unit 4. Then, the histogram creation unit 5 stores the created histogram h[H0] of H in the ring buffer 6.

[0063] If the ring buffer 6 is already full when step ST5 starts, the histogram information of the oldest frame stored in the ring buffer 6 is replaced with the histogram of the latest frame F0.

[0064] Step ST6 The dissimilarity calculation unit 7 determines whether the number of frames for which histograms have been stored has reached a predetermined value N, i.e., whether the ring buffer 6 is full. If the number of frames for which histograms have been stored has not reached the predetermined value N, i.e., if there is free space in the ring buffer 6, the dissimilarity calculation unit 7 returns the process to step ST1.

[0065] Step ST7 When the number of frames for which histograms have been stored reaches a predetermined value N, that is, when the ring buffer 6 is full, the state of the image processing device 100 transitions from the frame saving state to the dissimilarity determination state. For example, as shown in FIG. 3, the dissimilarity calculation unit 7 may store state information INF, which indicates that the state of the image processing device 100 is the dissimilarity determination state, in the state information storage unit 9. In this case, the dissimilarity calculation unit 7 calculates the dissimilarity between the latest frame F0 and the past frames F1 to F2. N Using the histogram of H, the latest frame F0 and past frames F1 to F N The degree of difference D between H and H1 ~D HN Calculate.

[0066] Step ST8 The difference determination unit 8 compares the latest frame F0 with the past frames F1 to F2. N The difference D between H and H1 ~D HNThe gain fixed threshold H TH1 The gain is fixed by comparing with the difference D. H1 ~D HN Either of these is the gain fixed threshold H TH1 If so, the dissimilarity determination unit 8 returns the process to step ST1. In this case, the state of the image processing device 100 transitions from the dissimilarity determination state to the frame storage state. The dissimilarity determination unit 8 may, for example, store state information INF indicating that the state of the image processing device 100 is the frame storage state in the state information storage unit 9.

[0067] Step ST9 Dissimilarity D of H H1 ~D HN All of these are at the fixed gain threshold H TH1 If the difference is smaller than the difference, the dissimilarity determination unit 8 resets the ring buffer 6 and returns the process to step ST1. At this time, the state of the image processing device 100 transitions from the dissimilarity determination state to the fixed WB gain state. The dissimilarity determination unit 8 stores state information INF, which indicates that the state of the image processing device 100 is the fixed WB gain state, in the state information storage unit 9.

[0068] Step ST10 The WB gain output unit 3 outputs the latest WB gain WBG that it holds to the WB adjustment unit 1003 as the WB gain to be applied to the latest frame F0.

[0069] As described above, this configuration makes it possible to determine differences between frames based on the distribution of H values, which uniquely indicate the color information of pixels in an image, rather than RGB values. This makes it possible to reliably identify pixels of different colors and detect differences between frames with higher accuracy.

[0070] As a result, the image processing device 100 can obtain a more appropriate WB gain for the latest frame and apply the obtained WB gain in a fixed manner.

[0071] After the WB gain is fixed, it may be necessary to change the WB gain depending on the state of the frame. In response to this, the image processing device 100 can also update the WB gain that has been fixed as needed depending on the state of the input frame.

[0072] Fig. 7 is a flowchart of the WB gain control operation of the image processing device 100 according to the first embodiment. In Fig. 7, steps ST11 to ST15 for updating the WB gain are added after step ST10, as compared to Fig. 5. Note that steps ST1 to ST10 in Fig. 7 are the same as those in Fig. 5, and therefore redundant explanations will be omitted.

[0073] Step ST11 The data conversion unit 4 converts the RGB statistical values ​​of the latest frame F0 into HSV values.

[0074] Step ST12 The histogram creation unit 5 creates a histogram h[H0] of H based on the HSV values ​​of the latest frame F0. The histogram creation unit 5 then stores the created histogram h[H0] of H in the ring buffer 6.

[0075] Step ST13 The dissimilarity calculation unit 7 calculates the dissimilarity D of H between the latest frame F0 and the previous frame F1 using the histogram h[H0] of H of the latest frame F0 and the histogram h[H1] of H of the previous frame F1. H1 Calculate.

[0076] Step ST14 The difference determination unit 8 determines the difference D of H between the latest frame F0 and the previous frame F1. H1 is the gain release threshold H TH2 Determine whether it is smaller than the difference D of H. H1 is the gain release threshold H TH2If the difference is smaller than the threshold, the dissimilarity determination unit 8 returns the process to step ST1. In this case, the state of the image processing device 100 is maintained in the WB gain fixed state. The gain fix release threshold is also referred to as the second threshold.

[0077] Step ST15 Dissimilarity D of H H1 is the gain release threshold H TH2 If so, the dissimilarity determination unit 8 resets the ring buffer 6 and returns the process to step ST1. At this time, the state of the image processing device 100 transitions from the WB gain fixed state to the frame storage state. The dissimilarity determination unit 8 stores state information INF, which indicates that the state of the image processing device 100 is the frame storage state, in the state information storage unit 9.

[0078] As a result, if the state of a frame captured after the WB gain has been fixed changes significantly, the WB gain can be released and the processing of steps ST1 to ST9 can be resumed to update the WB gain as appropriate. Note that, for example, by setting the gain release threshold to a value greater than the WB gain fixation threshold, it is possible to prevent the WB gain once determined from being updated too frequently. As a result, by using the same WB gain when changes in the image are small, it is possible to prevent frequent changes in image visibility.

[0079] Although the comparison between the latest frame and one past frame has been described here, two or more past frames may be compared as necessary.

[0080] As described above, with this configuration, the WB gain to be applied to the latest frame can be determined according to the distribution of H values ​​that uniquely indicate the color information of the pixels in the image, and the WB gain can be continuously updated.

[0081] As a result, the image processing device 100 can continuously apply an appropriate WB gain to the latest frame.

[0082] Embodiment 2 In the first embodiment, the case where H, which uniquely specifies the color of a pixel, is used among the HSV values, has been described. In contrast, in the present embodiment, an image processing device will be described that uses not only H but also S and V to more accurately evaluate the degree of difference between frames.

[0083] 8 is a block diagram schematically illustrating a configuration of an image processing device 200 according to the second embodiment. The image processing device 200 has a configuration in which the histogram creation unit 5, the dissimilarity calculation unit 7, and the dissimilarity determination unit 8 of the image processing device 100 are replaced with a histogram creation unit 10, a dissimilarity calculation unit 11, and a dissimilarity determination unit 12, respectively.

[0084] The histogram creation unit 10 creates a histogram of H in the same way as the histogram creation unit 5, and also creates histograms of S and V converted by the data conversion unit 4. Here, the histogram creation unit 10 creates a histogram of 20 bins at intervals of 0.05, for example.

[0085] The dissimilarity calculation unit 11 calculates the dissimilarity of H, S, and V between the latest frame and each of the predetermined number N of past frames based on the histograms of H, S, and V stored in the ring buffer 6. The calculation of the dissimilarity of H in the dissimilarity calculation unit 11 is similar to that in the dissimilarity calculation unit 7.

[0086] The dissimilarity calculation unit 11 calculates the histogram h[S0] of S of the latest frame F0 and the histogram h[S0] of S of the past frames F1 to F2. N Calculate the dissimilarity between the histogram h[S1]~h[SN] of S and the number of bins of the histogram of S is M S , Frame F i The frequency of the bin is bS i,j The dissimilarity calculation unit 11 calculates the frequency bS of the j-th bin in the histogram of S of the latest frame F0. 0,j and the kth past frame F k The frequency of the jth bin of the histogram of S is bS k,j Absolute value of the difference between |bS 0,j -bS k,jThen, the dissimilarity calculation unit 11 calculates the calculated M S The sum of the absolute values ​​of the latest frame F0 and the previous frame F k The degree of difference D between Sk This is calculated as follows:

number

[0087] The dissimilarity calculation unit 11 calculates the histogram h[V0] of V of the latest frame F0 and the histogram h[V0] of V of the past frames F1 to F2. N Calculate the dissimilarity between the histograms h[V1]~h[VN] of V and the number of bins of the histogram of V is M V , Frame F i The frequency of the bin is bV i,j The dissimilarity calculation unit 11 calculates the frequency bV of the j-th bin in the histogram of V of the latest frame F0. 0,j and the kth past frame F k The frequency of the jth bin of the histogram of V is bV k,j Absolute value of the difference between |bV 0,j -bV k,j Then, the dissimilarity calculation unit 11 calculates the calculated M v The sum of the absolute values ​​of the latest frame F0 and the previous frame F k The degree of difference D between Vk This is calculated as follows:

number

[0088] As described above, the dissimilarity calculation unit 11 calculates the dissimilarity D of the histogram of H. H1 ~D HN , the dissimilarity D of the histograms of S S1 ~D SN and the dissimilarity D of the histograms of V V1 ~D VNFor the sake of simplicity, in FIG. 8, the HSV histograms are represented by h[H*], h[S*], and h[V*], and the dissimilarity measures are represented by D H* , D S* and D V* This notation is also used in the following figures.

[0089] The dissimilarity determination unit 12, like the dissimilarity determination unit 8, performs threshold determination of the calculated dissimilarity of H. The dissimilarity determination unit 8 also performs threshold determination of the calculated dissimilarity of S and V. The fixed gain threshold S for S used for threshold determination is TH1 and a fixed gain threshold V for V TH1 is a predetermined value.

[0090] The difference determination unit 12 determines the difference D H1 ~D HN All of these are fixed gain H TH1 Furthermore, the dissimilarity determining unit 12 determines whether the dissimilarity D of S is smaller than S1 ~D SN All of these are fixed thresholds S TH1 Furthermore, the dissimilarity determining unit 12 determines whether the dissimilarity D V1 ~D VN All of these are fixed at the gain threshold V TH1 Determine whether it is smaller than

[0091] As a result, the dissimilarity determining unit 12 determines the dissimilarity D H1 ~D HN All of these are fixed gain H TH1 is smaller than the dissimilarity D of S. S1 ~D SN All of these are fixed thresholds S TH1 is smaller than V and the dissimilarity D V1 ~D VN All of these are fixed at the gain threshold V TH1 It is determined whether the state is smaller than

[0092] Below, the difference D of H H1 ~D HN All of these are fixed gain HTH1 is smaller than the dissimilarity D of S. S1 ~D SN All of these are fixed thresholds S TH1 is smaller than V and the dissimilarity D V1 ~D VN All of these are fixed at the gain threshold V TH1 If the difference D of H is smaller than H1 ~D HN , the dissimilarity D of S S1 ~D SN and the dissimilarity D of V V1 ~D VN is equal to or greater than the corresponding gain fixation threshold, the result of the gain fixation determination is deemed false.

[0093] The dissimilarity determining unit 12 transitions the state of the image processing device 200 to a specific state depending on the result of the gain fixation determination.

[0094] Next, a description will be given of the WB gain control operation of the image processing device 200. Fig. 9 is a flowchart of the WB gain control operation of the image processing device 200 according to the second embodiment.

[0095] Steps ST1 to ST4 Steps ST1 to ST4 are the same as those in FIGS. 5 and 7, and therefore redundant explanations will be omitted.

[0096] Step ST21 The histogram creation unit 10 creates histograms for H, S, and V based on the HSV values ​​of the latest frame F0. The histogram creation unit 5 then stores the created histograms for H, S, and V in the ring buffer 6.

[0097] Step ST6 Step ST6 is the same as in FIGS. 5 and 7, so a duplicated description will be omitted.

[0098] Step ST22 When the number of frames for which histograms have been stored reaches a predetermined value, that is, when the ring buffer 6 is full, the state of the image processing device 200 transitions from the frame storage state to the dissimilarity determination state. In this case, the dissimilarity calculation unit 11 calculates the dissimilarity of the latest frame F0 and the past frames F1 to F2. N Using the histograms of H, S, and V, the latest frame F0 and past frames F1 to F N Calculate the differences between H, S, and V.

[0099] Step ST23 The difference determination unit 12 compares the latest frame F0 with the past frames F1 to F2. N As described above, the gain fixation determination is performed based on the differences in H, S, and V between the image processing device 200 and the frame saving state. If the result of the gain fixation determination is false, the dissimilarity determination unit 12 returns the process to step ST1. In this case, the state of the image processing device 200 transitions from the dissimilarity determination state to the frame saving state.

[0100] Step ST24 If the result of the fixed gain determination is true, the dissimilarity determination unit 12 resets the ring buffer 6 and returns the process to step ST1. At this time, the state of the image processing device 200 transitions from the dissimilarity determination state to the fixed WB gain state. The dissimilarity determination unit 12 stores state information INF, which indicates that the state of the image processing device 200 is the fixed WB gain state, in the state information storage unit 9.

[0101] Steps ST10 and ST11 Steps ST10 and ST11 are the same as those in FIGS. 5 and 7, and therefore redundant explanations will be omitted.

[0102] Step ST25 The histogram creation unit 10 creates histograms for H, S, and V based on the HSV values ​​of the latest frame F0. The histogram creation unit 5 then stores the created histograms for H, S, and V in the ring buffer 6.

[0103] Step ST26 The dissimilarity calculation unit 11 calculates the dissimilarity D of H, S, and V between the latest frame F0 and the previous frame F1 using the histograms h[H0], h[S0], and h[V0] of H, S, and V of the latest frame F0 and the histograms h[H1], h[S1], and h[V1] of H, S, and V of the most recent previous frame F1. H1 , D S1 and D V1 Calculate.

[0104] Step ST27 The difference determination unit 12 determines the difference D of H, S, and V between the latest frame F0 and the previous frame F1. H1 , D S1 and D V1 are the gain release thresholds H TH2 , S TH2 and V TH2 Determine whether the difference D between H, S, and V is smaller than H1 , D S1 and D V1 are the gain release thresholds H TH2 , S TH2 and V TH2 If the difference is smaller than , the dissimilarity determining unit 12 determines that there is no need to release the fixed WB gain, and returns the process to step ST1 In this case, the state of the image processing device 200 is maintained in the fixed WB gain state.

[0105] Step ST28 Difference D between H, S and V H1 , D S1 and D V1 If any of these is equal to or greater than the corresponding gain fixation release threshold, the dissimilarity determination unit 12 resets the ring buffer 6 and returns the process to step ST1. At this time, the state of the image processing device 200 transitions from the WB gain fixation state to the frame storage state. The dissimilarity determination unit 12 stores state information INF, which indicates that the state of the image processing device 200 is the frame storage state, in the state information storage unit 9.

[0106] As a result, if the state of the frame captured after the WB gain is fixed changes significantly, the WB gain can be released from the fixed state and the WB gain fixation process of steps ST1 to ST9 can be performed again to appropriately update the WB gain.

[0107] As described above, this configuration makes it possible to determine differences between frames based on the distribution of HSV values ​​rather than RGB values, thereby enabling pixels of different colors to be identified reliably and differences between frames to be detected with higher accuracy.

[0108] Furthermore, with this configuration, it is also possible to continuously update the WB gain to be applied to the latest frame in accordance with the distribution of HSV values ​​that uniquely indicate color information of pixels in an image.

[0109] As a result, the image processing device 200 can continuously and more effectively apply an appropriate WB gain to the latest frame.

[0110] Embodiment 3 In the first and second embodiments, the image processing device that performs processing on a frame-by-frame basis has been described. In contrast, in the present embodiment, an image processing device that divides a frame of a RAW image into a plurality of RGB blocks of a predetermined size and performs processing will be described.

[0111] 10 is a block diagram schematically illustrating the configuration of an image processing device 300 according to the third embodiment. The image processing device 300 has a configuration in which the statistical value acquisition unit 1, the data conversion unit 4, the histogram creation unit 10, the dissimilarity calculation unit 11, and the dissimilarity determination unit 12 of the image processing device 200 are replaced with a statistical value acquisition unit 13, a data conversion unit 14, a histogram creation unit 15, a dissimilarity calculation unit 16, and a dissimilarity determination unit 17, respectively.

[0112] The statistical value acquisition unit 13 divides the latest frame F0 into a predetermined number n of RGB blocks and acquires RGB statistical values ​​for each of the divided RGB blocks. Hereinafter, the RGB statistical values ​​acquired for each RGB block are referred to as RGB block statistical values. Here, n is an integer equal to or greater than 2.

[0113] Fig. 11 is a diagram showing an example of an RGB block. In a RAW image, R, G, and B pixels are generally arranged according to a Bayer array. Fig. 11 shows an RGGB type Bayer array. As shown in Fig. 11, the statistical value acquisition unit 13 divides the latest frame F0 into RGB blocks of 16 pixels, each consisting of, for example, 4 pixels vertically and 4 pixels horizontally.

[0114] However, the size of the RGB blocks is merely an example, and the RGB blocks may be blocks containing any number of pixels other than 16. Furthermore, the RGB blocks may be blocks obtained by dividing a single frame without overlapping, or may be blocks that overlap with adjacent blocks.

[0115] The statistical value acquisition unit 13 calculates the average value R of the values ​​of R, G, and B of the n RGB blocks included in the latest frame F0. A1 ~R An , G A1 ~G An and B A1 ~B An is obtained as the RGB block statistics.

[0116] When one frame is divided into a plurality of RGB blocks and an RGB statistical value is calculated for each RGB block, the WB gain calculation unit 2 can calculate the WB gain appropriately based on a known method.

[0117] The data conversion unit 14 converts the block BL p RGB block statistics R obtained for Ap , G Ap and B Ap , which is the HSV value H p , S p and V p That is, the data conversion unit 14 converts the blocks BL1 to BL n RGB block statistics R obtained for A1 ~R An , G A1 ~G Anand B A1 ~B An H1~H n , S1~S n and V1 to V n In Fig. 10, for simplicity, the RGB block statistics of each RGB value are converted into R A* , G A* and B A* , converted multiple HSV to H * , S * and V * It is written as follows.

[0118] The histogram creation unit 15 creates histograms for H, S, and V of each RGB block. n Histogram of S1~S n Histogram of V1 to V n Create a histogram of

[0119] The dissimilarity calculation unit 16 calculates the dissimilarity between the latest frame F0 and the past frames F1 to F2 based on the histograms of H, S, and V of each RGB block stored in the ring buffer 6. N The RGB blocks BL at the same position p Sex between p , S p and V p The dissimilarity is calculated.

[0120] The dissimilarity calculation unit 16 calculates the dissimilarity of each block BL p Histograms of H, S and V of the previous frames F1 to F N Calculate the dissimilarity between the H, S, and V histograms of and.

[0121] Frame F i RGB block BL p The frequency of the bins of the histogram of H is bH i,p,j The dissimilarity calculation unit 16 calculates the RGB block BL p The frequency of the jth bin of the histogram of H is bH 0,p,j and past frame F kRGB block BL p The frequency of the jth bin of the histogram of H is bH k,p,j Absolute value of the difference between |bH 0,p,j -bH k,p,j Then, the dissimilarity calculation unit 16 calculates the calculated M H The sum of the absolute values ​​of the RGB blocks BL p and past frame F k RGB block BL p The difference between H and DH k,p This is calculated as follows:

number

[0122] Frame F i RGB block BL p The frequency of the bins of the histogram of S is bS i,p,j The dissimilarity calculation unit 16 calculates the RGB block BL p The frequency of the jth bin of the histogram of S is bS 0,p,j and past frame F k RGB block BL p The frequency of the jth bin of the histogram of S is bS k,p,j Absolute value of the difference between |bS 0,p,j -bS k,p,j Then, the dissimilarity calculation unit 16 calculates the calculated M S The sum of the absolute values ​​of the RGB blocks BL p and past frame F k RGB block BL p The difference DS between S and k,pThis is calculated as follows:

number

[0123] Frame F i RGB block BL p The frequency of the bins of the histogram of V is bV i,p,j The dissimilarity calculation unit 16 calculates the RGB block BL p The frequency of the jth bin of the histogram of V is bV 0,p,j and past frame F k RGB block BL p The frequency of the jth bin of the histogram of V is bV k,p,j Absolute value of the difference between |bV 0,p,j -bV k,p,j Then, the dissimilarity calculation unit 16 calculates the calculated M V The sum of the absolute values ​​of the RGB blocks BL p and past frame F k RGB block BL p DV: the difference between V and k,p This is calculated as follows:

number

[0124] The dissimilarity determining unit 17 determines whether all of the calculated N×n dissimilarity values ​​of H are greater than the fixed gain threshold H TH1 Furthermore, the dissimilarity determining unit 17 determines whether all of the calculated N×n dissimilarity values ​​S are smaller than the fixed gain threshold S TH1 Furthermore, the dissimilarity determining unit 17 determines whether all of the calculated N×n dissimilarity values ​​of V are smaller than the fixed gain threshold V TH1 Determine whether it is smaller than

[0125] As a result, the dissimilarity determining unit 17 determines whether all of the dissimilarity values ​​of H are equal to or greater than the fixed gain threshold H TH1 is smaller than the gain fixed threshold S TH1 and all of the dissimilarity values ​​of V are below the fixed gain threshold V TH1 It is determined whether the state is smaller than

[0126] As in the second embodiment, if all of the dissimilarities of H, S, and V are smaller than the corresponding gain fixation threshold, the result of the gain fixation judgment is determined to be true. On the other hand, if any of the dissimilarities of H, S, and V is equal to or greater than the corresponding gain fixation threshold, the result of the gain fixation judgment is determined to be false.

[0127] The dissimilarity determining unit 17 transitions the state of the image processing device 300 to a specific state depending on the result of the gain fixation determination.

[0128] Next, a description will be given of the WB gain control operation of the image processing device 300. Fig. 12 is a flowchart of the WB gain control operation of the image processing device 300 according to the third embodiment.

[0129] Step ST31 The statistical value acquisition unit 13 divides the latest frame F0 into a predetermined number of RGB blocks, and calculates the RGB statistical value of each RGB block.

[0130] Steps ST2 and ST3 Steps ST2 and ST3 are the same as those in FIGS. 5 and 7, and therefore redundant explanations will be omitted.

[0131] Step ST32 The data conversion unit 14 converts the RGB statistical values ​​of each RGB block of the latest frame F0 into HSV values.

[0132] Step ST33 The histogram creation unit 15 creates histograms for H, S, and V based on the HSV values ​​of each RGB block in the latest frame F0. The histogram creation unit 15 then stores the created histograms for H, S, and V in the ring buffer 6.

[0133] Step ST6 Step ST6 is the same as in FIGS. 5 and 7, so a duplicated description will be omitted.

[0134] Step ST34 When the number of frames for which histograms have been stored reaches a predetermined value, that is, when the ring buffer 6 is full, the state of the image processing device 300 transitions from the frame storage state to the dissimilarity determination state. In this case, the dissimilarity calculation unit 16 calculates the dissimilarity of the latest frame F0 and the past frames F1 to F2. N Using the histograms of H, S, and V, the latest frame F0 and past frames F1 to F N Calculate the differences between H, S, and V.

[0135] Step ST35 The difference determination unit 17 compares the latest frame F0 with the past frames F1 to F2. N As described above, the gain fixation determination is performed based on the differences in H, S, and V between the image processing device 300 and the frame saving state. If the result of the gain fixation determination is false, the dissimilarity determination unit 17 returns the process to step ST1. In this case, the state of the image processing device 300 transitions from the dissimilarity determination state to the frame saving state.

[0136] Step ST36 If the result of the fixed gain determination is true, the dissimilarity determination unit 17 resets the ring buffer 6 and returns the process to step ST1. At this time, the state of the image processing device 300 transitions from the dissimilarity determination state to the fixed WB gain state. The dissimilarity determination unit 17 stores state information INF, which indicates that the state of the image processing device 300 is the fixed WB gain state, in the state information storage unit 9.

[0137] Step ST10 Step ST10 is the same as in FIGS. 5 and 7, so a duplicated description will be omitted.

[0138] Step ST37 The data conversion unit 14 converts the RGB statistical values ​​of each RGB block of the latest frame F0 into HSV values.

[0139] Step ST38 The histogram creation unit 15 creates histograms for H, S, and V based on the HSV values ​​of each RGB block in the latest frame F0. The histogram creation unit 15 then stores the created histograms for H, S, and V in the ring buffer 6.

[0140] Step ST39 The dissimilarity calculation unit 16 calculates the dissimilarity of H, S, and V for each block of the latest frame F0 and the previous frame F1 using the histogram of H, S, and V for each RGB block of the latest frame F0 and the histogram of H, S, and V for each RGB block of the most recent previous frame F1.

[0141] Step ST40 The dissimilarity determination unit 17 determines whether the dissimilarity of H, S, and V between the latest frame F0 and the previous frame F1 is smaller than the corresponding gain fixation release threshold. If the dissimilarity of H, S, and V is smaller than the corresponding gain fixation release threshold, the dissimilarity determination unit 17 determines that there is no need to release the WB gain fixation, and returns the process to step ST31. In this case, the state of the image processing device 300 is maintained in the WB gain fixation state.

[0142] Step ST41 If any of the H, S, and V dissimilarities is equal to or greater than the corresponding gain fixation release threshold, the dissimilarity determination unit 17 resets the ring buffer 6 and returns the process to step ST31. At this time, the state of the image processing device 300 transitions from the WB gain fixation state to the frame storage state. The dissimilarity determination unit 17 stores state information INF, which indicates that the state of the image processing device 300 is the frame storage state, in the state information storage unit 9.

[0143] As a result, similarly to the second embodiment, the image processing device 300 can continuously and more effectively apply an appropriate WB gain to the latest frame.

[0144] Furthermore, the image processing device 300 divides one frame into multiple RGB blocks. This makes it possible to capture changes in pixels within the RGB blocks within one frame, i.e., local changes. This makes it possible to more sensitively capture differences between frames compared to the image processing devices 100 and 200.

[0145] As a result, the image processing device 300 can continuously and effectively apply a more appropriate WB gain to the latest frame.

[0146] Embodiment 4 In this embodiment, an image processing device will be described that performs a smoothing process on a histogram prior to calculating the dissimilarity in order to prevent frequent fluctuations in WB gain in response to changes in frames.

[0147] 13 is a block diagram schematically illustrating a configuration of an image processing device 400 according to the fourth embodiment. The image processing device 400 has a configuration in which a smoothing processing unit 19 is added to the image processing device 200.

[0148] The smoothing processor 19 performs smoothing filtering on the histogram created by the histogram creator 10 and stores the processed histogram in the ring buffer 6 .

[0149] In this embodiment, the smoothing processing unit 19 applies a smoothing filter to each bin of one histogram. Fig. 14 is a diagram schematically showing an example of the smoothing filter. The smoothing filter in Fig. 14 is made up of filters FL0, FL1, FL2, and FL -1 and FL -2 It has.

[0150] The filter FL0 multiplies the frequency b0 of a bin B0 selected from one histogram by a filter coefficient K0.

[0151] The filter FL1 multiplies the frequency b1 of the bin B1 adjacent to the bin B0 on ​​the higher side by a filter coefficient K1, and the filter FL2 multiplies the frequency b2 of the bin B2 adjacent to the bin B1 on the higher side by a filter coefficient K2.

[0152] Filter FL -1 is the value of the bin B adjacent to the bin B0 on ​​the lower side. -1 Degree b -1 is multiplied by the filter coefficient K1. -2 is Bin B -1 Bin B is adjacent to the lower side -2 Degree b -2 is multiplied by the filter coefficient K2.

[0153] For example, the filter coefficient K0 may be 0.4, the filter coefficient K1 may be 0.2, and the filter coefficient K2 may be 0.1.

[0154] The adder 19A outputs the filters FL0, FL1, FL2, and FL -1 and FL -2 The adder 18A adds the five filtered values ​​output from the adders 18A and 18B. The adder 18A then outputs the added value as the frequency b'0 of the bin B0 after the smoothing process.

[0155] When smoothing the histogram of H, the value of H is the same when it is 0° as when it is 360°, so the bin in the range of 0° to 5° and the bin in the range of 355° to 360° are treated as adjacent bins.

[0156] The smoothing processing unit 19 creates a histogram based on the bin values ​​after the smoothing processing, and stores the created histogram in the ring buffer 6.

[0157] In this configuration, the dissimilarity calculation unit 11 obtains the smoothed histogram from the ring buffer 6 and calculates the dissimilarity.

[0158] Next, a description will be given of the WB gain control operation of the image processing device 400. Fig. 15 is a flowchart of the WB gain control operation of the image processing device 400 according to the third embodiment.

[0159] Steps ST1 to ST4 Steps ST1 to ST4 are the same as those in FIGS. 5 and 7, and therefore redundant explanations will be omitted.

[0160] Step ST51 The histogram creation unit 18 creates histograms of H, S, and V based on the HSV values ​​of each RGB block of the latest frame F0, similar to the histogram creation unit 10. The histogram creation unit 18 then outputs the created histograms of H, S, and V to the smoothing processing unit 19.

[0161] Step ST52 The smoothing processing unit 19 processes the H, S, and V histograms created by the histogram creation unit 18 using the smoothing filter described above. The smoothing processing unit 19 stores the histograms after the smoothing processing in the ring buffer 6. As a result, the dissimilarity calculation unit 11 calculates the dissimilarity using the histograms after the smoothing processing.

[0162] Steps ST6, ST10, ST11, ST22 to ST28 Steps ST6, ST10, ST11, and ST22 to ST28 are the same as those in FIG. 9, so redundant explanations will be omitted.

[0163] As described above, this configuration makes it possible to smooth the frequency distribution in the histogram used to calculate the dissimilarity. This reduces the sensitivity for detecting differences between frames and makes it easier to fix the WB gain. It also makes it possible to prevent the WB gain from being updated too frequently.

[0164] Other embodiments Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0165] In the image processing device according to the above-described embodiment, the gain release threshold may be set to a value greater than the gain fixation threshold, thereby preventing the WB gain, once fixed, from being frequently updated.

[0166] In the image processing device according to the first embodiment, similarly to the third embodiment, WB gain control may be performed by dividing one frame into a predetermined number of RGB blocks, rather than in units of frames.

[0167] In the image processing device according to the fourth embodiment, WB gain may be controlled using only the histogram of H, as in the first embodiment. Also, in the image processing device according to the fourth embodiment, WB gain may be controlled by dividing one frame into a predetermined number of RGB blocks, rather than on a frame-by-frame basis, as in the third embodiment.

[0168] The image processing device according to the above-described embodiment may have other functions and configurations as appropriate in addition to the configuration for controlling the WB gain. For example, the image processing device may adaptively expand the histogram toward the bright side or increase the brightness and contrast of the frame. Furthermore, the image processing device may adaptively expand the histogram toward the dark side or calculate R, G, and B offsets to increase the contrast of the image.

[0169] In the above-described embodiment, a determination of the magnitude of two values ​​has been described. However, this is merely an example, and cases where the two values ​​are equal in a determination of the magnitude of two values ​​may be handled as needed. That is, either the determination of whether a first value is greater than or equal to a second value or less than the second value, or the determination of whether a first value is greater than or equal to the second value, may be adopted as needed. Either the determination of whether a first value is less than or equal to a second value or greater than the second value, or the determination of whether a first value is less than or equal to the second value or greater than or equal to the second value, may be adopted. In other words, when two values ​​are determined to be greater than or equal to each other to obtain two determination results, a case where the two values ​​are equal may be included in either of the two determination results as needed.

[0170] In the above-described embodiment, the image processing device according to the present disclosure has been described primarily as a hardware configuration. However, this is not limiting, and any processing can be realized by having a central processing unit (CPU) execute a computer program. In this case, the computer program can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, programmable ROMs (PROMs), erasable PROMs (EPROMs), flash ROMs, and random access memories (RAMs)). The program may also be supplied to a computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire or an optical fiber, or via a wireless communication path.

[0171] An example of the hardware configuration of an image processing device is shown below. FIG. 16 is a diagram showing an example of the hardware configuration for realizing the image processing device. The image processing device can be realized by a computer 9000, such as a dedicated computer or a personal computer (PC). However, the computer does not need to be physically single; multiple computers may be used when performing distributed processing. As shown in FIG. 16, the computer 9000 has a CPU (Central Processing Unit) 9001, a ROM (Read Only Memory) 9002, and a RAM (Random Access Memory) 9003, which are interconnected via a bus 9004. Note that although explanation of the OS software and the like for operating the computer will be omitted, it is assumed that the computer that builds this network analysis system also has such software.

[0172] An input / output interface 9005 is also connected to the bus 9004. To the input / output interface 9005, for example, an input unit 9006 including a keyboard, mouse, sensor, etc., a display including a CRT, LCD, etc., an output unit 9007 including headphones, speakers, etc., a storage unit 9008 including a hard disk, etc., and a communication unit 9009 including a modem, terminal adapter, etc. are connected.

[0173] The CPU 9001 executes various processes (processing of an image processing device in this embodiment) in accordance with various programs stored in the ROM 9002 or various programs loaded from the storage unit 9008 to the RAM 9003. A graphics processing unit (GPU) may be provided to execute various processes (processing of an image processing device in this embodiment) in accordance with various programs stored in the ROM 9002 or various programs loaded from the storage unit 9008 to the RAM 9003, similar to the CPU 9001. The GPU is suitable for performing routine processing in parallel, and by applying it to neural network processing, for example, it is possible to improve processing speed compared to the CPU 9001. The RAM 9003 also stores data necessary for the CPU 9001 and GPU to execute various processes.

[0174] The communication unit 9009 performs communication processing via the Internet (not shown), for example, transmits data provided by the CPU 9001, and outputs data received from a communication partner to the CPU 9001, RAM 9003, and storage unit 9008. The storage unit 9008 exchanges data with the CPU 9001 and stores and erases information. The communication unit 9009 also performs communication processing of analog or digital signals with other devices.

[0175] The input / output interface 9005 is also connected to a drive 9010 as needed, and, for example, a magnetic disk 9011, an optical disk 9012, a flexible disk 9013, or a semiconductor memory 9014 is appropriately attached, and computer programs read from these are installed in the memory unit 9008 as needed.

[0176] Each drawing is merely an example for describing one or more embodiments. Each drawing may relate not only to one particular embodiment, but also to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate. [Explanation of symbols]

[0177] 1, 13 Statistics acquisition section 2. WB gain calculation section 3 WB gain output section 4, 14 Data conversion section 5, 10, 15, 18 Histogram Creation Section 6 Ring Buffer 7, 11, 16 Dissimilarity calculation part 8, 12, 17 Dissimilarity judgment part 9 State information storage section 19 Smoothing processing section 19A Adder 100, 200, 300, 400 Image Processing Device 1000 Electronic Imaging Device 1001 Imaging unit 1002 Pre-processing section 1003 White balance adjustment section 1004 Post-processing section 1005 Display section 1006 Storage section 9000 computers 9001 CPU 9002 ROM 9003 RAM 9004 Bus 9005 Input / Output Interface 9006 Input section 9007 Output section 9008 Storage section 9009 Communications Department 9010 Drive 9011 Magnetic disk 9012 Optical disc 9013 Flexible Disk 9014 Semiconductor memory DAT RAW data IMG image data INF State Information

Claims

1. a statistical value acquisition means for continuously inputting frames of a raw image and acquiring statistical values ​​of each of the RGB values ​​of the latest frame; a white balance gain calculation means for calculating a white balance gain of the latest frame; a white balance gain output means for holding the white balance gain calculated by the white balance gain calculation means and outputting the held white balance gain as a white balance gain to be applied to the latest frame; a conversion means for converting the statistical value of the latest frame into an HSV value; a histogram creation means for creating a histogram of H values ​​from the HSV values; a difference calculation means for calculating a difference between the histogram of the latest frame and each of histograms of one or more previous frames captured earlier than the latest frame; a difference determination means for controlling update of the white balance gain held by the white balance gain output means based on the difference. Image processing device.

2. the histogram creation means further creates a histogram of S and H values ​​from the HSV values; The image processing device according to claim 1 .

3. the dissimilarity calculation means calculates, for all bins, absolute values ​​of frequencies between a bin in a histogram of a value of interest among the HSV values ​​of the latest frame and the same bin in a histogram of the value of interest of the previous frame, and calculates the sum of the calculated values ​​as the dissimilarity between the bin in the histogram of the value of interest of the latest frame and the histogram of the value of interest of the previous frame.

3. The image processing device according to claim 1 or 2.

4. the difference determination means compares the difference with a first threshold, and when all of the calculated differences are smaller than the first threshold, stops output of the white balance gains from the white balance gain calculation means to the white balance gain output means; The image processing device according to claim 3 .

5. the first threshold is predetermined for each of the values ​​of interest; The image processing device according to claim 4 .

6. the statistical value acquisition means acquires statistical values ​​of each of the RGB values, using a frame input after the output of the white balance gain from the white balance gain calculation means to the white balance gain output means as the latest frame, and the difference calculation means calculates a difference between a histogram of the latest frame input after the output of the white balance gain from the white balance gain calculation means to the white balance gain output means has stopped and a histogram of one previous frame captured earlier than the latest frame; the difference determination means compares the difference with a second threshold, and when any of the calculated differences is greater than the second threshold, resumes output of the white balance gains from the white balance gain calculation means to the white balance gain output means; The image processing device according to claim 4 .

7. the second threshold value is predetermined for each of the HSV values ​​of interest; The image processing device according to claim 6 .

8. For each of the values ​​of interest, the second threshold is greater than the first threshold. The image processing device according to claim 7 .

9. the statistical value acquisition means divides the latest frame into a plurality of blocks of a predetermined size, and acquires statistical values ​​of the RGB values ​​of each block; the conversion means converts the statistical values ​​of the plurality of blocks into HSV values; the histogram creation means creates a histogram of each of the HSV values ​​of interest for each of the plurality of blocks, the dissimilarity calculation means calculates, for the plurality of blocks, a dissimilarity between the block in the histogram of the latest frame and a block in the histogram of the frame older than the latest frame that is at the same position as the block in the histogram of the latest frame; 3. The image processing device according to claim 1 or 2.

10. further comprising a smoothing processing means for smoothing the histogram created by the histogram creating means, the dissimilarity calculation means calculates the dissimilarity based on the histogram smoothed by the smoothing processing means.

3. The image processing device according to claim 1 or 2.

11. further comprising a histogram storage means for storing the latest frame and the past frame created by the histogram creation means, the dissimilarity calculation means is configured to be able to read out the latest frame and the past frame from the histogram storage means.

3. The image processing device according to claim 1 or 2.

12. the histogram storage means is configured as a ring buffer; the histogram of the oldest frame among the already stored histograms of frames prior to the latest frame is overwritten by the histogram of the latest frame; The image processing device according to claim 11 .

13. further comprising a state information storage means for storing state information indicating a state of the image processing device; the difference determination means stores the state information in a state information storage means, thereby controlling whether the state of the image processing device is a state in which the white balance gain calculation means outputs the white balance gain to the white balance gain output means, or a state in which the output of the white balance gain from the white balance gain calculation means to the white balance gain output means is stopped; the white balance gain calculation means refers to the state information and switches whether or not to calculate the white balance gain.

3. The image processing device according to claim 1 or 2.

14. The statistical value is an average value or an integrated value of each of the RGB values.

3. The image processing device according to claim 1 or 2.

15. Acquires statistical values ​​of each of the RGB values ​​of the latest frame of the continuously input RAW image; Calculating the white balance gain of the latest frame; retaining the white balance gain and outputting the retained white balance gain as a white balance gain to be applied to the latest frame; converting the statistics of the most recent frame into HSV values; A histogram of H values ​​is created from the HSV values; calculating a degree of difference between the histogram of the latest frame and each of histograms of one or more previous frames captured earlier than the latest frame; and controlling update of the held white balance gain based on the degree of difference. Image processing methods.

16. an imaging means for capturing a RAW image; a statistical value acquisition means for continuously inputting frames of the RAW image and acquiring statistical values ​​of each of the RGB values ​​of the latest frame; a white balance gain calculation means for calculating a white balance gain of the latest frame; a white balance gain output means for holding the white balance gain calculated by the white balance gain calculation means and outputting the held white balance gain as a white balance gain to be applied to the latest frame; a conversion means for converting the statistical value of the latest frame into an HSV value; a histogram creation means for creating a histogram of H values ​​from the HSV values; a difference calculation means for calculating a difference between the histogram of the latest frame and each of histograms of one or more previous frames captured earlier than the latest frame; a difference determination means for controlling update of the white balance gain held by the white balance gain output means based on the difference. Imaging device.

17. A process of acquiring statistical values ​​of each of the RGB values ​​of the latest frame of the continuously input RAW image; A process of calculating a white balance gain of the latest frame; A process of retaining the white balance gain and outputting the retained white balance gain as a white balance gain to be applied to the latest frame; converting the statistical values ​​of the latest frame into HSV values; A process of creating a histogram of H values ​​from the HSV values; a process of calculating a degree of difference between the histogram of the latest frame and each of histograms of one or more previous frames captured earlier than the latest frame; and controlling update of the stored white balance gain based on the degree of difference. program.

Images